using System;
using System.Threading;
using System.Collections.Generic;
using System.Net;
using System.IO;
namespace WebClientApp
{
class MainClassApp
{
private static int requests = 0;
private static object requests_lock = new object();
public static void Main() {
List<string> urls = new List<string> { "http://www.google.com", "http://www.slashdot.org"};
foreach(var url in urls) {
ThreadPool.QueueUserWorkItem(GetUrl, url);
}
int cur_req = 0;
while(cur_req<urls.Count) {
lock(requests_lock) {
cur_req = requests;
}
Thread.Sleep(1000);
}
Console.WriteLine("Done");
}
private static void GetUrl(Object the_url) {
string url = (string)the_url;
WebClient client = new WebClient();
Stream data = client.OpenRead (url);
StreamReader reader = new StreamReader(data);
string html = reader.ReadToEnd ();
/// Do something with html
Console.WriteLine(html);
lock(requests_lock) {
requests++;
}
}
}
}
WebClient probably has a more simple api but both should work.
As far as running a lot of requests you should implement it using multiple threads or a thread pool. If the urls are on the same server you should be careful not to overload it.
If you want examples to implement it via a thread pool I can provide them.
Update
using System;
using System.Threading;
using System.Collections.Generic;
using System.Net;
using System.IO;
namespace WebClientApp
{
class MainClassApp
{
private static int requests = 0;
private static object requests_lock = new object();
public static void Main() {
List<string> urls = new List<string> { "http://www.google.com", "http://www.slashdot.org"};
foreach(var url in urls) {
ThreadPool.QueueUserWorkItem(GetUrl, url);
}
int cur_req = 0;
while(cur_req<urls.Count) {
lock(requests_lock) {
cur_req = requests;
}
Thread.Sleep(1000);
}
Console.WriteLine("Done");
}
private static void GetUrl(Object the_url) {
string url = (string)the_url;
WebClient client = new WebClient();
Stream data = client.OpenRead (url);
StreamReader reader = new StreamReader(data);
string html = reader.ReadToEnd ();
/// Do something with html
Console.WriteLine(html);
lock(requests_lock) {
requests++;
}
}
}
发布评论
评论（2）
WebClient 可能有一个更简单的 api，但两者都应该可以工作。
至于运行大量请求，您应该使用多个线程或线程池来实现它。如果网址位于同一服务器上，则应小心不要使其过载。
如果您想要通过线程池实现它的示例，我可以提供它们。
更新
}
WebClient probably has a more simple api but both should work.
As far as running a lot of requests you should implement it using multiple threads or a thread pool. If the urls are on the same server you should be careful not to overload it.
If you want examples to implement it via a thread pool I can provide them.
Update
}
使用
Parallel.Invoke设置所有请求并给予它慷慨的MaxDegreesOfParallelism。您将花费大部分时间等待 I/O，因此尽可能多地使用多线程。
Use
Parallel.Invoketo set up all the requests and give it a generousMaxDegreesOfParallelism.You'll be spending most of your time waiting on I/O, so make as much use of multi-threading as possible.