扩展多个 HttpWebRequest？
我正在构建一个服务器应用程序，需要持续向其他几个服务器执行大量 http 请求。目前，我基本上设置了大约 30 个线程，并在每个线程上持续同步运行 HttpWebRequest，实现每秒大约 30 个请求的吞吐量。
我确实在 app.config 中设置了 ServicePoint ConnectionLimit，因此这不是限制因素。
我需要大幅扩大规模。至少我需要更多的 CPU 马力，但我想知道通过使用 HttpWebRequest 对象的异步方法（例如： .BeginGetResponse() ）是否会获得任何优势，而不是自己创建线程并使用这些线程上的同步方法（例如： .GetResponse() ）。
如果我使用异步方法，我显然必须重新设计我的应用程序，所以我想知道在我去重新编码所有内容之前是否有人可能有一些见解，以防我出去吃午饭。
谢谢！
I'm building a server application that needs to perform a lot of http requests to a couple other servers on an ongoing basis. Currently, I'm basically setting up about 30 threads and continuously running HttpWebRequests synchronously on each thread, achieving a throughput of about 30 requests per second.
I am indeed setting the ServicePoint ConnectionLimit in the app.config so that's not the limiting factor.
I need to scale this up drastically. At the very least I'll need some more CPU horse power, but I'm wondering if I would gain any advantages by using the async methods of the HttpWebRequest object (eg: .BeginGetResponse() ) as opposed to creating threads myself and using the synchronous methods (eg: .GetResponse() ) on these threads.
If I go with the async methods, I obviously have to significantly redesign my app, so I'm wondering if anyone might have some insight before I go and recode everything, in case I'm out to lunch.
Thanks!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
如果您使用的是 Windows NT，则 System.Net.Sockets.Socket 类始终使用 IO 完成端口进行异步操作。异步模式下的 HTTPWebRequest 使用异步套接字，因此将使用 IOCP。
如果不进行详细的基准测试，很难说我们的瓶颈是在 HttpWebRequest 内部，还是在应用程序的堆栈中，或者在远程端的服务器中。但顺便说一句，asyncc 肯定会给你带来更好的性能，因为它最终会在幕后使用 IOCP。重新实现异步应用程序并不那么困难。
因此，我建议您首先将应用程序架构更改为异步。然后看看您获得的最大吞吐量是多少。然后您可以开始进行基准测试并找出瓶颈所在并消除它。
If you are on Windows NT, then System.Net.Sockets.Socket class always uses IO Completion ports for async operations. And HTTPWebRequest in async mode uses async sockets, and hence will be using IOCP.
Without doing detailed benchmarking, it is difficult to say if our bottleneck is inside HttpWebRequest, or up the stack, in your application, or on the remote side, in the server. But offhand, for sure, asyncc will give you better performance, because it will end up using IOCP under the covers. And reimplementing the app for async is not that difficult.
So, I would suggest that you first change your app architecture to async. Then see how much max throughput you are getting. Then you can start benchmarking and finding out where the bottleneck is, and removing that.
到目前为止，对我来说最快的结果是使用 75 个线程运行同步 httpwebrequest。
Windows 2003 服务器、4 核 3ghz、100mb 连接上每秒大约有 140 个请求。
异步 Httprequest/winsock 卡在大约 30-50 请求/秒。没有测试同步winsock，但我想它会给你与httpwebrequest 相同的结果。
测试针对 1,200,000 个博客源。
上个月一直在努力解决这个问题，所以知道是否有人设法从 .net 中榨取更多内容会很有趣？
编辑
新测试：使用 xfserver iocp 组件获得 350req/sec。在获得更大的结果之前，使用了一组线程，每个线程都有一个实例。该库的“客户端部分”有一些非常烦人的错误，这些错误使得实现比“服务器部分”更困难。不是您所要求的，也不推荐，而是某种步骤。
接下来：之前的winsock测试没有使用3.5 SocketAsyncEventArgs，这将是下一个。
答案
你的问题的答案，不，不值得付出努力。
异步 HttpWebRequest 方法卸载主线程，同时将下载保持在后台，它不会提高请求的数量/可扩展性。 （至少在 3.5 中没有，在 4.0 中可能有所不同？）
但是，可能值得一看的是围绕异步套接字/SocketAsyncEventArgs 构建自己的包装器，其中 iocp 工作，并且可能实现类似于 HttpWebRequest 的开始/结束模式（为了最简单的可能）当前代码中的实现）。改进确实是巨大的。
Fastest result so far for me is using 75 threads running sync httpwebrequest.
About 140 requests per second on a windows 2003 server, 4core 3ghz, 100mb connection.
Async Httprequest / winsock got stuck at about 30-50 req/sec. Did not test sync winsock but I guess it would give you about the same result as httpwebrequest.
Tests was against 1 200 000 blog feeds.
Been struggling with this the last month so it would be interesting to know if someone managed to squeeze more out of .net?
EDIT
New test: Got 350req/sec with the xfserver iocp component. Used a bunch of threads with one instance each before any greater result. The "client part" of the lib had a couple of really annoying bugs that made implementation harder then the "server part". Not what you're asking for and not recommended but some kind of step.
Next: Former winsock test did not use the 3.5 SocketAsyncEventArgs, that will be next.
ANSWER
The answer to your question, no it will not be worth the effort.
The async HttpWebRequest methods offloads main thread while keeping download in background, it does not improve the number/scalability of requests. (at least not in 3.5, might be different in 4.0?)
However, what might be worth looking at is building your own wrapper around async sockets/SocketAsyncEventArgs where iocp works and perhaps implement a begin/end pattern similar to HttpWebRequest (for easiest possible implementation in current code). The improvement is really enormous.