对于实时/并行 HTTP 爬虫来说，有什么好的库/平台吗？
我正在构建一个爬虫，它可以实时从多个网站并行获取信息，以响应客户端对此信息的请求。我需要从 10-20 个网站请求特定页面，解析其内容以获取特定的信息片段，并尽快将此信息返回给客户端。我想异步执行此操作，以便客户端在准备好后立即获取显示的第一个结果，而其他请求仍处于待处理状态。
我有 Ruby 背景，因此更喜欢在 Ruby 中构建解决方案 - 然而，并行性和速度正是 Ruby 所不擅长的。我相信 EventMachine 和 Typhoeus 可以解决这个问题，但我也在强烈考虑 node.js ，因为我非常了解 javascript 并且似乎是为这种事情而构建的。
无论我选择什么，我还需要一种有效的方式将结果传达给客户。我正在考虑使用普通的 AJAX（但这需要轮询服务器）、Web 套接字（但这需要旧版浏览器的回退）以及用于持久客户端/服务器通信的特定解决方案，例如 抽筋、剑圣 和 推手。
有人想分享一些经验和/或建议吗？
I am building a crawler that fetches information in parallel from a number of websites in real-time in response to a request for this information from a client. I need to request specific pages from 10-20 websites, parse their contents for specific snippets of information and return this information to the client as fast as possible. I want to do it asynchronously, so the client gets the first result displayed as soon as it is ready, while the other requests are still pending.
I have a Ruby background, and would therefore prefer to build the solution in a Ruby - however, parallelism and speed is exactly what Ruby is known NOT to excel at. I believe that libraries such as EventMachine and Typhoeus can remedy that, but I am also strongly considering node.js, because I know javascript quite well and seems to be built for this kind of thing.
Whatever I choose, I also need an efficient way to communicate the results back to the client. I am considering plain AJAX (but that would require polling the server), web sockets (but that would require fallback for older browsers) and specific solutions for persistent client/server communication such as Cramp, Juggernaut and Pusher.
Does anyone have any experience and/or recommendations they would like to share?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
如果您的抓取工具需要 Javascript 支持，我建议使用 http://htmlunit.sourceforge.net/。
有一个可用的 JRuby 包装器 http://celerity.rubyforge.org/
功能（取自站点）包括：
渲染或不必要的下载
背景）
If your crawler needs Javascript support, I recommend http://htmlunit.sourceforge.net/.
There is a JRuby wrapper available http://celerity.rubyforge.org/
Features (taken from site) include:
rendering or unessential downloads
background)
Node 绝对有能力处理这种类型的任务 - 异步套接字和 http 通信是内置的，使用起来非常愉快。
我的大部分工作都是 j/Ruby，我发现向服务器端 JavaScript 的过渡非常轻松 - 多年的 Web 开发经验意味着我非常了解 js，而且无论使用哪种语言，服务器开发概念基本上都是相同的。
在通信方面 Socket.io 是一个优秀的客户端和服务器框架，用于处理节点中的套接字通信 - 它支持 flash， ajax 和 websocket 通道这意味着它可以在几乎任何现代（和一些较旧的）浏览器上使用。
node is definitely capable of handling this type of task - async socket and http communciation is baked in and really pleasant to work with.
Most of my work is j/Ruby and I have found the transition to server-side JavaScript pretty painless - years of web dev mean I know js pretty well and the server development concepts are largely the same regardless of language.
In terms of communication Socket.io is an excellent client and server framework for handling socket communication in node - it supports flash, ajax and websocket channels which means it can be used on just about any modern (and some older) browsers.