Scrapy is a python library that crawls web sites. It is fairly small (compared to Nutch) and designed for limited site crawls. It has a Django type MVC style that I found pretty easy to customize.
For the crawling part, I really like anemone and crawler4j. They both allow you to add your custom logic for links selection and page handling. For each page that you decide to keep, you can easily add the call to Solr.
It depends on how many web sites and so URLs you think crawl. Apache Nutch stores page documents on Apache HBase (which relies on Apache Hadoop), it's solid but very hard to setup and administrate.
Since a crawler is just a page fetch (like a CURL) and retrieve list of links to feed your URLs data base, I am sure you can write a crawler on your own (especially if you have a few web sites), use a simple MySQL database (maybe a queue software like RabbitMQ to schedule the crawl jobs).
On other side, a crawler could be more sophisticated, you could want to remove from your HTML document the HEAD part, and keep only the real "content" of the page etc...
Also, Nutch can rank your pages, with a PageRank algo., you could use Apache Spark to do the same thing (more efficiently because Spark can cache data in memory).
I do believe the nutch is the best choice for you application, but if you want, there is a simple tool: Heritrix. Besides that, I recommand js for the front-end language, because solr returns json which is easily handled by js.
发布评论
评论（5）
Scrapy 是一个用于抓取网站的 Python 库。它相当小（与 Nutch 相比），专为有限的站点爬行而设计。它有 Django 类型的 MVC 风格，我发现很容易定制。
Scrapy is a python library that crawls web sites. It is fairly small (compared to Nutch) and designed for limited site crawls. It has a Django type MVC style that I found pretty easy to customize.
对于抓取部分，我非常喜欢 anemone 和 crawler4j。它们都允许您添加用于链接选择和页面处理的自定义逻辑。对于您决定保留的每个页面，您可以轻松添加对 Solr 的调用。
For the crawling part, I really like anemone and crawler4j. They both allow you to add your custom logic for links selection and page handling. For each page that you decide to keep, you can easily add the call to Solr.
这取决于您认为可以抓取的网站和 URL 的数量。 Apache Nutch 在 Apache HBase（依赖于 Apache Hadoop）上存储页面文档，它很可靠，但很难设置和管理。
由于爬虫只是一个页面获取（如 CURL）并检索链接列表以提供您的 URL 数据库，我相信您可以自己编写一个爬虫（特别是如果您有几个网站），请使用一个简单的MySQL 数据库（可能是像 RabbitMQ 这样的队列软件来安排爬网作业）。
另一方面，爬虫可能更复杂，您可能希望从 HTML 文档中删除 HEAD 部分，并仅保留页面的真实“内容”等...
此外，Nutch 可以使用 PageRank 对您的页面进行排名algo.，您可以使用 Apache Spark 来做同样的事情（更有效，因为 Spark 可以在内存中缓存数据）。
It depends on how many web sites and so URLs you think crawl. Apache Nutch stores page documents on Apache HBase (which relies on Apache Hadoop), it's solid but very hard to setup and administrate.
Since a crawler is just a page fetch (like a CURL) and retrieve list of links to feed your URLs data base, I am sure you can write a crawler on your own (especially if you have a few web sites), use a simple MySQL database (maybe a queue software like RabbitMQ to schedule the crawl jobs).
On other side, a crawler could be more sophisticated, you could want to remove from your HTML document the HEAD part, and keep only the real "content" of the page etc...
Also, Nutch can rank your pages, with a PageRank algo., you could use Apache Spark to do the same thing (more efficiently because Spark can cache data in memory).
在，C#中，但是简单很多并且可以直接与作者交流。 （我）
我曾经使用过 Nutch，你是对的；它是一只值得共事的熊。
http://arachnode.net
In, C#, but a lot simpler and you can communicate directly with the author. (me)
I used to use Nutch and you are correct; it is a bear to work with.
http://arachnode.net
我确实相信 nutch 是您应用程序的最佳选择，但如果您愿意，有一个简单的工具： Heritrix。
除此之外，我推荐前端语言使用js，因为solr返回的json很容易被js处理。
I do believe the nutch is the best choice for you application, but if you want, there is a simple tool: Heritrix.
Besides that, I recommand js for the front-end language, because solr returns json which is easily handled by js.