scrapy 在爬取部分内容后自动关闭,为什么?
2018-09-19 11:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.zhihu.com/question/265749263> (referer: https://www.zhihu.com/question/265749263/answer/298529974)
2018-09-19 11:58:25 [scrapy.extensions.logstats] INFO: Crawled 20 pages (at 5 pages/min), scraped 10 items (at 5 items/min)
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/24252236>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/24252236>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/60410308>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>
None
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>
None
2018-09-19 11:58:25 [scrapy.core.engine] INFO: Closing spider (finished)
2018-09-19 11:58:25 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 7823,
'downloader/request_count': 20,
'downloader/request_method_count/GET': 20,
'downloader/response_bytes': 7113,
'downloader/response_count': 20,
'downloader/response_status_count/400': 20,
'dupefilter/filtered': 9,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 9, 19, 3, 58, 25, 849646),
'item_scraped_count': 28,
'log_count/DEBUG': 202,
'log_count/INFO': 10,
'request_depth_max': 3,
'response_received_count': 20,
'scheduler/dequeued': 20,
'scheduler/dequeued/memory': 20,
'scheduler/enqueued': 20,
'scheduler/enqueued/memory': 20,
'start_time': datetime.datetime(2018, 9, 19, 3, 55, 14, 766276)}
2018-09-19 11:58:25 [scrapy.core.engine] INFO: Spider closed (finished)### 问题描述
爬虫在运行一段时间后自动关闭,爬取了一部分的数据,但是start_urls(start_url
s中有多条链接,)中的链接并未爬取完的情况下爬虫就自动关闭了
请问
2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263> None
这个反馈是在表达什么,为何会自动关闭
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
请问你这个问题解决了吗