在一个 scrapy-spider 中抓取多个站点
我正在6种不同的蜘蛛中刮下6个站点。但是现在,我必须将这些站点刮在一个蜘蛛中。有没有一种方法可以在同一蜘蛛中刮擦多个链接?
I am scraping 6 sites in 6 different spiders. But now, I have to scrape these sites in one single spider. Is there a way of scraping multiple links in the same spider??
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

发布评论
评论(2)
厌味2025-01-27 11:49:39
import spider1
import spider2
import spider3
from scrapy.crawler import CrawlerProcess
if require_spider1:
spider = spider1
urls = ['https://site1.com/']
elif require_spider2:
spider = spider2
urls = ['https://site2.com/', 'https://site2-1.com/']
elif require_spider3:
spider = spider3
urls = ['https://site3.com']
process = CrawlerProcess()
process.crawl(spider, urls=urls)
process.start()
~没有更多了~
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
我这样做是通过
I did this by