如何忽略废纸中的URL参考
我正在使用零工来刮擦一个包含很多菜单菜单的网站。 问题在于，我正在提取与网站中同一项目/subitem相对应的多个URL。我正在提取它们，就好像它们是不同的项目一样，因为URL包含“ Ref =”部分。 例如：
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_1
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_2
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_3
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_4
所有这些URL都对应于网站中的同一ssubsubitem_abc。 我想以这种方式提取与subbitem_abc相对应的一个URL
https://thestore/category1/subitem/subsubitem_ABC
，而是要减少爬虫的时间消耗，并避免对同一subsubitem或subItem或item或项目的重复URL。
到目前为止，我有这些规则：
rules = [
Rule(
LinkExtractor(
restrict_xpaths=['my_xpath"]//a',],
),
follow=True,
callback='parse_categories'
)
]
是否可以将某些内容添加到规则/Linkextractor中以避免URL中的引用？
I'm using Scrapy to scrape a website that contains a menu with a lot of sublevel menus.
The problem is that I'm extracting multiple URLs that correspond to the same item/subitem in the website. I'm extracting them as if they were different items because the URLs contain a "ref=" section.
For example:
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_1
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_2
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_3
https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_4
All these URLs correspond to the same ssubsubitem_ABC in the website.
Instead of this, I would like to extract only one URL corresponding to the subsubitem_ABC
https://thestore/category1/subitem/subsubitem_ABC
This way, mi intention is to reduce the time consumption of the crawler and avoid duplicated URLs for the same subsubitem or subitem or item.
So far I have these rules:
rules = [
Rule(
LinkExtractor(
restrict_xpaths=['my_xpath"]//a',],
),
follow=True,
callback='parse_categories'
)
]
Is there something I can add to the Rule/LinkExtractor to avoid the references in the URLs?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
如果您只想刮擦“ https：// thestore/category1/subitem/subiTem_abc/ref = asd_asd_1”，则可以使用正则表达式而不是x_path。它可以允许= r'https：//thestore/category1/subitem/subitem_abc/ref（。*？）1'。希望这可以帮助您。
If you like to scrape only "https://thestore/category1/subitem/subsubitem_ABC/ref=asd_asd_1", you can use regular expression rather than X_path. It could be allow = r'https://thestore/category1/subitem/subsubitem_ABC/ref(.*?)1'. Hope this can help you.