scrapy 跟踪大型 XML feed 链接

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

要求提供代码的问题必须表现出对所解决问题的最低限度的了解。包括尝试的解决方案、为什么它们不起作用以及预期结果。另请参阅：Stack Overflow 问题清单

已关闭 10 年前。

Answer 1

首先阅读： http://readthedocs.org/docs/scrapy/ en/latest/intro/tutorial.html

我在 scrapy 中创建了一个项目。以下是获取该特定 XML 的所有 url 的代码。您应该使用蜘蛛目录。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
from scrapy.item import BaseItem
from scrapy.spider import BaseSpider
from scrapy.selector import XmlXPathSelector
from scrapy.utils.spider import create_spider_for_request
from scrapy.utils.misc import load_object
from scrapy.utils.response import open_in_browser

class TestSpider(BaseSpider):
    name = "test"
    start_urls = ["http://fgeek.kapsi.fi/test.xml"]

    def parse(self, response):
        xpath = XmlXPathSelector(response)
        count = 0
        for url in xpath.select('entries/entry/url').extract():
            print url