如何在砂纸中构建自己的中间件？

Question

我刚刚开始学习纸巾，我有一个问题。对于我的“蜘蛛”，我必须从Google表格表中获取一个URL（start_url）列表，并且我有此代码：

import gspread
from oauth2client.service_account import ServiceAccountCredentials

scope = ['https://spreadsheets.google.com/feeds','https://www.googleapis.com/auth/drive']
creds = ServiceAccountCredentials.from_json_keyfile_name('token.json', scope)


client = gspread.authorize(creds)
sheet = client.open('Sheet_1')
sheet_instance = sheet.get_worksheet(0)
records_data = sheet_instance.col_values(col=2)

for link in records_data:
    print(link)
    ........

我如何配置中间件，以便当蜘蛛（scrappy crawl my_spider）时是否启动，此代码中的链接会自动替换为start_urls？也许我需要在Middlewares.py中创建一个类？ 我将感谢任何帮助，例如。 该规则有必要适用于所有新蜘蛛，从start_requests中的文件生成列表（例如start_urls = [l.strip（）for Open String（'urls.txt'）.readline（）] .Readline（）]）不方便...

Answer 1

阅读

” py：

import scrapy


class ExampleSpider(scrapy.Spider):
    name = 'example'

    custom_settings = {
        'SPIDER_MIDDLEWARES': {
            'tempbuffer.middlewares.ExampleMiddleware': 543,
        }
    }
    
    def parse(self, response):
        print(response.url)

middlewares.py：

class ExampleMiddleware(object):
    def process_start_requests(self, start_requests, spider):
        # change this to your needs:
        with open('urls.txt', 'r') as f:
            for url in f:
                yield scrapy.Request(url=url)

urls.txt：

https://example.com
https://example1.com
https://example2.org

输出：

[scrapy.core.engine] DEBUG: Crawled (200) <GET https://example2.org> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://example.com> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://example1.com> (referer: None)
https://example2.org
https://example.com
https://example1.com