scrapy 不能提取下一页

Question

问题描述

不能获取下一页

相关代码

// 请把代码文本粘贴到下方（请勿用图片代替代码）

import scrapy
from qsbk.items import QsbkItem
from scrapy.http.response.html import HtmlResponse
from scrapy.selector.unified import SelectorList

class QsbkSpiderSpider(scrapy.Spider):

name = 'qsbk_spider'
allowed_domains = ['qiushibaike.com']
start_urls = ['https://www.qiushibaike.com/text/page/1/']
base_domain = 'https://www.qiushibaike.com/'

def parse(self, response):
    duanzidivs = response.xpath("//div[@id='content-left']/div")
    for duanzidiv in duanzidivs:
        author = duanzidiv.xpath(".//h2/text()").get().strip()
        content = duanzidiv.xpath(".//div[@class='content']//text()").getall()
        content = "".join(content).strip()

        item = QsbkItem(author=author,content=content)
        yield item

    next_url = response.xpath("//ul[@class='pagination']/li[last()]/a/@href").get()
    if not next_url:
        return
    else:
        yield scrapy.Request(self.base_domain+next_url,callback=self.parse)

从这里开始发生错误，前一个页面正常提取，到第二页就发生错误了？

2019-02-23 14:07:32 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://text/&gt; from <GET https://www.qiushibaike.com//...;
2019-02-23 14:07:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://text/&gt; (failed 1 times): DNS lookup failed: no results for hostname lookup: text.
2019-02-23 14:07:37 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://text/&gt; (failed 2 times): DNS lookup failed: no results for hostname lookup: text.
2019-02-23 14:07:40 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://text/&gt; (failed 3 times): DNS lookup failed: no results for hostname lookup: text.
2019-02-23 14:07:40 [scrapy.core.scraper] ERROR: Error downloading <GET https://text/&gt;
Traceback (most recent call last):
File "D:venvarticle_spiderlibsite-packagestwistedinternetdefer.py", line 1416, in _inlineCallbacks

result = result.throwExceptionIntoGenerator(g)

File "D:venvarticle_spiderlibsite-packagestwistedpythonfailure.py", line 491, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

File "D:venvarticle_spiderlibsite-packagesscrapycoredownloadermiddleware.py", line 43, in process_request

defer.returnValue((yield download_func(request=request,spider=spider)))

File "D:venvarticle_spiderlibsite-packagestwistedinternetdefer.py", line 654, in _runCallbacks

current.result = callback(current.result, *args, **kw)

File "D:venvarticle_spiderlibsite-packagestwistedinternetendpoints.py", line 975, in startConnectionAttempts

"no results for hostname lookup: {}".format(self._hostStr)

twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: text.

问题描述

问题出现的平台版本及自己尝试过哪些方法

相关代码

// 请把代码文本粘贴到下方（请勿用图片代替代码）

你期待的结果是什么？实际看到的错误信息又是什么？