关于scrapy中ImagePipeline运行报错404的问题

Question

1 这两天看到有人利用爬虫框架scrapy中的ImagePipeline将图片下载到本地，本来想造个轮子，随便找个网站来熟悉一下，没想到被搞了一天还是一直报404错误，关键是我还不知道问题是在哪里，Google、百度、知乎到看了，跟官方文档的事例代码也差不多，但还是一直报404错误，实在是奔溃，特来此求助，望大神赐教！

2 没有图片下载下来；有200、有404状态码；都显示的是Item contains no images

3 `

pixabay.py:

import scrapy
from scrapy.http.request import Request
from pixabayweb.items import PixabaywebItem

class PixabaySpider(scrapy.Spider):
    name = 'pixabay'
    start_urls=['https://pixabay.com/zh/photos/?q=beauty&pagi=1']

    def parse(self, response):
        base_url='https://pixabay.com'
        links=response.xpath('//div[@class="flex_grid credits"]/div[@class="item"]')
        for link in links:
            url=base_url+link.xpath('.//a/@href').extract_first()
            yield Request(url=url,callback=self.parse_detail)

    def parse_detail(self,response):
        item=PixabaywebItem()
        item['image_urls']=response.xpath('//*[@id="media_container"]/img/@src').extract_first()
        yield item

pipelines.py:

import scrapy
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline
headers={
    'Referer':'https://pixabay.com',
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
}
class MyImagePipeline(ImagesPipeline):

    def get_media_requests(self,item,info):
        for image_url in item['image_urls']:
            yield scrapy.Request(url=image_url,headers=headers)

    def item_completed(self,results,item,info):
        image_paths=[x['path'] for ok,x in results if ok]
        if not image_paths:
            raise DropItem('Item contains no images')
        item['image_paths']=image_paths
        return item

settings.py:

BOT_NAME = 'pixabayweb'

SPIDER_MODULES = ['pixabayweb.spiders']
NEWSPIDER_MODULE = 'pixabayweb.spiders'

ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 2
COOKIES_ENABLED = False
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding':'gzip, deflate, sdch, br',
    'Accept-Language':'zh-CN,zh;q=0.8',
    'User-Agent':'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML like Gecko) Chrome/23.0.1271.95 Safari/537.11',
    'Host':'pixabay.com',
    'Connection':'keep-alive',
    'Upgrade-Insecure-Requests':'1',
    'Cookie':'is_human=1; csrftoken=9pFSFS4w495FfCFC33KecCXC2TLvevvprCs2g3ALaZTQYReQGJ1f8WStRcfndDPU; sessionid="eyJfbGFuZ3VhZ2UiOiJ6aCJ9:1dvDvQ:_DMlwOr3XOGC3i19M_U4BNEvK7U"; img_pp=100; _ga=GA1.2.2001502836.1505890547; _gid=GA1.2.1360455098.1505994393; client_width=1349',
}

ITEM_PIPELINES = {
   # 'pixabayweb.pipelines.PixabaywebPipeline': 300,
    'pixabayweb.pipelines.MyImagePipeline':1
}

IMAGES_STORE='E:\Python\pycharm\pixabay'

IMAGES_EXPIRES=90

items.py:

import scrapy

class PixabaywebItem(scrapy.Item):
    # define the fields for your item here like:
    image_urls=scrapy.Field()
    images=scrapy.Field()
    image_paths=scrapy.Field()

 

不知道是哪里有问题，望指教，不胜感激.
ps.想利用imagepipeline保存煎蛋网的妹子图也是想同的情况,报404错误，而且所有Url返回的都是Item contains no images。
我运行的环境是：win7 32位 pycharm:python3.4

Answer 1

404状态码 （未找到） 服务器找不到请求的网页。

pipelines.py:

异常是由这段代码产生的，也就是说images_paths为None.

  if not image_paths:
            raise DropItem('Item contains no images')

import scrapy
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline
headers={
    'Referer':'https://pixabay.com',
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
}
class MyImagePipeline(ImagesPipeline):

    def get_media_requests(self,item,info):
        for image_url in item['image_urls']:
            yield scrapy.Request(url=image_url,headers=headers)

    def item_completed(self,results,item,info):
        image_paths=[x['path'] for ok,x in results if ok]
        if not image_paths:
            raise DropItem('Item contains no images')
        item['image_paths']=image_paths
        return item

Answer 2

links=response.xpath('//div[@class="flex_grid credits"]/div[@class="item"]')
        for link in links:
            url=base_url+link.xpath('.//a/@href').extract_first()
            yield Request(url=url,callback=self.parse_detail)

这一步中，你想解析的是图片的详情页，继而拿到图片的原始url， 但是xpath('.//a/@href')既解析了详情页，同时也把user用户页面提取了出来，见下图，因此，你在下一步请求详情页的时候，在使用xpath提取，其中一部分提取到了图片的url，但是用户页面是提取不出来结果的，所以图片下载管道从你给的链接中，也就无法下载图片，因为他根本就不是图片下载链接.

简单地解决办法，将xpath('.//a/@href') 变成 xpath('./a/@href') 就是把a前面的斜线变成一条,只提取图片详情页，而不是用户页面。