Question

现在越来越多的网站都采用了Ajax异步加载技术，爬取难度加大，同时反爬手段对HTTP请求的检测也越来越成熟，那么如何使用PySpider爬取Ajax请求和修改HTTP请求呢？这就是本节要讲的内容。

18.5.1　Ajax爬取

一般对于Ajax请求的爬取主要有两种方法，已经在第9章进行了详细的讲解，同样PySpider也支持这两种方法。其中一种方式是直接模拟Ajax请求。还是以MTime电影网为例，如果大家不熟悉，可以回顾一下第9章。如图18-15所示，票房链接为：http://service.library.mtime.com/Movie.apiAjax_CallBack=true&？Ajax_CallBackType=Mtime.Library.Services&Ajax_CallBackMethod=GetMovieOverviewRating&Ajax_CrossDomain=1&Ajax_RequestUrl=http%3A%2F%2Fmovie.mtime.com%2F217130%2F&t=2016111321341844484&Ajax_CallBackArgument0=217130

图18-15　Ajax请求

响应信息为：

  var result_2016111321341844484 = { "val：ue":{"isRelease":true,"movieRating":{"Mov-
ieId":217130,"RatingFinal":8,"RDirectorFinal":8.3,"ROtherFinal":7.5,"RPictureFinal":
8.8,"RShowFinal":0,"RStoryFinal":7.7,"RTotalFinal":0,"Usercount":3106,"AttitudeCount":2736,"UserId":0,"EnterTime":0,"JustTotal":0,"RatingCount":0,"TitleCn":"","TitleEn":"","Year":"","IP":0},"movieTitle":"比利·林恩的中场战事","tweetId":0,"userLastComment
":"","userLastCommentUrl":"","releaseType":1,"boxOffice":{"Rank":14,"TotalBoxOffice":"1.59","TotalBoxOfficeUnit":"亿","TodayBoxOffice":"9.5","TodayBoxOfficeUnit":"万","ShowDays":22,"EndDate":"2016-12-02 11:30","FirstDayBoxOffice":"2542.51","First
DayBoxOfficeUnit":"万"}},"error":null};var movieOverviewRatingResult=result_20161113
 21341844484;

我们可以通过如下代码对请求进行模拟。

  from pyspider.libs.base_handler import *
  class Handler(BaseHandler):
     @every(minutes=10)
     def on_start(self):
       self.crawl('http://service.library.mtime.com/Movie.api'
            'Ajax_CallBack=true&Ajax_CallBackType=Mtime.Library.Services'
            '&Ajax_CallBackMethod=GetMovieOverviewRating'
            '&Ajax_CrossDomain=1'
            '&Ajax_RequestUrl=http%3A%2F%2Fmovie.mtime.com%2F217130%2F'
            '&t=2016111321341844484'
            '&Ajax_CallBackArgument0=217130',
            callback=self.index_page)
  
     @config(age=10*60)
     def index_page(self, response):
       return response.json

如果想解析响应的内容，直接可以使用第9章的代码，这里不再赘述。

18.5.2　HTTP请求实现

1.设置请求头

  headers ={'User-Agent':random.choice(USER_AGENTS),
     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
     'Accept-Language': 'zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3',
     'Accept-Encoding': 'gzip, deflate, br'}
  @every(minutes=24 * 60)
  def on_start(self):
     self.crawl('http://movie.douban.com/tag/', headers = self.headers,callback=
       self.index_page)

2.设置cookie

  def on_start(self):
     self.crawl('http://movie.douban.com/tag/', cookies={"key": value},callback=self.index_page)

3.GET请求（带参数）

  self.crawl('http://httpbin.org/get', callback=self.callback,params={'a': 123, 'b': 
     'c'})
  self.crawl('http://httpbin.org/geta=123&b=c', callback=self.callback)

这两个请求效果是一样的。

4.POST请求

  def on_start(self):
     self.crawl('http://httpbin.org/post', callback=self.callback,
              method='POST', data={'a': 123, 'b': 'c'})

5.代理设置

PySpider仅支持HTTP代理，代理格式：username：password@hostname：port。可以使用如下代码对项目进行代理设置：

  class Handler(BaseHandler):
     crawl_config = {
       'proxy': 'localhost:8080'
     }