scrapy.FormRequest使用代理请求超时,但是requests请求正常
同样的代理ip,用requests请求正常,用scrapy.FormRequest请求就超时
相关代码
In [11]: r = requests.post('http://httpbin.org/post', proxies={'http': proxy_server, 'https': proxy_server})
2018-07-10 14:10:04 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 113.128.26.187
2018-07-10 14:10:04 [urllib3.connectionpool] DEBUG: http://113.128.26.187:4352 "POST http://httpbin.org/post HTTP/1.1" 200 275
In [12]: r.text
Out[12]: '{"args":{},"data":"","files":{},"form":{},"headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate","Connection":"close","Content-Length":"0","Host":"httpbin.org","User-Agent":"python-requests/2.18.4"},"json":null,"origin":"113.128.26.187","url":"http://httpbin.org/post"}\n'
In [13]: proxy_server
Out[13]: 'http://113.128.26.187:4352'
In [14]: req = scrapy.FormRequest('http://httpbin.org/post',formdata={'foo':'bar'}, meta={'proxy': proxy_server})
In [15]: fetch(req)
2018-07-10 14:13:58 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <POST http://httpbin.org/post> (failed 1 times): User timeout caused connection failure: Getting http://httpbin.org/post took longer than 180.0 seconds..
如图
2018-07-10 14:27:39 使用FormRequest
请求
2018, 7, 10, 14, 28, 19使用Requests
请求,以证明代理ip没有失效
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论