Pyspider webui调试错误
1.在Ubuntu上搭建好Pyspider环境后,用Pyspider all命令启动,然后在webui中单步调试测试时,遇到下面异常:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pyspider/run.py", line 360, in <lambda>
app.config['fetch'] = lambda x: webui_fetcher.fetch(x)
File "/usr/local/lib/python2.7/dist-packages/pyspider/fetcher/tornado_fetcher.py", line 114, in fetch
return self.async_fetch(task, callback).result()
File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 129, in result
raise_exc_info(self.__exc_info)
File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 227, in wrapper
runner.run()
File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 531, in run
yielded = self.gen.send(next)
File "/usr/local/lib/python2.7/dist-packages/pyspider/fetcher/tornado_fetcher.py", line 136, in async_fetch
result = self.handle_error(type, url, task, e)
TypeError: handle_error() takes exactly 6 arguments (5 given)
测试所用的python脚本是github上官方demo程序;然后又使用本人在Windows平台上已经测试能正常运行的程序,同样抛出以上异常
终端报错信息是:
[E 160817 11:00:53 tornado_fetcher:135] 'module' object has no attribute 'maybe_future'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pyspider/fetcher/tornado_fetcher.py", line 127, in async_fetch
result = yield gen.maybe_future(self.data_fetch(url, task))
AttributeError: 'module' object has no attribute 'maybe_future'
本人测试的程序是:
from pyspider.libs.base_handler import *
class Handler(BaseHandler):
crawl_config = {
}
@every(minutes=24 * 60)
def on_start(self):
self.crawl('http://scrapy.org/', callback=self.index_page)
@config(age=10 * 24 * 60 * 60)
def index_page(self, response):
for each in response.doc('a[href^="http"]').items():
self.crawl(each.attr.href, callback=self.detail_page)
def detail_page(self, response):
return {
"url": response.url,
"title": response.doc('title').text(),
}
本人刚接触Pyspider,想研究源码但是心有余力不足,所以在这儿虚心向各位大神请教,望得到大神指点!
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
我在Github上提交问题后,Pyspider作者,Binux大神问我tornado版本,这个提醒了我,原tornado版本是3.2,于是我用命令
升级了tornado版本,问题解决了,多谢足兆叉虫,您应该就是Binux吧?