scrapy启动后导入自建模块失败,找不到,哪里出了问题?
已经标记源码根了(解除又标记了几次),scrapy项目crawl启动后提示找不到自建的py文件,导入失败,sys.path.append也不行,运行单个py文件却能正常使用导入的自建模块
导入的模块点进去可以跳转进那个文件,现在该怎么办?
安装了一个第三方useragent库fake-useragent,使用的时候也报错,说找不到这个模块
2020-07-31 01:07:17 [scrapy.utils.log] INFO: Scrapy 2.2.1 started (bot: bttiantang)
2020-07-31 01:07:17 [scrapy.utils.log] INFO: Versions: lxml 4.5.1.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)],
pyOpenSSL 19.1.0 (OpenSSL 1.1.1g 21 Apr 2020), cryptography 3.0, Platform Windows-10-10.0.18362-SP0
2020-07-31 01:07:17 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-07-31 01:07:17 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'bttiantang',
'DOWNLOAD_DELAY': 3,
'NEWSPIDER_MODULE': 'bttiantang.spiders',
'SPIDER_MODULES': ['bttiantang.spiders'],
'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
'(KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'}
2020-07-31 01:07:17 [scrapy.extensions.telnet] INFO: Telnet Password: xxxxxxxxxxx
2020-07-31 01:07:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
Unhandled error in Deferred:
2020-07-31 01:07:17 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 192, in crawl
return self._crawl(crawler, *args, **kwargs)
File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
d = crawler.crawl(*args, **kwargs)
File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks
_inlineCallbacks(None, g, status)
--- <exception caught here> ---
File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 87, in crawl
self.engine = self._create_engine()
File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 101, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "d:\softwares\python3.8\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
self.downloader = downloader_cls(crawler)
File "d:\softwares\python3.8\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "d:\softwares\python3.8\lib\site-packages\scrapy\utils\misc.py", line 50, in load_object
mod = import_module(module)
File "d:\softwares\python3.8\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "F:\Python\Test_1\bttiantang\bttiantang\middlewares.py", line 7, in <module>
from fake_useragent import UserAgent
builtins.ModuleNotFoundError: No module named 'fake_useragent'
2020-07-31 01:07:17 [twisted] CRITICAL:
Traceback (most recent call last):
File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 87, in crawl
self.engine = self._create_engine()
File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 101, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "d:\softwares\python3.8\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
self.downloader = downloader_cls(crawler)
File "d:\softwares\python3.8\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "d:\softwares\python3.8\lib\site-packages\scrapy\utils\misc.py", line 50, in load_object
mod = import_module(module)
File "d:\softwares\python3.8\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "F:\Python\Test_1\bttiantang\bttiantang\middlewares.py", line 7, in <module>
from fake_useragent import UserAgent
ModuleNotFoundError: No module named 'fake_useragent'
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论