scrapy启动后导入自建模块失败,找不到,哪里出了问题?

发布于 2022-09-12 03:56:58 字数 6178 浏览 32 评论 0

已经标记源码根了(解除又标记了几次),scrapy项目crawl启动后提示找不到自建的py文件,导入失败,sys.path.append也不行,运行单个py文件却能正常使用导入的自建模块
QQ截图20200730232527.png
导入的模块点进去可以跳转进那个文件,现在该怎么办?
安装了一个第三方useragent库fake-useragent,使用的时候也报错,说找不到这个模块

2020-07-31 01:07:17 [scrapy.utils.log] INFO: Scrapy 2.2.1 started (bot: bttiantang)
2020-07-31 01:07:17 [scrapy.utils.log] INFO: Versions: lxml 4.5.1.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)],
pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 3.0, Platform Windows-10-10.0.18362-SP0
2020-07-31 01:07:17 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-07-31 01:07:17 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'bttiantang',
 'DOWNLOAD_DELAY': 3,
 'NEWSPIDER_MODULE': 'bttiantang.spiders',
 'SPIDER_MODULES': ['bttiantang.spiders'],
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'}
2020-07-31 01:07:17 [scrapy.extensions.telnet] INFO: Telnet Password: xxxxxxxxxxx
2020-07-31 01:07:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
Unhandled error in Deferred:
2020-07-31 01:07:17 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 87, in crawl
    self.engine = self._create_engine()
  File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 101, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "d:\softwares\python3.8\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
    self.downloader = downloader_cls(crawler)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\utils\misc.py", line 50, in load_object
    mod = import_module(module)
  File "d:\softwares\python3.8\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 783, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "F:\Python\Test_1\bttiantang\bttiantang\middlewares.py", line 7, in <module>
    from fake_useragent import UserAgent
builtins.ModuleNotFoundError: No module named 'fake_useragent'

2020-07-31 01:07:17 [twisted] CRITICAL:
Traceback (most recent call last):
  File "d:\softwares\python3.8\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 87, in crawl
    self.engine = self._create_engine()
  File "d:\softwares\python3.8\lib\site-packages\scrapy\crawler.py", line 101, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "d:\softwares\python3.8\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
    self.downloader = downloader_cls(crawler)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "d:\softwares\python3.8\lib\site-packages\scrapy\utils\misc.py", line 50, in load_object
    mod = import_module(module)
  File "d:\softwares\python3.8\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "F:\Python\Test_1\bttiantang\bttiantang\middlewares.py", line 7, in <module>
    from fake_useragent import UserAgent
ModuleNotFoundError: No module named 'fake_useragent'

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文