需要帮助理解' scrapy'错误信息

发布于 2025-02-10 11:51:32 字数 2282 浏览 2 评论 0原文

最近,我已经完成了一项更新使用网络刮擦程序的网站的任务,但是,我是新的。该程序的先前创建者给我的说明是运行main_executor.py,网站应该毫无问题地运行。但是,事实并非如此,我收到了我不确定的一堆错误消息。我希望有更多经验的人能够看待它,并向我解释这个问题。

错误消息:

Python 3.10.5 (tags/v3.10.5:f377153, Jun  6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.

= RESTART: C:\Users\khoih\OneDrive\Desktop\RA Files\CyberSite\research\research\main_executor.py
Traceback (most recent call last):
  File "C:\Users\khoih\OneDrive\Desktop\RA Files\CyberSite\research\research\main_executor.py", line 4, in <module>
    setting = get_project_settings()
  File "C:\Users\khoih\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\utils\project.py", line 68, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "C:\Users\khoih\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\settings\__init__.py", line 287, in setmodule
    module = import_module(module)
  File "C:\Users\khoih\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'research'

任何帮助都将在APR上进行,如果需要,我可以提供代码。

main_executor.py代码:

from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess

setting = get_project_settings()
process = CrawlerProcess(setting)

for spider_name in process.spiders.list():
    print ("Running spider %s" % (spider_name))
    process.crawl(spider_name,query="dvh") #query dvh is custom argument used in your scrapy

process.start()

I've recently gotten a task to update a website that utilizes a web scraping program, however, I am fairly new to it. The instructions given to me by the prior creator of the program were to run the main_executor.py and the website should run with no problem. However, that wasn't the case I am getting a bunch of error messages that I'm unsure of. I was hoping someone with more experience could look at it and explain to me the problem.

Error message:

Python 3.10.5 (tags/v3.10.5:f377153, Jun  6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.

= RESTART: C:\Users\khoih\OneDrive\Desktop\RA Files\CyberSite\research\research\main_executor.py
Traceback (most recent call last):
  File "C:\Users\khoih\OneDrive\Desktop\RA Files\CyberSite\research\research\main_executor.py", line 4, in <module>
    setting = get_project_settings()
  File "C:\Users\khoih\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\utils\project.py", line 68, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "C:\Users\khoih\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\settings\__init__.py", line 287, in setmodule
    module = import_module(module)
  File "C:\Users\khoih\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'research'

Any help would be aprrcieated and I can provide the code if needed.

The main_executor.py code:

from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess

setting = get_project_settings()
process = CrawlerProcess(setting)

for spider_name in process.spiders.list():
    print ("Running spider %s" % (spider_name))
    process.crawl(spider_name,query="dvh") #query dvh is custom argument used in your scrapy

process.start()

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文