我应该如何处理代码中的 celeryd_multi 工作?

发布于 2024-12-03 09:28:19 字数 3379 浏览 0 评论 0原文

到目前为止,我只使用 python manage.py celeryd,像这样启动:

python manage.py celeryd -l info --settings=settings

代码在我看来,这样做是这样的:

BinaryExecTask.delay(request.POST["binary_path"])

我的 settings.py 中的代码是这样的:

import djcelery
djcelery.setup_loader()

BROKER_BACKEND = "djkombu.transport.DatabaseTransport"
#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

它将在后台执行一些二进制文件。问题是,一些二进制文件需要很短的时间来运行,而另一些则可能需要长达半小时。使用 celeryd 时,我的所有任务都会被阻止,直到当前任务完成执行。我看到这里启动 celeryd_multi 的一些示例,但正在运行:

python manage.py celeryd_multi start 3 --settings=settings -l info

给出此错误:

celeryd-multi v2.3.1
> Starting nodes...
        > celery1.x: Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
 line 438, in execute_manager
    utility.execute()
  File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
 line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "c:\code\python27\lib\site-packages\django_celery-2.3.3-py2.7.egg\djcelery\manage
s\celeryd_multi.py", line 22, in run_from_argv
    ["%s %s" % (argv[0], argv[1])] + argv[2:])
  File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
 172, in execute_from_commandline
    self.commands[argv[0]](argv[1:], cmd)
  File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
 205, in start
    retcode = self.waitexec(argv)
  File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
 354, in waitexec
    pipe = Popen(argstr, env=self.env)
  File "c:\code\python27\lib\subprocess.py", line 672, in __init__
    errread, errwrite)
  File "c:\code\python27\lib\subprocess.py", line 882, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

celeryd-multi start 3 -c 3 抛出相同的错误 错误。我应该怎么做才能成功启动一个 celery 实例,让我可以并行运行更多任务?另外,我认为我需要做一些不同的事情吗?

编辑:一些调试引导我来到这里( site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_multi.py(354)waitexec )

351         def waitexec(self, argv, path=sys.executable):
352             args = " ".join([path] + list(argv))
353  ->         argstr = shlex.split(args.encode("utf-8"))
354             pipe = Popen(argstr, env=self.env)
(Pdb) p argstr
['c:codepython27python.exe', 'manage.py', 'celeryd_detach', '-l', 'info', '[email protected]',
 '-n', 'celery1.x', '[email protected]']
(Pdb) p Popen(argstr, env=self.env)
*** WindowsError: WindowsError(2, 'The system cannot find the file specified')
(Pdb)

因此,正如我们所看到的,Python 的路径被破坏了: )。接下来我应该做什么?

EDIT2:我在此处打开了一个问题

So far, I've been working only with python manage.py celeryd, starting it like this:

python manage.py celeryd -l info --settings=settings

The code from my view, does this:

BinaryExecTask.delay(request.POST["binary_path"])

And the code from my settings.py, is this:

import djcelery
djcelery.setup_loader()

BROKER_BACKEND = "djkombu.transport.DatabaseTransport"
#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

and it will execute some binaries in the background. The thing is, some of the binaries take pretty short time to run, while others may take up to half an hour. Working with celeryd, all my tasks are blocked until the current one finishes it's execution. I saw here some examples of starting celeryd_multi, but running:

python manage.py celeryd_multi start 3 --settings=settings -l info

gives this error:

celeryd-multi v2.3.1
> Starting nodes...
        > celery1.x: Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
 line 438, in execute_manager
    utility.execute()
  File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
 line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "c:\code\python27\lib\site-packages\django_celery-2.3.3-py2.7.egg\djcelery\manage
s\celeryd_multi.py", line 22, in run_from_argv
    ["%s %s" % (argv[0], argv[1])] + argv[2:])
  File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
 172, in execute_from_commandline
    self.commands[argv[0]](argv[1:], cmd)
  File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
 205, in start
    retcode = self.waitexec(argv)
  File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
 354, in waitexec
    pipe = Popen(argstr, env=self.env)
  File "c:\code\python27\lib\subprocess.py", line 672, in __init__
    errread, errwrite)
  File "c:\code\python27\lib\subprocess.py", line 882, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

The celeryd-multi start 3 -c 3 throws the same error. What should I do so that I could succesfully start a celery instance that will allow me to run more tasks in parallel? Also, would I need to do something different in my view?

EDIT: some debugging led me here ( site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_multi.py(354)waitexec )

351         def waitexec(self, argv, path=sys.executable):
352             args = " ".join([path] + list(argv))
353  ->         argstr = shlex.split(args.encode("utf-8"))
354             pipe = Popen(argstr, env=self.env)
(Pdb) p argstr
['c:codepython27python.exe', 'manage.py', 'celeryd_detach', '-l', 'info', '[email protected]',
 '-n', 'celery1.x', '[email protected]']
(Pdb) p Popen(argstr, env=self.env)
*** WindowsError: WindowsError(2, 'The system cannot find the file specified')
(Pdb)

So, as we can see, the path to Python gets destroyed :). What should I do next?

EDIT2: I opened an issue here

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

浅语花开 2024-12-10 09:28:19

看起来像是 Windows 特定的...您是否尝试在命令行中提供 python 可执行文件的完整路径,例如

C:\code\python27\bin\python.exe manage.py celeryd_multi start 3 --settings=settings -l info

另外,使用 python 文件的完整路径也很有用

C:\code\python27\bin\python.exe  C:\path\to\your\project\manage.py celeryd_multi start 3 --settings=settings -l info

Looks like smth Windows-specific... Did you try to provide full path to python executable in command line like

C:\code\python27\bin\python.exe manage.py celeryd_multi start 3 --settings=settings -l info

Also, use full paths for python file can be usefull

C:\code\python27\bin\python.exe  C:\path\to\your\project\manage.py celeryd_multi start 3 --settings=settings -l info
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文