我应该如何处理代码中的 celeryd_multi 工作?
到目前为止,我只使用 python manage.py celeryd,像这样启动:
python manage.py celeryd -l info --settings=settings
代码在我看来,这样做是这样的:
BinaryExecTask.delay(request.POST["binary_path"])
我的 settings.py
中的代码是这样的:
import djcelery
djcelery.setup_loader()
BROKER_BACKEND = "djkombu.transport.DatabaseTransport"
#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"
它将在后台执行一些二进制文件。问题是,一些二进制文件需要很短的时间来运行,而另一些则可能需要长达半小时。使用 celeryd 时,我的所有任务都会被阻止,直到当前任务完成执行。我看到这里启动 celeryd_multi 的一些示例,但正在运行:
python manage.py celeryd_multi start 3 --settings=settings -l info
给出此错误:
celeryd-multi v2.3.1
> Starting nodes...
> celery1.x: Traceback (most recent call last):
File "manage.py", line 14, in <module>
execute_manager(settings)
File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
line 438, in execute_manager
utility.execute()
File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "c:\code\python27\lib\site-packages\django_celery-2.3.3-py2.7.egg\djcelery\manage
s\celeryd_multi.py", line 22, in run_from_argv
["%s %s" % (argv[0], argv[1])] + argv[2:])
File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
172, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
205, in start
retcode = self.waitexec(argv)
File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
354, in waitexec
pipe = Popen(argstr, env=self.env)
File "c:\code\python27\lib\subprocess.py", line 672, in __init__
errread, errwrite)
File "c:\code\python27\lib\subprocess.py", line 882, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
celeryd-multi start 3 -c 3
抛出相同的错误 错误。我应该怎么做才能成功启动一个 celery 实例,让我可以并行运行更多任务?另外,我认为我需要做一些不同的事情吗?
编辑:一些调试引导我来到这里( site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_multi.py(354)waitexec )
351 def waitexec(self, argv, path=sys.executable):
352 args = " ".join([path] + list(argv))
353 -> argstr = shlex.split(args.encode("utf-8"))
354 pipe = Popen(argstr, env=self.env)
(Pdb) p argstr
['c:codepython27python.exe', 'manage.py', 'celeryd_detach', '-l', 'info', '[email protected]',
'-n', 'celery1.x', '[email protected]']
(Pdb) p Popen(argstr, env=self.env)
*** WindowsError: WindowsError(2, 'The system cannot find the file specified')
(Pdb)
因此,正如我们所看到的,Python 的路径被破坏了: )。接下来我应该做什么?
EDIT2:我在此处打开了一个问题
So far, I've been working only with python manage.py celeryd
, starting it like this:
python manage.py celeryd -l info --settings=settings
The code from my view, does this:
BinaryExecTask.delay(request.POST["binary_path"])
And the code from my settings.py
, is this:
import djcelery
djcelery.setup_loader()
BROKER_BACKEND = "djkombu.transport.DatabaseTransport"
#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"
and it will execute some binaries in the background. The thing is, some of the binaries take pretty short time to run, while others may take up to half an hour. Working with celeryd
, all my tasks are blocked until the current one finishes it's execution. I saw here some examples of starting celeryd_multi, but running:
python manage.py celeryd_multi start 3 --settings=settings -l info
gives this error:
celeryd-multi v2.3.1
> Starting nodes...
> celery1.x: Traceback (most recent call last):
File "manage.py", line 14, in <module>
execute_manager(settings)
File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
line 438, in execute_manager
utility.execute()
File "c:\code\python27\lib\site-packages\django-1.3-py2.7.egg\django\core\management\_
line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "c:\code\python27\lib\site-packages\django_celery-2.3.3-py2.7.egg\djcelery\manage
s\celeryd_multi.py", line 22, in run_from_argv
["%s %s" % (argv[0], argv[1])] + argv[2:])
File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
172, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
205, in start
retcode = self.waitexec(argv)
File "c:\code\python27\lib\site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_mul
354, in waitexec
pipe = Popen(argstr, env=self.env)
File "c:\code\python27\lib\subprocess.py", line 672, in __init__
errread, errwrite)
File "c:\code\python27\lib\subprocess.py", line 882, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
The celeryd-multi start 3 -c 3
throws the same error. What should I do so that I could succesfully start a celery instance that will allow me to run more tasks in parallel? Also, would I need to do something different in my view?
EDIT: some debugging led me here ( site-packages\celery-2.3.1-py2.7.egg\celery\bin\celeryd_multi.py(354)waitexec )
351 def waitexec(self, argv, path=sys.executable):
352 args = " ".join([path] + list(argv))
353 -> argstr = shlex.split(args.encode("utf-8"))
354 pipe = Popen(argstr, env=self.env)
(Pdb) p argstr
['c:codepython27python.exe', 'manage.py', 'celeryd_detach', '-l', 'info', '[email protected]',
'-n', 'celery1.x', '[email protected]']
(Pdb) p Popen(argstr, env=self.env)
*** WindowsError: WindowsError(2, 'The system cannot find the file specified')
(Pdb)
So, as we can see, the path to Python gets destroyed :). What should I do next?
EDIT2: I opened an issue here
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
看起来像是 Windows 特定的...您是否尝试在命令行中提供 python 可执行文件的完整路径,例如
另外,使用 python 文件的完整路径也很有用
Looks like smth Windows-specific... Did you try to provide full path to python executable in command line like
Also, use full paths for python file can be usefull