在 Docker 上使用 Ray 多重处理
我正在尝试在 Docker 容器中使用 Ray 并行化进程。
from ray.util.multiprocessing import Pool
with Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
虽然它在本地工作得很好，但当它在 docker 容器中运行时，会出现以下错误：
✗ failed cod_simulation-run 作业“cod_simulation-run”失败： | AttributeError：“StreamCapture”对象没有属性“fileno” 编写“details cod_simulation-run”来检查错误。 已处理 1 个作业（1 个失败）。
我之前使用 python 多处理执行相同的操作：
import multiprocessing as mp
with mp.Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
这在本地和 docker 容器中都有效。但出于效率原因，我宁愿坚持使用 Ray。
FROM python:3.9
WORKDIR /project_name
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN find .
ENV DISABLE_CONTRACTS=1
RUN pipdeptree
RUN python setup.py develop --no-deps
RUN cod-demo --help
CMD ["cod-demo"]
这是我的 DockerFile，我根据要求安装 ray。
预先感谢您的任何建议
I am trying to parallelize processes using Ray in a docker container.
from ray.util.multiprocessing import Pool
with Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
while it is perfectly working locally, when it gets run in the docker container, the following error occurs:
✗ failed cod_simulation-run
Job 'cod_simulation-run' failed:
| AttributeError: 'StreamCapture' object has no attribute 'fileno'
Write "details cod_simulation-run" to inspect the error.
Processed 1 jobs (1 failed).
I was previously performing the same thing with python multiprocessing:
import multiprocessing as mp
with mp.Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
and this worked both locally and in the docker container. But for efficiency reasons, I would prefer to stick to Ray.
FROM python:3.9
WORKDIR /project_name
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN find .
ENV DISABLE_CONTRACTS=1
RUN pipdeptree
RUN python setup.py develop --no-deps
RUN cod-demo --help
CMD ["cod-demo"]
This is my DockerFile and I am installing ray as a requirement.
Thanks in advance for any suggestion
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
根据您的需求从
上的他们的存储库中提取适当的标签后dockerhub。您可以简单地使用以下命令运行测试：或
如果您使用的是 GPU。如果您需要一些未预先安装在映像上的其他软件包，您必须通过映像内的终端安装它们，或者使用下载的映像作为基础映像制作一个新的 docker 文件（
FROM命令参数）。根据此处提到的内容！
After pulling the appropriate tag based on your needs from their repo on
dockerhub. You can simply run tests with:or
In case you're using GPU. If you need some additional packages which aren't pre-installed on images you must install them through the terminal inside the image or fabricate a new docker-file with the downloaded image as the base image (the
FROMcommand argument).According to what was mentioned here!