使用slurm运行python文件的正确方法是什么？
我使用的HPC要求我使用Slurm在计算上运行程序。目前，我在笔记本电脑上本地运行一个Python脚本，该脚本在循环中调用了一堆bash命令。例如，Python文件具有一个代码，
command = "docker run -v %s:/usr/local/share/foo/ xyzw mpirun -n %d abcd %d /usr/local/share/foo/%s /usr/local/share/foo/%s" % (directory, numberOfCores, binaryPrecision, inFile, outFile)
command = shlex.split(command)
subprocess.call(command)
在这里看起来像xyzw是Docker容器和abcd我使用MPI运行的程序。现在，我想在HPC上做类似的事情，但使用奇异性和诽谤。我的困惑是以下内容。假设我将上面的python文件脚本调用。然后我有两个选择。我要么将命令保存在文件中 命令=“ singularity exec -v ...”并使用srun python3 script.py或我将文件中的命令更改为command = = “ srun Singularity exec -v ...”并从登录节点以python3 script.py运行python文件。
两者中哪一个是做事的合法方式？我最担心的是，假设我选择srun python3 script.py的选择。然后，我分配给脚本的资源会有效地使用脚本，还是将奇异性内容降低到单个核心，从而使整个练习徒劳无功？
The HPC I am using requires me to run programs on the compute using Slurm. Currently I am locally running a Python script on my laptop that calls a bunch of bash commands in a loop. As an example, the Python file has a piece of code that looks like
command = "docker run -v %s:/usr/local/share/foo/ xyzw mpirun -n %d abcd %d /usr/local/share/foo/%s /usr/local/share/foo/%s" % (directory, numberOfCores, binaryPrecision, inFile, outFile)
command = shlex.split(command)
subprocess.call(command)
Here xyzw is the Docker container and abcd the program that I run using MPI. Now I would like to do something similar on the HPC but using Singularity and Slurm. My confusion is the following. Let's say I call the above Python file script.py. Then I have two options. I either keep the command inside the file ascommand = "singularity exec -v ..." and run the Python file using srun python3 script.py or I change the command inside the file to command = "srun singularity exec -v ..." and run the Python file from the login node as python3 script.py.
Which of the two is the legal way of doing things? What I am worried the most about is that let's say I go with the option of srun python3 script.py. Then will the resources I allocate to the script be utilised efficiently or is it going to relegate the singularity stuff to a single core, hence making the whole exercise futile?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
除非另有专门配置，否则以奇异性运行的过程可以完全访问主机OS的硬件资源。使用这些资源的使用方式取决于容器内部运行的任何应用程序。
MPI是它自己的特殊野兽，因此我强烈建议通过有关如何使用它的奇异性文档阅读。它还直接解决了如何与slurm一起使用。
Processes run in singularity, unless specifically configured otherwise, have full access to the hardware resources of the host OS. How those resources are used is determined by whatever application is running inside the container.
MPI is its own special beast, so I strongly suggest reading through the Singularity docs on how to use it. It also directly addresses how to use it with Slurm.