slurm-如何使用不在同一文件夹中的作业创建作业数组

发布于 2025-01-26 03:35:43 字数 1347 浏览 2 评论 0原文

我有一个像这样的文件夹结构：

/home/01/01/script.R
/home/01/02/script.R
/home/01/03/script.R
/home/02/01/script.R
/home/02/02/script.R
/home/02/03/script.R
/home/03/01/script.R
/home/03/02/script.R
/home/03/03/script.R

我想将所有这些脚本共同发送给Slurm作为一个作业数组。但是，我遇到问题，因为它们不在同一文件夹中。我目前知道如何做的是如何将这些脚本发送到slurm作为三个独立的作业阵列 - 其中之一是/home/01，第二个位于/home/02 和第三个AT /HOME/03。我想知道是否有一种简单的方法可以将所有九个作业一起作为数组的一部分发送在一起，没有将它们全部放在同一文件夹中 - 文件夹结构需要严格待在这里。

这是我目前正在使用的脚本，这是不起作用的：

#!/bin/bash
# submit_array.sh

#SBATCH --job-name=array_test
#SBATCH [email protected]
#SBATCH --mail-type=end
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=50                      
#SBATCH --time=0-00:01:00               
#SBATCH --qos=standard

declare -a combinations
index=0
for dataset in `seq -w 01 03`
do
    for chain in `seq -w 01 03`
    do
        combinations[$index]="$dataset $chain"
        index=$((index + 1))

    done
done

parameters=(${combinations[${SLURM_ARRAY_TASK_ID}]})

dataset=${parameters[0]}
chain=${parameters[1]}

module add R

cd /home/$dataset/$chain
R CMD BATCH script.R

任何帮助将不胜感激，谢谢！

原文

I have a folder structure which is like this:

/home/01/01/script.R
/home/01/02/script.R
/home/01/03/script.R
/home/02/01/script.R
/home/02/02/script.R
/home/02/03/script.R
/home/03/01/script.R
/home/03/02/script.R
/home/03/03/script.R

I want to send all of these scripts jointly to the Slurm as one job array. However, I am running into problems because they are not in the same folder. What I currently know how to do is how to send these scripts to Slurm as three separate job arrays - one of which is at /home/01, second one at /home/02 and the third one at /home/03. I was wondering if there was an easy way to send all nine jobs together as a part of the array, WITHOUT putting them all in a same folder - the folder structure needs to strictly stay as is here.

This is the script that I am currently using, which doesn't work:

#!/bin/bash
# submit_array.sh

#SBATCH --job-name=array_test
#SBATCH [email protected]
#SBATCH --mail-type=end
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=50                      
#SBATCH --time=0-00:01:00               
#SBATCH --qos=standard

declare -a combinations
index=0
for dataset in `seq -w 01 03`
do
    for chain in `seq -w 01 03`
    do
        combinations[$index]="$dataset $chain"
        index=$((index + 1))

    done
done

parameters=(${combinations[${SLURM_ARRAY_TASK_ID}]})

dataset=${parameters[0]}
chain=${parameters[1]}

module add R

cd /home/$dataset/$chain
R CMD BATCH script.R

Any help would be appreciated, thanks!

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

停滞 2025-02-02 03:35:43

一种方法是将文件夹组合组合用作Sbatch数组中的单独ID，其关联$ {Slurm_array_task_id}可以通过壳牌剧本中的子主题参数扩展如下：

SBATCH -A 101,102,102,102,201,201,201,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,202,303,303,303,303,303,303,303,303,303,303,303333 ./submit_array.sh

其中submit_array.sh的内容是：

#!/bin/bash
# submit_array.sh

#SBATCH --job-name=array_test
#SBATCH [email protected]
#SBATCH --mail-type=end
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=50                      
#SBATCH --time=0-00:01:00               
#SBATCH --qos=standard

# job arrays do not usually support 4 digits,
# so we append "0" for the dataset variable
dataset="0${SLURM_ARRAY_TASK_ID::1}"
# then chain uses the last two digits
chain=${SLURM_ARRAY_TASK_ID:1:3}

module add R

cd /home/${dataset}/${chain}
R CMD BATCH script.R

One method is to use combine the folder combinations as separate IDs in the sbatch array whose associated ${SLURM_ARRAY_TASK_ID} can be parsed through substring parameter expansion in the shell script as follows:

sbatch -a 101,102,103,201,202,203,301,302,303 ./submit_array.sh

where the contents of submit_array.sh are:

#!/bin/bash
# submit_array.sh

#SBATCH --job-name=array_test
#SBATCH [email protected]
#SBATCH --mail-type=end
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=50                      
#SBATCH --time=0-00:01:00               
#SBATCH --qos=standard

# job arrays do not usually support 4 digits,
# so we append "0" for the dataset variable
dataset="0${SLURM_ARRAY_TASK_ID::1}"
# then chain uses the last two digits
chain=${SLURM_ARRAY_TASK_ID:1:3}

module add R

cd /home/${dataset}/${chain}
R CMD BATCH script.R

回复收藏 0 原文

~没有更多了~