是否有一种干净的方法可以在单个节点上运行NextFlow(基于DSL2)的子工作流?

发布于 2025-01-26 05:39:53 字数 1175 浏览 4 评论 0原文

我在子工作流中定义了一个管道:

workflow pipeline{
    main:
            task1()
            task2(task1.out)
            ...

}

由主工作流来调用不同输入:

workflow{
    params.in = "$baseDir/input"

    input_ch = Channel.fromPath(params.in + "/*.*")
    main:
            pipeline(input_ch)

}

它易于添加标签以处理以定义其资源。我想做的就是将这样的标签添加到我的(整个)子Workflow中。他们的标签应该迫使子处理程序执行为一个单个过程(这意味着它不会调用所有进程的新slurm调用,而仅调用每个进程的呼叫),例如:

                - pipeline for input 1 on node 1 as one slurmjob
                
                - pipeline for input 2 on node 2 as one slurmjob
- main:                                                                 - collect all the outputs
                        ...

                - pipeline for input n on node n as one slurmjob

我是通过在nextflow内部调用nextflow来做到的:

process nextflowofnextflows {
    label 'use_whatever_is_needed'


    input:
    file input1

    output:
    path("$out/*.*", emit: output)

    
    script:
    """
    nextflow run pipline.nf --in=${input1}

    """
}

...-但这是超级不洁的,我想以一种干净的方式对其进行处理。

实现所描述行为的最佳实践是什么?

最好,t。

i have a pipeline defined in a sub-workflow:

workflow pipeline{
    main:
            task1()
            task2(task1.out)
            ...

}

that gets called by the main-workflow for different inputs:

workflow{
    params.in = "$baseDir/input"

    input_ch = Channel.fromPath(params.in + "/*.*")
    main:
            pipeline(input_ch)

}

its easy to add a label to process to define its resources. what i want to do is add such a label to my (whole) subworkflow. they label is supposed to force the subworkflow to execute as a if it was a single process (meaning that it does not invoke a new slurm call for all processes but only per subworkflow) like:

                - pipeline for input 1 on node 1 as one slurmjob
                
                - pipeline for input 2 on node 2 as one slurmjob
- main:                                                                 - collect all the outputs
                        ...

                - pipeline for input n on node n as one slurmjob

i did as above by calling nextflow inside of nextflow:

process nextflowofnextflows {
    label 'use_whatever_is_needed'


    input:
    file input1

    output:
    path("$out/*.*", emit: output)

    
    script:
    """
    nextflow run pipline.nf --in=${input1}

    """
}

... - but this is super unclean and i´d like to to it in a clean way.

whats the best practice to achieve the described behavior?

best, t.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文