适用于 Linux 的免费安全分布式制作系统
是否有任何良好的、与语言无关的、安全且免费的 Linux 分布式 make 系统？
背景信息：
我进行的科学实验（计算机科学实验）有时会具有大型依赖树，有时会包含数千或数万个树节点。 该依赖关系树位于数据文件、数据处理可执行文件和结果文件之上。
多年来，我尝试了各种技术，包括：
- 使用数据库滚动我自己的依赖项跟踪器并在每台工作计算机上运行脚本。 这可能会有点麻烦，尤其是在尝试使用非脚本语言时。
- 将所有处理命令放在单个 makefile 中，并使用可以在不同工作计算机上手动“构建”的伪目标。 这不需要特殊的工具，但手动将工作分解为大小均匀的伪目标块并在每个工作框上正确调用“make”可能会很痛苦。
- distmake：自动分配单个 makefile 中命令的执行...
我基本上正在寻找类似 distmake 的东西，但更安全。 据我所知，distmake 本质上在每个工作节点中留下了一个完全开放的后门。
如果替代品比 distmake 更强大，那就太好了。 如果您中断主 distmake 调用，它可以关闭后门服务器，但它不会正确终止工作节点上的执行进程。
说明：
我正在使用 makefile 处理数据，而不是使用 gcc 进行编译和链接。 从我在文档中读到的内容来看，distcc 是一个用于分发 gcc 的专用工具。 我将在共享文件系统上托管的非常大的数据文件上运行我自己的可执行文件，而不是在源文件上运行 gcc，因此 distcc 没有帮助。
工作节点是外部可见的机器，因此我希望任何工作守护进程至少与 ssh 一样安全。 据我所知，无需阅读源代码，distmake 工作守护进程就会打开一个端口，并接受来自连接到该端口的任何人的命令。 他们将以启动守护程序的用户身份执行命令。
Are there any good language-agnostic distributed make systems for linux that are secure and free?
Background Information:
I run scientific experiments (computer-science ones) that sometimes have large dependency trees, occasionally on the order of thousands or tens of thousands of tree nodes. This dependency tree is over the data files, data processing executables, and results files.
I've experimented with various techniques over the years including:
- Rolling my own dependency tracker using a database and running a script on each worker machine. This can get a bit cumbersome, especially when trying to work with non-scripting languages.
- Putting all the processing commands in single makefile, with pseudo-targets that can be manually "built" on different worker machines. This requires no special tools, but it can be a pain to manually break up the work into evenly-sized pseudo-target chunks and correctly invoking "make" on each worker box.
- distmake: automatically distribute the execution of commands from a single makefile...
I'm basically looking for something like distmake, but more secure. As far as I can tell, distmake essentially leaves a wide-open backdoor into each worker node.
It would also be nice if a replacement were more robust than distmake. If you break out of the main distmake call, it can shut down the backdoor servers, but it doesn't properly kill the executing processes on the worker nodes.
Clarifications:
I am processing data with the makefile, not compiling and linking with gcc. From what I read in the documentation, distcc is a specialized tool for distributing gcc. I'll be running my own executables on very large data files hosted on a shared filesystem, not gcc on source files, so distcc isn't helpful.
The worker nodes are externally-visible machines, so I want any worker daemons to be at least as secure as ssh. As best I can tell without reading the source, distmake worker daemons open up a port and will accept commands from anyone who attaches to it. They will execute the commands as the user who started the daemon.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（6）
依赖关系很难管理，而且我不知道有哪个完美的系统可以在不需要大量工作的情况下完成您想要的事情。
我使用过的最接近的是以下设置：
- 用于管理集群中机器的 Condor 队列
- Condor DAGMAN 元调度程序用于提交相互依赖的作业。 DAGMAN 是有向无环图 MANager 的缩写，其中有向无环图用于表示作业之间的依赖关系。
我们在实验室中非常成功地为迭代科学协议完成了这项工作，并且效果很好，尽管对于一位非常有才华的博士后来说，让初始实施运行是一次学习经历。 它确实要求您设置并运行一个 Condor 集群，这并不简单，但我假设您有 Condor 或类似的东西来管理所有机器。 Sun GridEngine 可能有一些我不知道的类似内容。
Dependencies are hard to manage, and I don't know of any perfect system that does what you want without a significant amount of work.
The closest thing that I've used is the following setup:
- a Condor queue to manage the machines in your cluster
- the Condor DAGMAN meta-scheduler to submit jobs that are interdependent. DAGMAN is an acronym for Directed Acyclic Graph MANager, in which a directed acyclic graph is used to represent the dependencies between your jobs.
We've done this for an iterative scientific protocol in our lab very successfully and it's worked great, although it was a learning experience for a very talented postdoc to get the initial implementation running. It does require that you set up and run a Condor cluster which is non-trivial, but I assume you have either Condor or something similar to manage all of your machines. It might be that Sun GridEngine has something analogous that I don't know about.
还有 distcc，它声称能够通过 SSH 进行操作（尽管除非 distmake 非常奇怪，否则你应该能够限制对 localhost 的访问并构建 SSH 隧道来运行构建）和 icecream。
更新：因为目标不是分布式编译，而是恰好使用
make作为引导程序的分布式计算，所以使用以下工具更有意义：专为分布式计算而设计，例如 BOINC。 下面的评论表明 condor 作为所选平台。There are also distcc, which claims to be able to operate over SSH (though unless distmake is somehow very strange, you should be able to restrict access to localhost and build SSH tunnels to run the build), and icecream.
Update: Because the goal isn't a distributed compile, but a distributed computation that just happens to be using
makeas a bootstrap, it makes more sense to use a tool that is designed for distributed computation like BOINC. The comments below indicate condor as the chosen platform.尽管与“make”透明地集成可能会很复杂，但 GNU 并行似乎提供了一个方便的选项来跨服务器分发命令。
Although it probably will be complicated to integrate transparently with 'make', GNU parallel seems to provide a convenient option to distribute commands across servers.
如果您勤于处理依赖项（即 make -jxx 在本地工作正常），distcc 可能就是您想要的。 它非常易于使用，并且可以与多种流行的 CC 缓存配合使用。 同样，正确的依赖关系是关键，尤其是在使用缓存来帮助加快重建过程时。
如果您使用 GCC 生成超出 makefile 本身模块依赖关系范围的依赖关系，您可能会喜欢 distcc。 我一直在一个小型构建农场上使用它并取得了巨大成功..但我的设置/树远没有您描述的那么复杂。
If you're diligent with dependencies (i.e. make -jxx works fine locally), distcc is probably what you want. Its very easy to utilize and works happily with several popular CC caches. Again, proper dependencies being the key, especially when using a cache to help speed up the process of re-building.
If your using GCC to generate dependencies beyond the scope of module dependencies in the makefile itself, you'll probably love distcc. I've been using it on a small build farm with great success .. but my setup/tree is nowhere near as elaborate as the one you describe.
您可以使用 AT&T
nmake结合使用来完成此操作coshell程序。 我不知道如何评估安全性，但 Glenn Fowler 的团队充满了伟大的工程师，他们做了很多非常好的事情。 我会把我的源代码交给他们:-) 他们最著名的工具可能是 graphviz。You can do this with AT&T
nmakecombined with thecoshellprogram. I don't know how to evaluate security, but Glenn Fowler's group is full of great engineers who have done a lot of seriously good stuff. I would trust them with my source code :-) Their best known tool might be graphviz.Makeflow 似乎也是一个很好的解决方案： http://www.cse.nd.edu /~ccl/software/makeflow/
Makeflow also seems like an excellent solution: http://www.cse.nd.edu/~ccl/software/makeflow/