并行制造的最佳硬件/软件解决方案？
我们有一个基于 Linux 的构建系统，其中构建由许多不同的嵌入式目标组成（启用了相应的不同驱动程序和功能集），每个目标都使用另一个主源代码树构建。
我们不想尝试将基于 make 的系统转换为对多进程更加友好的系统，而是只想找到同时为所有这些目标启动构建的最佳方法。 我不确定如何获得最佳性能。
我考虑了以下可能的解决方案：
- 许多单独的构建机器。 缺点：共享代码的大量副本，或从（慢）共享驱动器工作。 更多系统需要维护。
- 数量较少的多处理器机器（也许是双四核），具有快速条带 RAID 本地存储。 缺点：我不确定它将如何扩展。 看起来容量会成为瓶颈，但我不知道现在Linux处理SMP的情况如何。
- 类似的 SMP 机器，但具有虚拟机管理程序或运行 VMware 的 Solaris 10。 这是愚蠢的，还是会提供一些调度优势？ 缺点：没有解决存储瓶颈问题。
我打算坐下来尝试这些可能性，但我想检查一下我是否错过了任何东西。 谢谢！
We've got a Linux-based build system in which a build consists of many different embedded targets (with correspondingly different drivers and featuresets enabled), each one built with another single main source tree.
Rather than try to convert our make-based system to something more multiprocess-friendly, we want to just find the best way to fire off builds for all of these targets simultaneously. What I'm not sure about is how to get the best performance.
I've considered the following possible solutions:
- Lots of individual build machines. Downsides: lots of copies of the shared code, or working from a (slow) shared drive. More systems to maintain.
- A smaller number of multiprocessor machines (dual quadcores, perhaps), with fast striped RAID local storage. Downsides: I'm unsure of how it will scale. It seems that the volume would be the bottleneck, but I don't know how well Linux handles SMP these days.
- A similar SMP machine, but with a hypervisor or Solaris 10 running VMware. Is this silly, or would it provide some scheduling benefits? Downsides: Doesn't address the storage bottleneck issue.
I intend to just sit down and experiment with these possibilities, but I wanted to check to see if I've missed anything. Thanks!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
就软件解决方案而言，我可以推荐 Icecream。 它由 SUSE 维护并构建于 distcc 之上。
我们在我以前的公司中非常成功地使用了它，该公司的构建要求与您所描述的类似。
As far as software solutions go, I can recommend Icecream. It is maintained by SUSE and builds on distcc.
We used it very successfully at my previous company, which had similar build requirements to what you describe.
如果您对快速增量性能感兴趣，那么计算需要重建的文件的成本将主导实际编译时间，这将对机器之间的高效 I/O 提出更高的要求。
但是，如果您最感兴趣的是快速完整重建（例如，每晚构建），那么您可能最好将源代码树同步到每个构建从属设备，甚至让每个构建从属设备从源代码控制中检查自己的副本。 Hudson 等 CI 系统将有助于管理每个从属构建服务器。
If you're interested in fast incremental performance, then the cost of calculating the files which need to be rebuilt will dominate over the actual compile time and this will put higher demands on efficient I/O between the machines.
However, if you're mostly interested in fast full rebuilds (nightly builds, for example), then you may be better of rsyncing out the source tree to each build slave, or even have each build slave check out its own copy from source control. A CI-system such as Hudson would help to manage each of the slave build servers.
如果您的 makefile 足够完整且结构良好，并且您的构建机器有足够的内存，则
-j标志也可能有助于克服 I/O 瓶颈。 这可以让 make 并行运行多个独立任务，这样您的 CPU 理想情况下就不会阻塞等待 I/O。 一般来说，通过允许比机器中的 CPU 数量多的任务，我发现了良好的结果。从你的问题中不清楚你当前的 makefile 是否不适合这个，或者你只是不想跳转到与 make 完全不同的东西。
If your makefiles are sufficiently complete and well-structured, the
-jflag may also be useful in overcoming I/O bottlenecks, if your build machine(s) have enough memory. This lets make run multiple independent tasks in parallel, so that your CPUs will ideally never block waiting on I/O. Generally, I've found good results by allowing several more tasks than I have CPUs in a machine.It's not clear from your question if your current makefiles are not amenable to this, or if you just don't want to jump to something entirely different than make.