是否有一个足够智能的并行 make 系统可以智能地响应低内存/交换条件?

发布于 2024-10-31 12:34:33 字数 489 浏览 5 评论 0原文

我非常喜欢使用“make -j8”(当然,用我当前计算机的核心数量替换 8)来加速构建,并且并行编译 N 个文件通常对于减少编译时间非常有效。除非某些编译进程占用的内存足够多,以至于计算机耗尽了 RAM,在这种情况下,所有不同的编译进程都开始相互交换,一切都会变得缓慢——从而违背了并行的目的。首先编译。

现在,这个问题的明显解决方案是“购买更多 RAM”——但由于我的成本太低,无法做到这一点,所以我想到应该可以实现“make”(或等效的)监视系统的可用 RAM,当 RAM 降至接近零并且系统开始交换时,make 将自动介入并向其生成的一个或多个编译进程发送 SIGSTOP。这将允许停止的进程完全换出,以便其他进程可以完成编译而无需进一步交换;然后,当其他进程退出并且有更多 RAM 可用时,“make”进程将向暂停的进程发送 SIGCONT,允许它们恢复自己的处理。这样就可以避免大多数交换,并且我可以在所有内核上安全地进行编译。

有人知道实现这个逻辑的程序吗?或者相反,是否有一些充分的理由为什么这样的程序不能/不能工作?

I'm a big fan of speeding up my builds using "make -j8" (replacing 8 with whatever my current computer's number of cores is, of course), and compiling N files in parallel is usually very effective at reducing compile times... unless some of the compilation processes are sufficiently memory-intensive that the computer runs out of RAM, in which case all the various compile processes start swapping each other out, and everything slows to a crawl -- thus defeating the purpose of doing a parallel compile in the first place.

Now, the obvious solution to this problem is "buy more RAM" -- but since I'm too cheap to do that, it occurs to me that it ought to be possible to have an implementation of 'make' (or equivalent) that watches the system's available RAM, and when RAM gets down to near zero and the system starts swapping, make would automatically step in and send a SIGSTOP to one or more of the compile processes it had spawned. That would allow the stopped processes to get fully swapped out, so that the other processes could finish their compile without further swapping; then, when the other processes exit and more RAM becomes available, the 'make' process would send a SIGCONT to the paused processes, allowing them to resume their own processing. That way most swapping would be avoided, and I could safely compile on all cores.

Is anyone aware of a program that implements this logic? Or conversely, is there some good reason why such a program wouldn't/couldn't work?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

权谋诡计 2024-11-07 12:34:33

对于 GNU Make,有 -l 选项:

   -l [load], --load-average[=load]
        Specifies  that  no  new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-
        point number).  With no argument, removes a previous load limit.

不过,我认为没有标准选项。

For GNU Make, there's the -l option:

   -l [load], --load-average[=load]
        Specifies  that  no  new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-
        point number).  With no argument, removes a previous load limit.

I don't think there's a standard option for this, though.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文