python 多处理中的动态工作池管理
我想做的是监视系统资源的使用情况并动态增加/减少池中的工作人员数量。
我有一个 24 核节点和 48GB RAM，我所做的就是读取需要过滤的 4GB 数据文件。该系统也被其他人使用，因此可用内存随时间而变化。由于 4GB 输入数据被复制到所有工作人员（我还没有找到一种方法来避免这种情况，因为我只需要在工作人员中只读它，欢迎建议，它是 dicts() 和列表的 dict() ）这意味着我无法生成 24 个工作线程，因为我会立即耗尽内存。所以我想做的是用安全数量的 6 个工作线程启动进程，然后观察内存使用情况并向池中生成额外的工作线程，如果内存使用量变高，则减少工作线程的数量（即允许一些工作线程完成，但不再向这些工作人员生成新任务）。这样我就可以最大限度地利用节点，同时保持 RAM 使用率 <95%。
这样做之所以高效，是因为即使有 6-12 个并行工作线程，整个代码也会运行几个小时，所以如果我可以在一段时间内将工作线程数量增加 20-30%，它会减少总执行时间。
现在，我已经使用 Pool.map() 和 Pool.apply_async() 方法将任务发送到池，因此我没有直接偏好是否一种方法比另一种方法更好。
预先感谢您的建议。
What I'd like to do is monitor how the system resource usage is and dynamically increase/decrease the amount of workers in the pool.
I namely have a 24 core node with 48GB of RAM and what I do is read in a 4GB datafile that I need to filter through. The system is also used by others so the available memory varies with time. As the 4GB input data is replicated to all workers (I've yet to find a way how to avoid that as I only need it in read-only in the workers, recommendations welcome, it's a dict() of dicts() and lists) that means I cannot spawn 24 workers as I'd immediately run out of memory. So what I'd like to do is start the process with say a safe number of 6 workers and then observe memory usage and spawn additional workers to the pool and if the memory usage gets high reduce the number of workers (i.e. allow some to complete, but not spawn new tasks into those workers anymore). This way I could maximally utilize the node while keeping RAM usage <95% say.
The reason this would be efficient is because the whole code runs for a few hours even with 6-12 parallel workers so if I can even for some time increase the number of workers 20-30% it'd reduce the total execution time.
Right now I've used both Pool.map() and Pool.apply_async() methods to send tasks to the pool so I have no direct preference if one method works better than the other.
Thanks in advance for recommendations.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
多处理模块为多个进程提供了一种共享字典的方法：
也许通过使用一个共享的字典，你可以生成 24 个工作线程，并且使用的内存仍然少得多。当一名工作人员访问或更改字典时，其他工作人员如果也尝试访问或更改字典，则会阻塞，但如果这不是主要瓶颈，则能够使用 24 个工作人员可以极大地减少执行时间。
The multiprocessing module provides a way for multiple processes to share a dict:
Perhaps by using a shared dict, you could spawn 24 workers and still use far less memory. When one worker accesses or mutates the dict, other workers will block if they are also attempting to access or mutate the dict, but if this is not the primary bottleneck, being able to use 24 workers could dramatically reduce your execution time.