python 并发 维持tqdm进度条

Question

pool = multiprocessing.Pool(multiprocessing.cpu_count())
items = [......]
for item in items:
    pool.apply_async(func, args=item)
pool.join()

pool必须阻塞主线程, 我希望每个func运行完成后可以tqdm.update(1), 如何实现? 谢谢

Answer 1

在 callback 及 error_callback 中调用 tqdm.update(1) 即可。

示例

import os
import time
import multiprocessing

from tqdm import trange


def foo(_):
    time.sleep(0.1)


if __name__ == '__main__':
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    items = range(100)
    p = trange(len(items))
    results = []
    for item in items:
        result = pool.apply_async(
            foo,
            args=(item,),
            callback=lambda _: p.update(1),
            error_callback=lambda _: p.update(1),
        )
        results.append(result)
    [i.get() for i in results]
    pool.close()
    pool.join()