为什么 `pathlib.Path("xxx/yyy").unlink / mkdir / rmdir` 不是同步操作？

发布于 2025-01-19 20:21:14 字数 1213 浏览 1 评论 0 原文

我正在使用Pytorch的 distributeDataparallel 一起使用Python的 pathlib.path.path 模块。

当我使用distribationDataParallel使用多进程时，我将删除或创建一个使用 path（“ xxx / yyy”）的文件。RMDIR / .UNLINK / .MKDIR < / code>仅在过程0中，该文件仅在 local_rank：local_rank：local_rank： 0 。

然后发生了奇怪的事情，看来 path（“ xxx / yyy”）。rmdir / .unlink / .mkdir < / code < / code < / code>操作是同步的，我的意思是 rank0中的过程运行，但不会等到文件操作结束。因此，如果该操作之后有文件检查，例如 path（“ xxx/yyy”）。parent.iterdir ， rank0 和 rank1中的结果/code>或其他等级不等，这意味着 rank0 找到或找不到其他等级找不到或找到的某些文件。

该问题已通过添加同步锁定解决，代码很简单，如下所示：

from time import sleep

def wait_to_success(fn, no=False):
    while True:
        sleep(0.01)
        if (no and not fn()) or (not no and fn()):
            break

此功能如下：

remove_file_operation(filepath)
wait_to_success(filepath.exist, no=True)

或者

create_file_operation(filepath)
wait_to_success(filepath.exist, no=False)

需要注意 torch.distribed.barrier.barrier（） 功能不适用于此问题。因此，我很困惑为什么 pathlib.path 操作不等待此操作结束？

原文

I am using python's pathlib.Path module, with pytorch's DistributedDataParallel together.

When I use multi-process using DistributedDataParallel, I delete or create a file with Path("xxx/yyy").rmdir / .unlink / .mkdir only in process 0 which is local_rank: 0.

And then the weird thing happened，it seems that the Path("xxx/yyy").rmdir / .unlink / .mkdir operations is not synchronous, which i means that the process in rank0 runs but does not wait until the file opertion ends. So if there is a file check after that operation, for example Path("xxx/yyy").parent.iterdir, the results in rank0 and rank1 or other ranks are not equal, which means that rank0 finds or does not find some files that other ranks does not find or find.

The problem has been fixed by adding a synchronous lock, the code is simple and as follows:

from time import sleep

def wait_to_success(fn, no=False):
    while True:
        sleep(0.01)
        if (no and not fn()) or (not no and fn()):
            break

And this function is used as follows:

remove_file_operation(filepath)
wait_to_success(filepath.exist, no=True)

create_file_operation(filepath)
wait_to_success(filepath.exist, no=False)

And it is needed to notice that the torch.distributed.barrier() function does not work for this problem.
So I am very confused why pathlib.Path operation does not wait for this operation ends?