PYBIND11:多线程崩溃

发布于 2025-02-06 02:50:01 字数 2576 浏览 0 评论 0原文

我有一个名为l的项目列表,还有一个复杂的python函数,称为func;正常的方法是使用Python-loop:

out = [func(item) for item in L]

但是它是单线线程,因此我想在C ++中实现一个函数,并使用Pybind11绑定:

对于CPP:

m.def("test_func_iter", [](const py::object &func, const py::sequence &iter) {
        auto n = len(iter);
        py::list l(n);
        unsigned int k = std::thread::hardware_concurrency();
        
        std::thread threads[k];

        auto stride = n / k;

        // [0, n//k), [n//k, ...), [...,n)
        for (unsigned int w = 0; w < k; ++w) {
            if (w < k - 1) {
                threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
                    for (size_t i = start; i < end; ++i) {
                        std::cout << "h: "<< i << std::endl;
                        l[i] = func(iter[i]);
                    }
                }, w * stride, (w + 1) * stride);
            } else {
                threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
                    for (size_t i = start; i < end; ++i) {
                        std::cout << "h: "<< i << std::endl;
                        l[i] = func(iter[i]);
                    }
                }, w * stride, n);
            }
        }
        std::cout << "Done spawning threads! Now wait for them to finish.\n";
        for (auto& t: threads) {
            t.join();
        }

        std::cout << "end" << std::endl;
       
        return py::type::of(iter)(l);

并且当我调用Python中的相应绑定函数时


def func(i):
    # just simplify the actual logic, a sophisticated function that is hard to re-write totally in c++
    print(i, i == 0)
    return int(gmpy2.mpz(i) + 100)


b = test_func_iter(func, list(range(100)))
print(b)

,获取输出和错误,例如:

h: h: Done spawning threads! Now wait for them to finish.
050

0 True

进程已结束,退出代码为 139 (interrupted by signal 11: SIGSEGV)

我做了一些尝试:

  1. 不使用线程:Python
  2. 中的一切都可以使用thread&amp; k = 1 :只需使用一个线程,python
  3. 使用thread&amp; k&gt; = 2 :崩溃。

顺便说一句,我使用Mac M1笔记本电脑,Clang的版本为12.05;

我是C ++的新手,猜猜原因可能是 thread 的使用,但是在Google中找不到一些建议,任何人都可以提示吗?(或一些有关起源问题的建议: pybind11 的优雅方式用于多线程支持)谢谢!

I have a list of item called L, and a sophisticated python function called func; the normal way is to use python-loop like:

out = [func(item) for item in L]

But it's single-thread, so I want to implement a function in c++, and bind with pybind11:

For cpp:

m.def("test_func_iter", [](const py::object &func, const py::sequence &iter) {
        auto n = len(iter);
        py::list l(n);
        unsigned int k = std::thread::hardware_concurrency();
        
        std::thread threads[k];

        auto stride = n / k;

        // [0, n//k), [n//k, ...), [...,n)
        for (unsigned int w = 0; w < k; ++w) {
            if (w < k - 1) {
                threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
                    for (size_t i = start; i < end; ++i) {
                        std::cout << "h: "<< i << std::endl;
                        l[i] = func(iter[i]);
                    }
                }, w * stride, (w + 1) * stride);
            } else {
                threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
                    for (size_t i = start; i < end; ++i) {
                        std::cout << "h: "<< i << std::endl;
                        l[i] = func(iter[i]);
                    }
                }, w * stride, n);
            }
        }
        std::cout << "Done spawning threads! Now wait for them to finish.\n";
        for (auto& t: threads) {
            t.join();
        }

        std::cout << "end" << std::endl;
       
        return py::type::of(iter)(l);

And when I invoke the corresponding bind function in python, like:


def func(i):
    # just simplify the actual logic, a sophisticated function that is hard to re-write totally in c++
    print(i, i == 0)
    return int(gmpy2.mpz(i) + 100)


b = test_func_iter(func, list(range(100)))
print(b)

And I get the output and error like:

h: h: Done spawning threads! Now wait for them to finish.
050

0 True

进程已结束,退出代码为 139 (interrupted by signal 11: SIGSEGV)

I have done some tries:

  1. Not use thread : everything is OK in python
  2. Use thread & k=1: just use one single thread, everything is OK in python
  3. Use thread & k>=2: crash.

BTW, I use Mac M1 laptop, and version of clang is 12.05 ;

I am new to c++, and guess the reason may be the use of thread, but can not find some suggestions in google, can anybody give some hints?(Or some suggestions about the origin problem: elegant way for multi-thread support with pybind11) Thanks!

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文