PYBIND11:多线程崩溃
我有一个名为l
的项目列表,还有一个复杂的python函数,称为func
;正常的方法是使用Python-loop:
out = [func(item) for item in L]
但是它是单线线程,因此我想在C ++中实现一个函数,并使用Pybind11绑定:
对于CPP:
m.def("test_func_iter", [](const py::object &func, const py::sequence &iter) {
auto n = len(iter);
py::list l(n);
unsigned int k = std::thread::hardware_concurrency();
std::thread threads[k];
auto stride = n / k;
// [0, n//k), [n//k, ...), [...,n)
for (unsigned int w = 0; w < k; ++w) {
if (w < k - 1) {
threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
for (size_t i = start; i < end; ++i) {
std::cout << "h: "<< i << std::endl;
l[i] = func(iter[i]);
}
}, w * stride, (w + 1) * stride);
} else {
threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
for (size_t i = start; i < end; ++i) {
std::cout << "h: "<< i << std::endl;
l[i] = func(iter[i]);
}
}, w * stride, n);
}
}
std::cout << "Done spawning threads! Now wait for them to finish.\n";
for (auto& t: threads) {
t.join();
}
std::cout << "end" << std::endl;
return py::type::of(iter)(l);
并且当我调用Python中的相应绑定函数时
def func(i):
# just simplify the actual logic, a sophisticated function that is hard to re-write totally in c++
print(i, i == 0)
return int(gmpy2.mpz(i) + 100)
b = test_func_iter(func, list(range(100)))
print(b)
,获取输出和错误,例如:
h: h: Done spawning threads! Now wait for them to finish.
050
0 True
进程已结束,退出代码为 139 (interrupted by signal 11: SIGSEGV)
我做了一些尝试:
- 不使用线程:Python
- 中的一切都可以使用thread&amp; k = 1 :只需使用一个线程,python
- 使用thread&amp; k&gt; = 2 :崩溃。
顺便说一句,我使用Mac M1笔记本电脑,Clang的版本为12.05;
我是C ++的新手,猜猜原因可能是 thread 的使用,但是在Google中找不到一些建议,任何人都可以提示吗?(或一些有关起源问题的建议: pybind11 的优雅方式用于多线程支持)谢谢!
I have a list of item called L
, and a sophisticated python function called func
; the normal way is to use python-loop like:
out = [func(item) for item in L]
But it's single-thread, so I want to implement a function in c++, and bind with pybind11:
For cpp:
m.def("test_func_iter", [](const py::object &func, const py::sequence &iter) {
auto n = len(iter);
py::list l(n);
unsigned int k = std::thread::hardware_concurrency();
std::thread threads[k];
auto stride = n / k;
// [0, n//k), [n//k, ...), [...,n)
for (unsigned int w = 0; w < k; ++w) {
if (w < k - 1) {
threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
for (size_t i = start; i < end; ++i) {
std::cout << "h: "<< i << std::endl;
l[i] = func(iter[i]);
}
}, w * stride, (w + 1) * stride);
} else {
threads[w] = std::thread([&l, &func, &iter](size_t start, size_t end) {
for (size_t i = start; i < end; ++i) {
std::cout << "h: "<< i << std::endl;
l[i] = func(iter[i]);
}
}, w * stride, n);
}
}
std::cout << "Done spawning threads! Now wait for them to finish.\n";
for (auto& t: threads) {
t.join();
}
std::cout << "end" << std::endl;
return py::type::of(iter)(l);
And when I invoke the corresponding bind function in python, like:
def func(i):
# just simplify the actual logic, a sophisticated function that is hard to re-write totally in c++
print(i, i == 0)
return int(gmpy2.mpz(i) + 100)
b = test_func_iter(func, list(range(100)))
print(b)
And I get the output and error like:
h: h: Done spawning threads! Now wait for them to finish.
050
0 True
进程已结束,退出代码为 139 (interrupted by signal 11: SIGSEGV)
I have done some tries:
- Not use thread : everything is OK in python
- Use thread & k=1: just use one single thread, everything is OK in python
- Use thread & k>=2: crash.
BTW, I use Mac M1 laptop, and version of clang is 12.05 ;
I am new to c++, and guess the reason may be the use of thread, but can not find some suggestions in google, can anybody give some hints?(Or some suggestions about the origin problem: elegant way for multi-thread support with pybind11) Thanks!
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论