当线程数超过 32705 时 boost::thread_resource_error
我正在实现一个消息传递算法。消息通过图的节点传播，阻塞直到它们收到足够的信息（来自其他邻居）来发送消息。
如果我将每条消息放入其自己的线程中并使用 boost::condition 来暂停线程，直到所有必需的信息可用，则该算法很容易编写。我创建了数千个线程，但大多数时候只有少数线程处于活动状态。这似乎运作得很好。
我的问题是，当单元测试时，我发现如果我创建超过 32705 个线程，我会得到
未知位置(0)：致命错误 “Tree_test”：std::异常： boost::thread_resource_error
我不知道是什么原因导致这个问题，也不知道如何修复它。
似乎有足够的可用内存（每个线程只保存两个指针 - 消息在它们之间传递的对象）。
来自这个问题： Linux 中每个进程的最大线程数？ 我认为以下信息是相关的（尽管我真的不知道它意味着什么......）
~> cat /proc/sys/kernel/threads-max
1000000
（我从 60120 增加了这个 - 我需要重新启动吗？）
~>ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
我尝试摆弄挂起的信号（我的限制是非常接近 2* 该数字） 和堆栈大小 ulimit -S -i 8191 - （我无法增加它）但这些更改似乎根本没有效果）
我使用的是 64 位 Ubuntu-10-10 如果这有帮助...
I'm implementing a message passing algorithm. The messages propagate through the nodes of the graph, blocking until they have have received enough information (from other neighbours) to send a message.
The algorithm is easy to write if I put each message in its own thread and use a boost::condition to pause the thread until all the required information is available. I create many thousands of threads, but mostly only a few are active at any time. This seems to work pretty well.
My problem is, when unit testing I find that if I create more than about 32705 threads, I get
unknown location(0): fatal error in
"Tree_test": std::exception:
boost::thread_resource_error
and I don't know what causes this, or how to fix it.
There seems to be pleanty of memory available (Each thread only holds two pointers - the objects that the message passes between).
From this question: Maximum number of threads per process in Linux? I think the following information is relevent (although I don't really know what any of it means...)
~> cat /proc/sys/kernel/threads-max
1000000
(I increased this from 60120 - do I need to restart?)
~>ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I tried fiddling with the pending signals (my limit is very close to 2* that number)
and stack size with ulimit -S -i 8191 - (I couldn't increase it) but these changes seemed to make no effect at all)
I'm on a 64 bit Ubuntu-10-10 if that helps...
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
我认为对于系统上的 32K 线程，您应该考虑潜在的解决方案，而不是如何拥有更多线程。例如，线程池（Boost 有一些用于此的东西）。
无论如何，在您的系统上，PID 不限制为 32768 或类似的值吗？我想，你迟早会用完的，不妨设计系统以允许处理比最大线程数更多的项目。
也就是说，查看 /proc/sys/kernel/pid_max 来查看你的最大 PID——并尝试增加它。这可能会让您超过 32K（但也可能会导致不是为异常大的 PID 设计的程序出现意外行为，因此要小心）。
然后您可能会受到堆栈空间（而不是虚拟内存空间）的限制。如果您愿意，您可以尝试创建具有较小堆栈的线程。
I think with 32K threads on the system, you should look at potential solutions other than how to have more threads. For example, a thread pool (Boost has some things for this).
Anyway, on your system, aren't PIDs limited to 32768 or some such value? You're going to run out sooner or later, may as well design the system to allow processing more items than the max number of threads, I'd think.
That said, look at /proc/sys/kernel/pid_max to see your max PID--and try increasing it. This may get you beyond 32K (but may also cause unexpected behavior with programs not designed for unusually large PIDs, so be cautious).
And then you may be limited by the stack space (as opposed to virtual memory space). You could try creating threads with smaller stacks if you like.
好的，回答这个问题：你需要增加
正如这里所讨论的：
https://listman.redhat.com/archives/phil-list/2003-August/msg00025.html
和此处：
http://www.kegel.com/c10k.html#limits.threads
< strong>但是：要获得更好的方法，请查看后续问题：
等待条件的非线程替代方案。 （编辑：前摄器模式与 boost.asio？）
Okay, to answer the question: you need to increase
As discussed here:
https://listman.redhat.com/archives/phil-list/2003-August/msg00025.html
and here:
http://www.kegel.com/c10k.html#limits.threads
HOWEVER: FOR BETTER WAYS TO DO THIS LOOK AT THE FOLLOW UP QUESTION:
Non-threaded alternative to waiting on a condition. (Edit: Proactor pattern with boost.asio?)
这实际上取决于您的堆栈有多大，但如果您创建大量线程，您将耗尽地址空间（32 位）或虚拟内存（64 位）。
在我上次检查时，Linux pthreads 中的默认堆栈大小为 10Mb；这意味着 32k 线程使用 320G 的地址空间（注意它可能会被延迟初始化，所以它不会使用那么多的虚拟内存）；这可能太多了。
即使您将堆栈设置得很小并且不会以这种方式耗尽内存，32k 线程也会使用大量堆栈虚拟内存。考虑使用不同的方法。
ulimit 仅影响初始线程的堆栈大小（在 Linux 下通常是动态的）；其他线程的堆栈大小是固定的，并由 pthread 库在线程创建时设置。
It really depends on how big your stacks are, but you're going to run out of address-space (32-bit) or virtual memory (64-bit) if you create a lot of threads.
In Linux pthreads the default stack size was 10Mb last time I checked; this means that 32k threads uses 320G of address space (note it will probably be lazily initialised, so it won't use that much virtual memory); this is probably too much.
Even if you make the stack quite small and don't exhaust the memory this way, 32k threads is going to use a lot of virtual memory for stacks. Consider using a different approach.
ulimit only affects the stack size of the initial thread (which is dynamic normally under Linux); other threads' stack size is fixed and set at thread creation time by the pthread library.