python测试脚本运行到3万多个子线程时出现`thread.error: start new thread`

Question

在公司服务器上运行了一个mosquitto server端，并在服务器上运行如下python脚本创建10万个moquitto_sub客户端（公司以前人写的）连接到服务器:

#!/usr/bin/env python
#coding=utf-8
import os
import threading
from time import ctime,sleep
import subprocess

process = []

def mqtt_sub_test(func):
#    print "mosquitto thread %d" %(func)
#    os.system("mosquitto_sub -h 120.237.96.34 -t rock_test_mosquitto -k 10")
    user = "rock_mosquitto_id-3w-%d" %(func)
    print user

    p = subprocess.Popen(["./mosquitto_sub", "-h", "127.0.0.1", "-t", "llltest", "-k", "10"])
        process.append(p)
    print "process.append %d" %(func)

threads = []

for i in range(1,100000):
    t = threading.Thread(target=mqtt_sub_test,args=(i,))
    threads.append(t)

if __name__ == '__main__':
    for t in threads:
        t.setDaemon(True)
        t.start()
    sleep(0.01)

    raw_input('q')

    for sub in process:
        print sub.pid
        a =sub.kill()
    if (a!= None):
        print a

    print "all over %s" %ctime()

但是当创建子线程数大概达到32300个的时候，程序报错：错误信息如下

 Traceback (most recent call last):
  File "mqtt_python_3w.py", line 45, in <module>
      t.start()
        File "/usr/lib/python2.6/threading.py", line 474, in start
            _start_new_thread(self.__bootstrap, ())
            thread.error: can't start new thread

Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory”

此时运行系统命令诸如ls等，都会出现Cannot allocate memory的错误。
查看下系统内存，16G的系统内存使用了4G，内存是足够的，运行ulimit -a输出信息如下

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

现在想解决的问题如下：

• 可以修改系统那些参数来扩展最大线程数？

• 或者说如何写测试程序

诸位前辈的任何回答将不甚感激，谢谢了

Answer 1

http://stackoverflow.com/a/344292/3278171

Answer 2

能看看subprocess怎么写的嘛？