在 python 中实现基本的队列/线程进程

Question

寻找一些眼球来验证下面的伪 python 块是否有意义。我希望生成多个线程来尽快实现一些 inproc 函数。这个想法是在主循环中生成线程，因此应用程序将以并行/并发的方式同时运行线程

chunk of code
 -get the filenames from a dir
 -write each filename ot a queue
 -spawn a thread for each filename, where each thread 
  waits/reads value/data from the queue
 -the threadParse function then handles the actual processing 
  based on the file that's included via the "execfile" function...


# System modules
from Queue import Queue
from threading import Thread
import time

# Local modules
#import feedparser

# Set up some global variables
appqueue = Queue()

# more than the app will need
# this matches the number of files that will ever be in the 
# urldir
#
num_fetch_threads = 200


def threadParse(q)
  #decompose the packet to get the various elements
  line = q.get()
  college,level,packet=decompose (line)

  #build name of included file
  fname=college+"_"+level+"_Parse.py"
  execfile(fname)
  q.task_done()


#setup the master loop
while True
  time.sleep(2)
  # get the files from the dir
  # setup threads
  filelist="ls /urldir"
  if filelist
    foreach file_ in filelist:
        worker = Thread(target=threadParse, args=(appqueue,))
        worker.start()

    # again, get the files from the dir
    #setup the queue
    filelist="ls /urldir"
    foreach file_ in filelist:
       #stuff the filename in the queue
       appqueue.put(file_)


    # Now wait for the queue to be empty, indicating that we have
    # processed all of the downloads.

  #don't care about this part

  #print '*** Main thread waiting'
  #appqueue.join()
  #print '*** Done'

感谢想法/评论/指针...

谢谢

Answer 1

如果我理解正确的话：您会产生大量线程来更快地完成工作。

仅当每个线程中完成的工作的主要部分在不持有 GIL 的情况下完成时，这才有效。因此，如果有大量等待来自网络、磁盘或类似数据的数据，这可能是一个好主意。
如果每个任务都使用大量 CPU，那么这将非常类似于在单核 1-CPU 计算机上运行，​​您也可以按顺序执行它们。

我应该补充一点，我所写的内容对于 CPython 是正确的，但不一定适用于 Jython/IronPython。
另外，我应该补充一点，如果您需要利用更多 CPU/核心，可以使用 多处理 可能有帮助的模块。