Python 中的线程、多进程、Twisted
我可以获得将此代码从线程转换为多进程的帮助吗? 那么任何人都可以帮助使用inf twins 转换此代码。
使用twisted上传数据库会有好处吗
Python 内与外部工具。
import os, pyodbc, sys, threading, Queue
class WorkerThread(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
def run(self):
while 1:
try: # take a job from the queue
type = self.queue.get_nowait()
except Queue.Empty:
raise SystemExit
try:
cxn = pyodbc.connect('DSN=MySQL;PWD=MLML;Option=3')
csr = cxn.cursor()
# Inserts,update, CRUD
except:
# count = count +1
print 'DB Error', type
if __name__ == '__main__':
connections = 25
sml = ('A', 'B', 'C','D',)
# build a queue with tuples
queue = Queue.Queue()
for row in sml:
if not row or row[0] == "#":
continue
queue.put(row)
threads = []
for dummy in range(connections):
t = WorkerThread(queue)
t.start()
threads.append(t)
# wait for all threads to finish
for thread in threads:
thread.join()
sys.stdout.flush()
#csr.close()
#cxn.close()
print 'Finish'
Can I get help converting this code from Threading to Mutliprocess.
Then can anyone help convert this code usinf twisted.
Would there be a gain from using twisted to upload db
within Python vs External tools.
import os, pyodbc, sys, threading, Queue
class WorkerThread(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
def run(self):
while 1:
try: # take a job from the queue
type = self.queue.get_nowait()
except Queue.Empty:
raise SystemExit
try:
cxn = pyodbc.connect('DSN=MySQL;PWD=MLML;Option=3')
csr = cxn.cursor()
# Inserts,update, CRUD
except:
# count = count +1
print 'DB Error', type
if __name__ == '__main__':
connections = 25
sml = ('A', 'B', 'C','D',)
# build a queue with tuples
queue = Queue.Queue()
for row in sml:
if not row or row[0] == "#":
continue
queue.put(row)
threads = []
for dummy in range(connections):
t = WorkerThread(queue)
t.start()
threads.append(t)
# wait for all threads to finish
for thread in threads:
thread.join()
sys.stdout.flush()
#csr.close()
#cxn.close()
print 'Finish'
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
更新:JP提到了txpostgres和txmysql 模块,我不知道。这些允许 Twisted 异步访问两个数据库(不使用线程池)。
请注意,如果您决定使用 Twisted 的企业 adbapi,它将最终使用线程池来处理数据库连接,大致相当于您现有的示例。请参阅有关使用 Twisted 企业数据库模块的文档。
您应该能够直接转换基于线程/队列的代码以使用多处理模块,方法是将线程替换为 Process 实例,并使用多处理 Queue 实现。请参阅多处理文档。
Updated: JP mentioned the txpostgres and txmysql modules, which I wasn't aware of. These allow Twisted to access both databases asynchronously (without using a thread pool).
Note that if you decide to use Twisted's enterprise adbapi, it will end up using a thread pool to handle database connections, roughly the equivalent of your existing example. See the docs on using Twisted's enterprise database module.
You should be able to directly translate your thread/queue-based code to use the multiprocessing module, by replacing your threads with
Process
instances, and uing the multiprocessingQueue
implementation. See the multiprocessing docs.