如何使用所有核心而不是python中的线
您能帮我找出处理以下内容的最佳方法吗?
我想进行多个过程,因此在最短的时间内,我在DB上执行了尽可能多的查询。我已经使用了多处理虚拟,但我认为这使用了线程吗?我似乎无法弄清楚如何使用所有内核,而不仅仅是所有线程?
import mysql.connector
import pandas as pd
from sqlalchemy import create_engine
import multiprocessing.dummy as mp
from multiprocessing import cpu_count
dates_to_extract = ['ff', 'fff']
def do_process(s):
#####SOURCE
try:
#source table conn
engine = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
iconn = engine.connect()
except:
print('connection failure to source')
######DEST
try:
#dest table conn
enginedest = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
iconndest = enginedest.connect()
except:
print('connection failure to destination')
print(s)
#data from source table
try:
data = pd.read_sql('SELECT * FROM ran.order_data', iconn)
except Exception as e:
print('extract: ' + str(e))
#data into destination table
try:
data.to_sql(name='tbjd', con=iconndest, if_exists = 'append')
except Exception as e:
print('insert: ' + str(e))
if __name__=="__main__":
p=mp.Pool(cpu_count())
p.map(do_process, dates_to_extract)
p.close()
p.join()
Can you help me figure out the best way to handle the below.
I want to multi process so I am executing as many queries on the DB as possible, in the shortest possible time. I've used multiprocessing dummy, I think this uses threads though? I can't seem to figure out how to use all cores, rather than just all threads?
import mysql.connector
import pandas as pd
from sqlalchemy import create_engine
import multiprocessing.dummy as mp
from multiprocessing import cpu_count
dates_to_extract = ['ff', 'fff']
def do_process(s):
#####SOURCE
try:
#source table conn
engine = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
iconn = engine.connect()
except:
print('connection failure to source')
######DEST
try:
#dest table conn
enginedest = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
iconndest = enginedest.connect()
except:
print('connection failure to destination')
print(s)
#data from source table
try:
data = pd.read_sql('SELECT * FROM ran.order_data', iconn)
except Exception as e:
print('extract: ' + str(e))
#data into destination table
try:
data.to_sql(name='tbjd', con=iconndest, if_exists = 'append')
except Exception as e:
print('insert: ' + str(e))
if __name__=="__main__":
p=mp.Pool(cpu_count())
p.map(do_process, dates_to_extract)
p.close()
p.join()
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论