如何使用所有核心而不是python中的线

发布于 2025-02-06 20:00:02 字数 1321 浏览 0 评论 0原文

您能帮我找出处理以下内容的最佳方法吗?

我想进行多个过程,因此在最短的时间内,我在DB上执行了尽可能多的查询。我已经使用了多处理虚拟,但我认为这使用了线程吗?我似乎无法弄清楚如何使用所有内核,而不仅仅是所有线程?

import mysql.connector
import pandas as pd
from sqlalchemy import create_engine
import multiprocessing.dummy as mp
from multiprocessing import cpu_count

dates_to_extract = ['ff', 'fff']

def do_process(s):
    #####SOURCE
    try:
        #source table conn
        engine = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
        iconn = engine.connect()
    except:
        print('connection failure to source')

    ######DEST
    try:
        #dest table conn
        enginedest = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
        iconndest = enginedest.connect()
    except:
        print('connection failure to destination')


    print(s)
    #data from source table
    try:  
        data = pd.read_sql('SELECT * FROM ran.order_data', iconn)
    except Exception as e: 
        print('extract: ' + str(e))
          
    #data into destination table
    try:
        data.to_sql(name='tbjd', con=iconndest, if_exists = 'append')
    except Exception as e:
        print('insert: ' + str(e))

if __name__=="__main__":
    p=mp.Pool(cpu_count())
    p.map(do_process, dates_to_extract) 
    p.close()
    p.join()

Can you help me figure out the best way to handle the below.

I want to multi process so I am executing as many queries on the DB as possible, in the shortest possible time. I've used multiprocessing dummy, I think this uses threads though? I can't seem to figure out how to use all cores, rather than just all threads?

import mysql.connector
import pandas as pd
from sqlalchemy import create_engine
import multiprocessing.dummy as mp
from multiprocessing import cpu_count

dates_to_extract = ['ff', 'fff']

def do_process(s):
    #####SOURCE
    try:
        #source table conn
        engine = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
        iconn = engine.connect()
    except:
        print('connection failure to source')

    ######DEST
    try:
        #dest table conn
        enginedest = create_engine("mysql+pymysql://kieran:cccer@localhost/ran",echo = True)
        iconndest = enginedest.connect()
    except:
        print('connection failure to destination')


    print(s)
    #data from source table
    try:  
        data = pd.read_sql('SELECT * FROM ran.order_data', iconn)
    except Exception as e: 
        print('extract: ' + str(e))
          
    #data into destination table
    try:
        data.to_sql(name='tbjd', con=iconndest, if_exists = 'append')
    except Exception as e:
        print('insert: ' + str(e))

if __name__=="__main__":
    p=mp.Pool(cpu_count())
    p.map(do_process, dates_to_extract) 
    p.close()
    p.join()

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文