当前位置：文江博客话题详情

Python 与 MySQL 的多处理

发布于 2024-11-04 16:48:14 字数 250 浏览 0 评论 0原文

我已经在 Google 上搜索过这个主题很多次，但找不到适合我需求的解决方案:(

我有一个 MySQL 数据库，其中包含一个包含电子邮件地址（10,000+）的表。

我想每 5 对它们运行一个批处理作业所以

我猜Python是从MySQL检索结果集然后使用电子邮件地址作为参数调用命令行的最佳选择

。整个结果集从 MySQL 开始，然后有一堆工作用参数调用命令行，直到不再有电子邮件地址。这可以通过简单但稳定的方式完成吗？

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

时光礼记 2024-11-11 16:48:14

您可以像这样使用多处理模块：

from multiprocessing import Pool
p = Pool()    # let python choose the optimal number of processes (= number of CPU cores)
def treat_email(email_adress):
    # do the stuff you need with email address

email_addresses = grab_the_list_from_mysql()  # something like "select mail from my_user_table"

p.map(treat_email, email_addresses)    # this will treat all the emails in the X processes

you could use the multiprocessing module like this :

from multiprocessing import Pool
p = Pool()    # let python choose the optimal number of processes (= number of CPU cores)
def treat_email(email_adress):
    # do the stuff you need with email address

email_addresses = grab_the_list_from_mysql()  # something like "select mail from my_user_table"

p.map(treat_email, email_addresses)    # this will treat all the emails in the X processes

回复收藏 0 原文

静若繁花 2024-11-11 16:48:14

使用 ORM 模块的替代方法是，您可以将电子邮件转储到 CSV 文件中：

SELECT name, address
FROM email
INTO OUTFILE '/tmp/emails.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'

来自：从命令行将 mysql 数据库转储到纯文本 (CSV) 备份

并在 python 中对 CSV 文件进行后处理：

import csv
data = csv.reader(open('/tmp/emails.csv', 'rb'), delimiter=',')
for row in data:
    name, address = row
    print '%s <%s>' % (name, address)

CSV 文件读取和写入： http://docs.python.org/library/csv.html

当您处理非常大的文件时，您可能需要使用 file.readlines() 函数来防止 Python 将整个文件读入内存：

with open('/tmp/emails.csv', 'rb') as f:
    for line in f.readlines():
        name, address = line.split(',')
        print '%s <%s>' % (name, address)

An alternative to using an ORM module you could dump the emails in a CSV file:

SELECT name, address
FROM email
INTO OUTFILE '/tmp/emails.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'

From: Dump a mysql database to a plaintext (CSV) backup from the command line

And post process the CSV file in python:

import csv
data = csv.reader(open('/tmp/emails.csv', 'rb'), delimiter=',')
for row in data:
    name, address = row
    print '%s <%s>' % (name, address)

CSV File Reading and Writing: http://docs.python.org/library/csv.html

When your dealing with very large files you might want to use the file.readlines() function to prevent Python from reading the whole file into memory:

with open('/tmp/emails.csv', 'rb') as f:
    for line in f.readlines():
        name, address = line.split(',')
        print '%s <%s>' % (name, address)

回复收藏 0 原文

~没有更多了~