Python 3.9+自定义模块中的类函数中的多处理
虽然 Python 文档充满了有关多处理的详细信息 和 https://zetcode.com/python/multiprocessing/ 提供了有趣的解释, 我无法成功运行任何示例而不遇到错误。 甚至 Python 自己的例子也给出了:
AttributeError:无法在<模块'main'(内置)>上获取属性'f'>
我的主要目标是能够将包含 pandas 数据帧部分的多个压缩 pickle 文件加载到内存中并将它们连接起来,或者直接将它们连接到一个数据帧中。 对于我的机器来说数据相当大,所以我需要将其压缩存储(因此 Dask 甚至不是一个选项)。
欢迎任何工作想法。 我的Python版本是3.9.7,Pandas 1.3.4
[更新] 可以在这里找到这方面的示例: https://www.journaldev.com/15631/python-multiprocessing-example 其中包含代码:
from multiprocessing import Process
def print_func(continent='Asia'):
print('The name of continent is : ', continent)
if __name__ == "__main__": # confirms that the code is under main function
names = ['America', 'Europe', 'Africa']
procs = []
proc = Process(target=print_func) # instantiating without any argument
procs.append(proc)
proc.start()
# instantiating process with arguments
for name in names:
# print(name)
proc = Process(target=print_func, args=(name,))
procs.append(proc)
proc.start()
# complete the processes
for proc in procs:
proc.join()
并且在清晰的控制台上运行时会发生相同的错误
While Python documentation is full of details about multiprocessing
and https://zetcode.com/python/multiprocessing/ provides interesting explanation,
I can't run successfully any of the example without running into errors.
Even Python's own examples give:
AttributeError: Can't get attribute 'f' on <module 'main' (built-in)>
My main goal is to be able to load multiple compressed pickle files containing parts of a pandas dataframe into the memory and join them, or directly join them into one dataframe.
The data is quite big for my machine, so I need to store it compressed (hence Dask isn't even an option).
Any working ideas are welcome.
My Python version is 3.9.7, Pandas 1.3.4
[update]
an example of this can be found here:
https://www.journaldev.com/15631/python-multiprocessing-example
which has the code:
from multiprocessing import Process
def print_func(continent='Asia'):
print('The name of continent is : ', continent)
if __name__ == "__main__": # confirms that the code is under main function
names = ['America', 'Europe', 'Africa']
procs = []
proc = Process(target=print_func) # instantiating without any argument
procs.append(proc)
proc.start()
# instantiating process with arguments
for name in names:
# print(name)
proc = Process(target=print_func, args=(name,))
procs.append(proc)
proc.start()
# complete the processes
for proc in procs:
proc.join()
and the same error occurs upon running on clear console
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论