对象列表的异步处理
速度是我正在从事的项目的严重问题。它需要大量的IO资源,主要来自数据库查询。我想知道最大化速度而不会引起内存问题等的最佳方法。
从本质上讲,我有一个称为model
的类。在该模型中是分配的数值属性。然后,我运行此模型,在该模型
实例中需要10-15个数据库调用。
但是我有1000多个型号。
因此,我创建了1000个对象并将它们放入列表中,即
model_objs = [Model(i) for i in range(1, 1000)] // simplistic example
我可以同步运行它:
for obj in model_objs:
obj.run()
但是它非常慢。 (〜5分钟)。
我一直在寻找使用threadpoolexecutor
的异步实现,但它引起了数据库连接的问题,使其挂起。
我所有的查询AE索引和优化,显然我对问题的描述非常基本,但我认为这不是一个查询优化问题。
我可以使用多处理
库来利用CPU(IE 2-4)来压缩速度,但我认为这不是解决问题的好解决方案。
最好的方法是什么?
道歉,因为没有更详细地描述,但工作很敏感且难以描述。
Speed is a serious issue with the project I am working on. It requires a huge amount of IO resourcing, primarily from database queries. I want to know the best way to maximise speed without causing memory issues etc..
Essentially, I have a class called Model
. Within that model are numerical attributes that are assigned. I then run this model, requiring 10-15 database calls within that Model
instance.
But I have over 1000 models.
So I create 1000 objects and put them into a list, i.e.
model_objs = [Model(i) for i in range(1, 1000)] // simplistic example
I could run it synchronously:
for obj in model_objs:
obj.run()
But it is very slow. (~ 5 minutes).
I've been looking at asynchronous implementations that use ThreadPoolExecutor
but it's causing issues with the database connection, making it hang.
All my queries ae indexed and optimised and obviously my description of the issue is very basic but I don't think its a query optimisation issue.
I could use multiprocessing
library to harness CPU's (i.e. 2-4) to compress the speed but I think that is not a good solution to the problem.
What is the best way to approach this?
Apologies for not describing in more detail but the work is sensitive and difficult to describe.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论