Google App Engine 任务队列
我想运行 50 个任务。所有这些任务都执行同一段代码。唯一的区别是数据。哪个会完成得更快？
一个。在队列中排队 50 个任务
b．在 10 个不同的队列中分别排队 5 个任务
在使用另一个队列之前，是否存在可以在 1 个队列中排队的理想数量的任务？
I want to run 50 tasks. All these tasks execute the same piece of code. Only difference will be the data. Which will be completed faster ?
a. Queuing up 50 tasks in a queue
b. Queuing up 5 tasks each in 10 different queue
Is there any ideal number of tasks that can be queued up in 1 queue before using another queue ?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（4）
任务执行的速率取决于两个因素：应用程序运行的实例数量以及任务所在队列的执行速率。
现在，最大任务队列执行率为每个队列每秒 100 个，因此这不太可能成为限制因素 - 因此将它们添加到同一队列中不会有任何坏处。无论如何，在队列之间进行分片以获得更高的执行率充其量只是一种黑客手段。队列是为功能分离而设计的，而不是作为性能衡量标准。
The rate at which tasks are executed depends on two factors: the number of instances your app is running on, and the execution rate of the queue the tasks are on.
The maximum task queue execution rate is now 100 per queue per second, so that's not likely to be a limiting factor - so there's no harm in adding them to the same queue. In any case, sharding between queues for more execution rate is at best a hack. Queues are designed for functional separation, not as a performance measure.
任务队列的突发率由桶大小控制。如果队列的存储桶中有令牌，则任务应立即运行。因此，如果您有：
并且没有在一秒钟内排队任何任务，那么所有任务都应该立即开始。
请参阅 http://code.google.com/appengine/docs/ python/config/queue.html#Queue_Definitions 了解更多信息。
将任务拆分到不同的队列中不会提高响应时间，除非存储桶没有足够的时间完全填充令牌。
The bursting rate of task queues is controlled by the bucket size. If there is a token in the queue's bucket the task should run immediately. So if you have:
And haven't queue any tasks in a second all tasks should start right away.
see http://code.google.com/appengine/docs/python/config/queue.html#Queue_Definitions for more information.
Splitting the tasks into different queues will not improve the response time unless the bucket hadn't had enough time to completely fill with tokens.
我会在混合中添加另一个因素——并发性。如果您的任务运行速度较慢（超过 30 秒左右），那么 AppEngine 似乎很难扩展正确的实例数量来处理请求（对我来说最大约为 7-8 个）。
从 SDK 1.4.3 开始，queue.xml 和 appengine-web.config 中有一项设置，您可以使用它告诉 AppEngine 每个实例一次可以处理多个任务：
这也解决了我执行任务时遇到的所有问题缓慢（尽管将所有其他队列参数设置为最大值）
更多详细信息 (http://blog.crispyfriedsoftware.com/post/7718668518/appenginetaskqueueproblems" rel="nofollow"> com）
I'd add another factor into the mix- concurrency. If you have slow running (more than 30 seconds or so) tasks, then AppEngine seems to struggle to scale up the correct number of instances to deal with the requests (seems to max out about 7-8 for me).
As of SDK 1.4.3, there's a setting in your queue.xml and your appengine-web.config you can use to tell AppEngine that each instance can handle more than one task at a time:
This solved all my problems with tasks executing too slowly (despite setting all other queue params to the maximum)
More Details (http://blog.crispyfriedsoftware.com)
将 50 个任务排队，并将队列设置为一次处理 10 个任务，或者设置为您想要的任何任务（如果它们可以彼此独立运行）。我看到一个类似的问题，我一次只运行 10 个任务来处理 3300 左右我需要运行的任务。处理所有这些需要 45 分钟左右，但令人惊讶的是，所使用的 CPU 时间可以忽略不计。
Queue up 50 tasks and set your queue to process 10 at a time or whatever you would like if they can run independently of each other. I see a similar problem and I just run 10 tasks at a time to process the 3300 or so that I need to run. It takes 45 minutes or so to process all of them but the CPU time used is negligible surprisingly.