创建许多实体的 AppEngine 任务
我正在为参加活动的人们构建一个应用程序。我需要为特定事件的人员实体子集创建一个票证实体。实体人数可能超过50,000人。
显然，我不能只执行一个 for 循环，迭代 Person 查询并生成这些票证。
我如何在 App Engine 上构建这个结构，有没有办法利用 MapReduce？
I'm building an application for people attending events. I need to create a Ticket entity for a subset of Person entities, for a specific Event. The amount of people may exceed 50,000 entities.
Obviously I can't just do a for-loop where I iterate over a Person query and generate these Tickets.
How do I structure this on App Engine, is there a way to take advantage of MapReduce?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
您可能需要查看延迟库。您可以并行排列一堆任务队列来完成您想要的工作。您可能需要查看 Google 文档中的Mapper 示例类，这可能会有所帮助将你推向正确的方向。
You may want to take a look at the Deferred library. You can spool up a bunch of task queues in parrallel to do the work that you want. You may want to look at the Mapper example class in the google docs which may help push you in the right direction.
如果后端的请求可以持续很长时间，则可以在单个 for 循环中进行迭代。但在我看来，如此长时间运行的进程并不是一件好事。我想正确使用任务队列就足够了。
我读到了关于延迟图书馆的内容。有时它的行为很奇怪，腌制数据可能会带来一些麻烦。我可以使用 TaskQueue API
You can iterate in a single for loop if you the Backend where a request can last for long. But such long running processes are not a good thing in my opinion. I guess proper use of task queues is more than enough.
I read about the Deferred library. Sometimes it behaves strangely and pickling your data can introduce some headaches. I could for the TaskQueue API
我不建议使用延迟库，虽然它很容易编写代码，但缺点它将腌制您的数据，将其放入一个实体中，然后加载并取消腌制它，这会花费大量开销。放置 30K 实体花费了我大约 3 个 CPU 小时！
最便宜的方法就是使用任务队列分割 Person 并使用键或其他位置信息入队。插入相同的 30K 实体，使用时间少于 1 CPU 小时。
在你的问题中，获取100万个实体并运行得非常快取决于GAE的设计，就这么做吧。最慢的部分是存储新的 Ticket 实体。
顺便说一句，
为什么不只是
Person.all().filter("类似参加活动的东西")。I do not suggest Deferred Library, although it's very easy to write code, the disadvantage is it will pickle your data, put it into one entity, and load and unpickle it later that cost a lot of overhead. Puts 30K entities costs me about 3 CPU Hours!
The cheapest way is just use the Task Queue that split the Person and enqueue with keys or other positional information. Insert same 30K entities used less than 1 CPU Hour.
In your question, fetch 1 million entities and run over is very fast depends on GAE's design, just do it. The slowest part is store the new Ticket entity.
BTW,
why not just
Person.all().filter("something like attending events").