将 EC2 实例设置为 Celery Workers，并以本地计算机作为主机
类似于我的问题此处我正在尝试设置多个亚马逊EC2实例进行一些多处理。我正在考虑使用 Celery 来管理工人。有人用 celery 在以本地计算机作为主机的 EC2 实例上工作吗？
有没有人有任何好的建议、教程、建议等可能有帮助？我已经使用 celery 在 django 中执行了一些简单的异步进程，但没有达到这种规模（工作进程和主机在同一台机器上）。
而且大部分处理都是“基于文件”（即读取和写入文件）...您认为用 celery 腌制和传输文件内容会更好吗（大多数文件是 1-2kb 的文本）还是跨 EC2 实例镜像文件系统，然后让工作人员返回结果（通常是 0.5 kb 的文本）。
Similar to my question here I'm trying to set up multiple amazon EC2 instances to do some multiprocessing. I was thinking of using Celery to manage the workers. Has anyone gotten celery to work on EC2 instances with a local computer as a host?
Does anyone have any good suggestions, tutorials, advice, etc. that may help? I've used celery to do some simple asynchronous processes in django but nothing of this scale (worker and hosts were on the same machine).
Also most of the processing is 'file-based' (ie reading and writing files) ... do you think it would be better to pickle and transmit the contents of the file with celery (most files are 1-2kb of text) or to mirror the filesystem across the EC2 instances and then just have the workers return the results (which are usually 0.5 kb of text).
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我使用 Amazon SQS 与 Amazon EC2 进行任务管理。这是一个非常可扩展的解决方案。据我发现，Boto 是管理亚马逊服务的最佳库。
要存储大量小文件，您可以使用 MongoDB GridFS，它将允许您存储大量本地文件。我使用 MongoDB 并在此类任务中获得了完美的性能。唯一的问题 - 32 位架构上的 MongoDB 非常有限。亚马逊有微型实例，其次是支持 64 个的大型实例。微型实例受到 CPU 和内存的限制，如果它不能满足您的需求，您需要设置大型实例，这可能会花费很多。
我的任务上的微型实例每天能够读取/写入多达 10 次数据，没有任何问题。
另请查看 Spot 实例。它的成本大约比按需成本低 3 倍，而且您可能会发现它们非常适合后台处理。
I've used Amazon SQS for tasks management with Amazon EC2. It's very scalable solution. Boto is the best library for managing Amazon services as I found.
For storing big amount of small files you can use MongoDB GridFS, it will allow you to store gigs of local files. I used MongoDB and got prefect performance for such tasks. The only problem - MongoDB on 32 bit architecture very limited. Amazon has micro-instance and next by cost large instance that supports 64. Micro instance very limited by CPU and memory and if it will not fit you needs you need to set up large, that may cost a lot.
Micro instance on my tasks was able to read/write up to 10 gigs a day without any problems.
Also please take a look at Spot instances. It costs about 3 times less than on demand and you may find them pretty good for background processing.