在 Django 请求/响应周期中使用 celery/RabbitMQ（队列）命中 API
我正在开发一个产品，需要抓取API并计算并返回结果。 我目前正在使用 Django 作为产品。我详细查看了 Celery 和 RabbitMQ 的文档以及一般队列。
我想要实现的是： 1. 监控/管理不同供应商调用的每个 API。 2. 具有多个消费者的多个队列。 3. 队列中的实时重新调度。
这里的主要问题是，在我们的整个周期中，主要耗时的过程是 API 请求，耗时在 1 秒到 20 秒之间。使用 celery / RabbitMQ 是个好主意吗？我无法使用带有确认的 Rabbit 队列来发出 API 请求并等待它完成，同时让其他人在队列中等待，没有确认的队列无济于事，因为跟踪失败的请求并重试它们是不可能的。 那么，我该如何使用队列来解决这个问题呢？
其次为了监控，我查看了这个 http://www.rabbitmq.com/management.html，管理监控插件，但我找不到一种方法来获取有关队列中已处理消息的信息，这对我来说非常重要。 有什么方法可以获取 RabbitMQ 的此信息吗？
我是否应该使用 ActiveMQ 或 ZeroMQ 等任何其他队列来获得更好的监控插件，特别是从队列中获取有关已处理消息的信息。
I am working on a product, that requires crawling APIs and computing and returning results.
I am currently using Django as product. I had a detailed look at Documentation of Celery and RabbitMQ and queues in general.
What I want to achieve is:
1. Monitoring / administration of each API called for different vendor.
2. Multiple queues with Multiple consumers.
3. Real time Re-Scheduling in Queues.
The main problem here is that in our entire cycle the main time consuming process is API requests taking anywhere between 1 sec to 20 sec. Is it a good idea to use celery / RabbitMQ. I cannot use a Rabbit queue with acknowledgement to make a API request and wait for it to get over while keeping others waiting in queues, queues without acknowledgement would not help as keeping track of failed requests and retrying them would not be possible.
So, how would I use queues to work around this issue?
Second For monitoring I had a look at this http://www.rabbitmq.com/management.html, the management n monitoring plugin, but I cannot find a way to get the information about processed messages in the queue which is very important in my case.
Is there any way I can get this info for RabbitMQ.
Should I use any other queue like ActiveMQ or ZeroMQ to get better monitoring plugins specially to get info about processed messages from the queue.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我已经使用 Beanstalk 一段时间并取得了巨大的成功。
服务器：https://github.com/kr/beanstalkd
客户端库：https://github.com/PeterScott/beanstalkc
Django 助手：https://github.com/jonasvp/django-beanstalkd
Django 管理应用程序： https://github.com/ humanfromearth/django-beanstalk
我认为 Beanstalk 有您正在寻找的许多选项。例如：
1.您可以设置TTR（运行时间），即作业在被释放回队列之前必须运行多长时间
2. 如果作业的 TTR 即将结束，您可以“触摸”它以重置 TTR 倒计时
3. Beanstalk 跟踪服务器、管道和作业级别的统计数据。所以你可以看到有多少作业是活跃的、延迟的、总共的等等。RabbitMQ
非常健壮，但它的重量很重。我发现 Beanstalk 更轻、更灵活且更易于使用。
I've used Beanstalk for a while now with great success.
Server: https://github.com/kr/beanstalkd
Client Library: https://github.com/PeterScott/beanstalkc
Django Helper: https://github.com/jonasvp/django-beanstalkd
Django Admin App: https://github.com/humanfromearth/django-beanstalk
I think Beanstalk has many of the options you are looking for. For example:
1. You can set TTR (time to run) for how long a job has to run before it is released back into the queue
2. If a job is nearing the end of its TTR, you can 'touch' it to reset the TTR countdown
3. Beanstalk keeps track of stats at the server, tube, and job level. So you can see how many jobs are active, delayed, total, etc.
RabbitMQ is very robust, but it's a heavy weight. I've found Beanstalk to be much lighter, flexible, and easier to use.