Web 应用程序架构 - 需要作业/任务队列吗？
我目前正在设计一个 Web 应用程序，该应用程序将允许用户安排将针对 HTTP API（代表他们）执行的任务。这些任务可以重复出现，并且可用于调度的最小时间分辨率为一分钟。由于任务的性质，我认为异步执行它们是有意义的。但是，这部分的架构应该是什么样的呢？
我考虑过使用任务队列由 Web 应用程序创建任务并让工作人员执行它们。在这种情况下，我有几个问题：
- 如何处理重复性任务？
- 如何轻松保存任务结果？
- 是否可以轻松地使队列“持久化”？
- 工作人员应该直接与数据库交互吗？
- 我应该手动将重复任务排队吗？
我还能考虑什么？由于我认为我不是唯一一个考虑这种 Web 应用程序架构的人，那么是否有任何“最佳实践”？任务队列是可行的方法吗？
I am currently designing a web application that will allow users to schedule tasks which will be executed against an HTTP API (on behalf of them). The tasks can be recurring and the minimal time resolution that can be used for scheduling will be one minute. Because of the nature of the tasks I think it makes sense to execute them asynchronously. However, how should the architecture of this part look like?
I thought about using a task queue to create tasks by the web application and let them be executed by a worker. In this case, I have several questions:
- How do I handle recurring tasks?
- How do I easily save the results of the tasks?
- Is it easily possible to make the queue "persistent"?
- Should the workers directly interact with a database?
- Should I queue recurring tasks manually up?
What else could I consider? Since I assume I am not the only one thinking about this kind of web application architecture, are there any "best practices"? Is a task queue the way to go?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
是的，这是在 Web 应用程序后端处理长期任务的众所周知的模式。根据您的语言和应用程序框架，有许多队列实现 - 例如 Resque 或 Beanstalkd 或 ActiveMQ 或者如果您的性能要求不高，您可以使用数据库表作为一种队列。
基本思想是您的 Web 应用程序将作业放入队列中，并提供足够的内容以使作业能够继续进行。后台的一组工作进程（理想情况下独立于 Web 应用程序运行）从队列中读取作业并执行它们。结果可以写回回复队列或写入数据库。这取决于您希望如何向用户显示结果。对于 Web 应用程序，将结果写入数据库可能更有意义。
根据您的队列处理程序，它们可以使作业持久化。例如，ActiveMQ 支持持久消息传递，以便在发生故障时恢复队列上的消息。
你问有关重复性工作的问题 - 我认为答案取决于它们何时需要重复性。
直接消息队列将在消息可用时立即处理/释放消息给工作人员。所以安排是很棘手的或者是不可能的。为了支持计划的作业（包括在给定时间或经过一段时间后重复发生的作业），您可能应该考虑使用数据库表作为具有“开始时间”属性的简单队列。
我最近在此处描述了类似的模式。
Yes this is a well-known pattern for handling long-lived tasks at the back end of a web application. Depending on your langauge and application framework there are a number of queue implementations out there - e.g. Resque or Beanstalkd or ActiveMQ or if your performance requirements are not high you can use a database table as a kind of queue.
The basic idea is your Web application places jobs on a queue with enough content to enable to job to proceed. A group of worker processes in the background (ideally running independent of your web application) read the jobs off the queue and execute them. The results can be written back onto a reply queue or perhaps written into a database. It depends on how you want to display the results back to the user. For a web application, writing results into the database probably makse more sense.
Depending on your queue handler, they can make jobs persistent. E.g. ActiveMQ supports persistent messaging so that messages on a queue are recovered in the event of a failure.
You ask about recurring jobs - and I think the answer depends on when they need to recur.
A straight message queue will process/release messages to workers as soon as they become available. So scheduling is tricky or impossible. To support scheduled jobs (including jobs which recur at a given time or after a time elapse) you should probably look at using a database table as a simple queue with a "start time" attribute.
I recently described a similar pattern here.