有哪些方法可以分发异步批处理任务？
我目前正在研究存在哪些 Java 兼容解决方案来满足我的需求，如下所示：
- 基于计时器/可调度任务进行批处理
- 分布式，并通过提供水平扩展的能力
- 弹性，无需 SPF
这些任务的性质（繁重的 XML 生成，以及基于 Web 的接收节点的交付）意味着使用 Quartz 之类的东西在单个服务器上运行它们是不可行的。
我听说过 Hadoop 和 JavaSpaces 等技术，它们有效地解决了问题的扩展和弹性问题。不知道这些是否非常适合我的要求，也很难知道其他哪些技术可能更适合。
我真的想知道这个领域的人们认为有哪些可用的选择，以及每个选项如何发挥其优势，或者比其他选项更适合某些问题。
注意：值得注意的是，日程安排能力可能是我们目前做事方式的一个遗留问题。是的，有些任务应该在特定的时间进行。当不存在设定时间的强制要求时，它有时也被用来限制吞吐量。
I am currently investigating what Java compatible solutions exist to address my requirements as follows:
- Timer based / Schedulable tasks to batch process
- Distributed, and by that providing the ability to scale horizontally
- Resilience, no SPFs please
The nature of these tasks (heavy XML generation, and the delivery to web based receiving nodes) means running them on a single server using something like Quartz isn't viable.
I have heard of technologies like Hadoop and JavaSpaces which have addressed the scaling and resilience end of the problem effectively. Not knowing whether these are quite suited to my requirements, its hard to know what other technologies might fit well.
I was wondering really what people in this space felt were options available, and how each plays its strengths, or suits certain problems better than others.
NB: Its worth noting that schedule-ability is perhaps a hangover from how we do things presently. Yes there are tasks which ought to go at certain times. It has also been used to throttle throughput at times when no mandate for set times exists.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
异步总是让我想起 JMS。将请求消息发送到队列；从池中取出一个 MessageListener 来处理它。
这可以扩展，因为队列和侦听器可以位于远程服务器上。监听线程池的大小是可以配置的。您可以为不同的任务设置不同的侦听器。
更新：您可以通过集群和负载平衡来避免单点故障。
您可以使用 ActiveMQ（开源）、JBOSS（可用的开源版本）或任何 Java EE 应用服务器免费获得 JMS，因此无需考虑预算。
而且没有锁定，因为除了使用 Java 之外，您还使用 JMS。
我建议使用 Spring 消息驱动的 POJO 来实现这一点。当然，社区版是开源的。
如果这不适合您，请查看 Spring Batch 和 Spring 集成。这两者都可能有用，并且社区版本是开源的。
Asynchronous always brings JMS to mind for me. Send the request message to a queue; a MessageListener is plucked out of the pool to handle it.
This can scale, because the queue and listener can be on a remote server. The size of the listener thread pool can be configured. You can have different listeners for different tasks.
UPDATE: You can avoid having a single point of failure by clustering and load balancing.
You can get JMS without cost using ActiveMQ (open source), JBOSS (open source version available), or any Java EE app server, so budget isn't a consideration.
And no lock-in, because you're using JMS, besides the fact that you're using Java.
I'd recommend doing it with Spring message driven POJOs. The community edition is open source, of course.
If that doesn't do it for you, have a look at Spring Batch and Spring Integration. Both of those might be useful, and the community editions are open source.
您是否研究过GridGain？我很确定它不会解决调度问题，但你可以扩展它，它会像“魔术”一样发生，要执行的代码被发送到一个节点并在那里执行。当您没有要发送的数据库连接（或任何不可序列化的内容）时，它可以正常工作。
Have you looked into GridGain? I am pretty sure it won't solve the scheduling problem, but you can scale it and it happens like "magic", the code to be executed is sent to a node and it is executed in there. It works fine when you don't have a database connection to be sent (or anything that is not serializable).