Java中最简单的点多点数据分发库/框架
我在中央数据库中有一个表，它不断被附加。我想每隔几分钟对这些附加进行批处理，并将它们发送到一堆“从属”服务器。这些服务器将依次处理该数据，然后丢弃它（想想分布式仓储）。每个“从属”服务器都需要不同的数据子集，并且我可以使用一个自然的分区键。
基本上，我需要它最终保持一致：每批数据最终都会传递到每个“从属”服务器（可靠性），即使“从属”在该批次准备交付时已关闭（持久性）。我不关心批次交付的顺序。
我考虑过可能的解决方案，
- MySQL 复制不符合我的要求，因为我必须在每台服务器上复制整个表。
- ETL& ESB 产品对此来说太臃肿了，我不做任何数据处理。
- 普通 JMS，我可以使用，但我正在寻找更简单的
- JGroups 很有趣，但离开该组的成员一旦重新加入就不会收到消息。
- 推送文件&跨服务器确认文件：可以，但我不知道任何框架，因此需要编写自己的框架。
注意：这个问题是关于如何将数据从中央服务器可靠地转移到其他 N 个服务器。耐用性；而不是如何创建或摄取它。
（8 月 24 日编辑，添加耐久性要求）
I have a table in a central database that gets constantly appended to. I want to batch these appends every couple of minutes and have them sent to a bunch of "slave" servers. These servers will, in turn, process that data and then discard it (think distributed warehousing). Each "slave" server needs a different subset of the data and there is a natural partitioning key I can use for that.
Basically, I need this to be eventually consistent : every batch of data to be eventually delivered to every "slave" server (reliability), even if the "slave" was down at the moment the batch is ready to be delivered (durability). I don't care about the order in which the batches are delivered.
Possible solutions I have considered,
- MySQL replication does not fit my requirement because I would have to replicate the whole table on each server.
- ETL & ESB products are too bloated for this, I am not doing any data processing.
- Plain JMS, I could use but I'm looking for something even simpler
- JGroups is interesting but members that are left the group will not get the messages once they rejoin.
- Pushing files & ack files across servers : can do but I don't know of any framework so would need to write my own.
Note : This question is about how to move the data from the central server to the N others with reliability & durability; not how to create or ingest it.
(Edited on Aug 24 to add durability requirement)
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
您可以使用 JGroups 进行同样的操作。它是一个用于可靠多播通信的工具包
You may use JGroups for same. Its a toolkit for reliable multicast communication
我最终找到了“Spring 集成”，其中包括通过 SFTP 轮询目录的插件。
http://www.springsource.org/spring-integration
I ended up finding "Spring integration" which includes plugins to poll directories via SFTP for example.
http://www.springsource.org/spring-integration