Java 中的通用 QoS 消息批处理和压缩
我们有一个用 Java 编写的自定义消息系统，我想实现一个基本的批处理/压缩功能，基本上在重负载下它会将一堆推送响应聚合为一个推送响应。
本质上：
- 如果我们检测到过去一秒内发送了 3 条消息，则开始批处理响应并安排一个计时器在 5 秒内触发。
- 计时器会将接下来 5 秒内收到的所有消息响应聚合到一条消息中，
我确信这已经在我只是寻找 Java 中最好的例子之前已经实现了。我不是在寻找一个完整的消息传递层，只是每秒检测消息并安排一些任务（显然我可以轻松地自己编写这个），我只是想将它与任何现有的算法进行比较，以确保我不会错过任何边缘情况或者我已经尽可能简化了问题）。
是否有构建基本 QoS 批处理/限制/压缩实现的良好开源示例？
We have a custom messaging system written in Java, and I want to implement a basic batching/compression feature that basically under heavy load it will aggregate a bunch of push responses into a single push response.
Essentially:
- if we detect 3 messages were sent in the past second then start batching responses and schedule a timer to fire in 5 seconds
- The timer will aggregate all the message responses received in the next 5 seconds into a single message
I'm sure this has been implemented before I'm just looking for the best example of it in Java. I'm not looking for a full blown messaging layer, just the basic detect messages per second and schedule some task (obviously I can easily write this myself I just want to compare it with any existing algorithms to make sure I'm not missing any edge cases or that I've simplified the problem as much as possible).
Are there any good open source examples of building a basic QoS batching/throttling/compression implementations?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我们正在使用非常相似的高负载机制。
它会像你描述的那样工作
* 聚合给定时间间隔内的消息
* 之后发送一个列表而不是一条消息。
* 再次开始聚合。
您应该警惕以下陷阱：
* 如果您使用像 JMS 这样的事务处理消息系统，您可能会遇到麻烦，因为您的实现将无法在 JMS 事务内部发送，因此它将继续聚合。根据保存消息的数据结构的大小，这可能会耗尽空间。如果您的事务很长，发送许多消息，这可能会造成问题。
* 以这种方式发送消息将是异步发生的，因为将有不同的线程发送消息，并且调用 send() 方法的线程只会将其放入数据结构中。
* 坚持使用 JMS 示例，您应该记住，这种方法也改变了消息的消费方式。因为您将从 JMS 收到消息列表作为单个消息。因此，一旦您提交了这条 JMS 消息，您就提交了整个消息列表。您应该检查这个问题是否符合您的要求。
we are using a very similar mechanism for high load.
it will work as you described it
* Aggregate messages over a given interval
* Send a List instead of a single message after that.
* Start aggregating again.
You should watch out for the following pitfalls:
* If you are using a transacted messaging system like JMS you can get into trouble because your implementation will not be able to send inside the JMS transaction so it will keep aggregating. Depending on the size of your data structure to hold the messages this can run out of space. If you are have very long transactions sending many messages this can pose a problem.
* Sending a message in such a way will happen asynchronous because a different thread will be sending the message and the thread calling the send() method will only put it in the data structure.
* Sticking to the JMS example you should keep in mind that they way messages are consumed is also changed by this approach. Because you will receive the list of messages from JMS as a single message. So once you commit this single JMS message you commited the entire list of messages. You should check if this i a problem to your requirements.