如何使用可靠的异步消息可靠地解析和存储数据?

发布于 2024-12-04 05:32:35 字数 467 浏览 1 评论 0原文

我正在开发一个系统,该系统接收与医院中患者活动相关的消息,解析并存储这些数据(HL7 2.x 消息)。 有些消息可能包含基本的患者信息,其他消息可能包含实验室结果、转录文档等。

我想使用异步可靠消息传递系统来执行此操作(例如 WCF + MSMQ 或 Apache Camel + ActiveMQ。看起来像这些 不过,

我很难理解如何应用这些技术来解决问题。 如果收到消息并且在解析和存储患者数据时出现问题,我需要停止处理该患者的传入消息,直到问题解决。

如果出现问题,我真的不想停止处理所有患者的消息,只想停止处理有问题的单个患者的消息。

是否有我缺少的设计模式或某种处理此类情况的方法?

我很可能会使用数据库来存储消息并跟踪各个患者队列。如果发生错误,我可以锁定患者,这样在问题解决之前将不再处理消息。

我只是在寻找健全性检查。有没有更简单的方法来做到这一点,或者手动管理数据库中的队列和患者是解决此问题的合理方法?

I'm developing a system that receives messages related to patient activity in a hospital, parses and stores this data (HL7 2.x messages).
Some messages may contain basic patient information, other messages could contain lab results, transcribed documents, etc.

I'd like to use an asynchronous reliable messaging system to do this (such as WCF + MSMQ or Apache Camel + ActiveMQ. It seems like these technologies would be a good fit.

I'm having trouble understanding how to apply these technologies to solve the problem though.
If a message comes in and something goes wrong when parsing and storing data for a patient I need to halt processing of incoming messages for this patient until the problem is resolved.

I don't really want to stop processing messages for all patients if a problem occurs, just messages for the single patient that has a problem.

Is there a design pattern or some method of handling situations like this that I'm missing?

I'll most likely use a database to store the messages and keep track of individual patient queues. If an error occurs I can lock the patient so messages will no longer be processed until the problem is resolved.

I'm just looking for a sanity check. Is there an easier way to do this, or is manually managing queues and patients in the database a reasonable way to solve this problem?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(4

甜宝宝 2024-12-11 05:32:35

这与客户最近的要求非常相似,以下是我们在 Camel/ActiveMQ 中解决此问题的方法...

  • 使用 ActiveMQ 消息组对给定患者进行单线程处理(保留顺序等),并且仍然允许多线程处理
  • 设置 Camel 异常处理,将患者添加到异常列表(存储在Hazelcast Cache) 当处理错误时
  • 使用 Camel 过滤器 将异常列表患者的消息路由到异常队列
  • 设置计时器以定期重试来自异常列表患者的消息异常队列
  • 设置发生异常时的电子邮件通知(如果将消息分组到的容量很大,请使用 聚合器更少的电子邮件等)
  • 使用ActiveMQ JMX 手动查看/重试/移动/delete 消息以对其进行适当分类(构建了一个基于 Web 的控制台来支持此操作)

This is very similar to a recent requirement from a client, here is how we solved this in Camel/ActiveMQ...

  • used ActiveMQ message groups to single-thread processing (preserve order, etc) for a given patient and still allow multi-threaded processing
  • setup Camel exception handling to add patients to an exception list (stored in Hazelcast or Cache) when processing errors occur
  • used a Camel filter to route messages for exception list patients to an exception queue
  • setup a timer to periodically retry messages from the exception queue
  • setup email notifications when exceptions occurred (use an aggregator if this is high volume to group messages into fewer emails, etc.)
  • used ActiveMQ JMX to manually view/retry/move/delete messages to triage them appropriately (built a web based console to support this)
时光倒影 2024-12-11 05:32:35

您可以使用基于内容的路由器
http://camel.apache.org/content-based-router.html

然后将“问题患者”的消息路由到特殊队列。对于没有问题的患者,您可以照常处理。

然后,当患者“修复”时,您可以使用 JMS 选择器从“问题队列”中拾取该患者的消息并将其放回常规队列,以便重新处理它们。

You could possible use a Content Based Router
http://camel.apache.org/content-based-router.html

And then route messages for "trouble patients" to a special queue. And for patients with no problems you can process them as usual.

Then when a patient is "fixed" you can use a JMS selector to pickup the messages for that patient from the "trouble queue" and put back on the regular queue, so they are re-processed.

扬花落满肩 2024-12-11 05:32:35

如何处理特定患者 ID 的失败。

每个患者都有一个消息队列根本无法扩展。您可能有 1000 名患者,因此您需要保持相同数量的队列。恶梦。

因此,要处理同一队列上的多个患者记录,您需要使从队列读取的服务具有容错能力。

我的意思是,如果服务无法处理出列的消息(可能是由于消息的数据问题,或者下游依赖项不可用),则服务可以执行以下一项或多项操作:

  1. 重试配置的次数(以配置的频率)
  2. 将该消息以及任何错误/诊断信息路由到另一个消息队列或另一个进程。
  3. 不执行任何操作并允许消息的生存时间超时到期,以便队列系统将其作为错误消息处理(不推荐)。

如果您要求始终按顺序处理单个患者的消息,那么您将面临不同的挑战,并且需要在允许处理每条消息之前对其实施执行/不执行查找。这是标准的有序交付实现。

希望这有帮助。

我认为也许您应该重新发布一个新问题或多个问题,并提出您面临的更有针对性的挑战。我感觉您目前面临着多重挑战。

How to handle failures for a particular patient ID.

Well having a message queue per patient does not scale at all. You could have 1000's of patients so you would need to maintain the same number of queues. Nightmare.

So to handle multiple patient records on the same queue you need to make the service which reads from you queue fault-tolerant.

What I mean is that if the service cannot process a de-queued message (maybe due to data issues with the message, or downstream dependencies being unavailable) then the service can do one or more of the following:

  1. Retry a configured number of times (with a configured frequency)
  2. Route that message, along with any error/diagnostics information, to another message queue or another process.
  3. Do nothing and allow the time-to-live time-out on the message to expire so it is handled as a bad message by the queueing system (not recommended).

If you have a requirement that messages for a single patient always be processed in order then you have a different challenge and need to implement a go/no-go lookup for each message before allowing it to be processed. This is a standard ordered-delivery implementation.

Hope this helps.

I think maybe you should re-post a new question or multiple questions with a more focused challenge you are facing. I get the feeling you are facing multiple challenges at the moment.

旧城空念 2024-12-11 05:32:35

http://ignaciosuay.com/unit-testing-hl7-messages -with-apache-camel/I 将建议您使用 activemq 和 Camel 并启用重新交付选项。因此,如果出现问题,消息将被重新传递,直到消息被处理为止。此外,您还可以指定可以重新发送的次数。请查看camel redilvery 政策:

http://camel.apache.org/redeliverypolicy.html< /a>

另外,我编写了一个使用camel和hl7的简单测试示例,也许会对您有所帮助:

http://ignaciosuay.com/unit-testing-hl7-messages-with-阿帕奇骆驼/

http://ignaciosuay.com/unit-testing-hl7-messages-with-apache-camel/I will suggest you to use activemq and camel with the redelivery option on. So if something goes wrong you the message will be redeliver until the message will be processed. Also, You can specify the number of times that could be redeliver. Please, have a look at the camel redilvery policy:

http://camel.apache.org/redeliverypolicy.html

Also, I have written an easy test example which use camel and hl7, that maybe will help you:

http://ignaciosuay.com/unit-testing-hl7-messages-with-apache-camel/

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文