通知系统 - 双重写入问题
示例方案：
在组中创建新用户（我们需要确保他们的电子邮件是唯一的）。
我们想发送一个事件用户创作（通过pubsub/kafka/rabbitmq），以触发一些其他业务逻辑异步：
- 发送确认电子邮件
- 通知组管理员，我可以想象一个新用户加入了组，
我可以想象 组合该组。我们可以将确认电子邮件视为射击任务，因为用户可以再次触发它。但是，通知组管理员的情况并非如此（失去此类事件是不可接受的）。我们不能简单地将新用户保存到DB，然后发布事件，因为它很容易失败（双写问题）。我们可以朝着纯事件驱动的方法迈进，但后来我不知道如何为此提供同步的REST API。
问题
人们在应用程序中实施通知/事件系统时如何处理现实生活中的双重写入问题？每个人都真正使用CDC（例如Debezium）使用交易量式折扣模式？对我来说，这似乎是个过度杀伤力，但我真的想不出一种更好的解决这个问题的方法（除非您也可以将API完全异步呼叫）。将DB表（而不是CDC）进行轮询吗？我们如何扩展？
如果您可以分享您的经验或将一些示例项目链接为很棒的参考！我发现的大多数教程似乎完全忽略了这个问题。
以防万一，我主要与Python（FastApi）合作，但是对我分析其他技术（例如Java/nodejs）的项目对我来说不是一个大问题。
Example scenario:
New user is created within a group (we need to ensure their email is unique etc.).
We'd like to send an event UserCreated (via PubSub/Kafka/RabbitMQ) in order to trigger some additional business logic asynchronously:
- send confirmation email
- notify group admin that a new user joined the group
I can imagine we can treat the confirmation email as fire-and-forget task as it can be triggered again by the user. However, that's not true for notifying the group admin (loosing such event can be unacceptable). We cannot simply save a new user to db and then publish an event as it can easily fail (dual write problem). We could move towards pure event-driven approach but then I have no idea how to provide a synchronous REST API for that it.
Question
How do people deal with dual write problem in real life when implementing notification/event system in their apps? Does everybody really use transactional outbox pattern with CDC (e.g. Debezium)? It seems like an overkill to me but I really can't think of a better way to tackle that problem (unless you can make your API calls fully asynchronous as well). Is polling a db table (instead of CDC) an acceptable solution? How could we scale that?
If you could share your experience or link some example projects as a reference that would be awesome! Most of the tutorials I was able to find seem to totally ignore the problem.
Just in case, I work mostly with Python (FastAPI) but it shouldn't be a big problem for me to analyse projects in other technologies (like Java/NodeJS).
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
如果您选择了分布式架构，那么您需要设计系统以考虑可用的消息传递保证。
恰好一次交货保证是不可能的/昂贵得令人望而却步。因此，您可以选择“最多一次”交付保证或“至少一次”交付保证。
至少一次意味着您的订阅者需要能够处理他们收到具有相同语义的消息的两个（或更多）副本的情况（因为他们可以检测到重复，或者因为重复处理的成本是可以接受的） 。
If you've elected a distributed architecture, then you need to design your system to account for the messaging guarantees that are available.
Exactly once delivery guarantees are (take your pick) impossible / prohibitively expensive. So you get to choose between "At Most Once" delivery guarantees or "At Least Once" delivery guarantees.
At Least Once means that your subscribers need to be able to handle the case where they receive two (or more) copies of a message with the same semantics (either because they can detect the duplication, or because the cost of duplicate processing is acceptable).
我可以想到 分裂 用户创建两个步骤。
首先，执行同步API请求在组中创建新用户，该请求立即为此请求返回某种“任务ID”。这只是意味着，好吧，我们收到了您创建新用户并将对其进行处理的请求。该任务ID可用于获取有关此请求状态的信息。如果从这一点上忘记了对客户端的响应，则在另一端收到创建用户的请求可能就足够了，并且任务ID（或请求ID）可能只有系统内部相关性，例如相关性，记录和背景中的实际处理。
当您的后端收到此请求时，例如，您可以将新命令放在 queue （例如创建用户命令）上，也可以作为事件实现（例如用户创建请求的事件）。请注意，通过排队，我宁愿参考排队的概念，因此可以通过交易量或某些持久的消息队列解决方案来不同地实现这一点。
如果您考虑在可靠的队列上使用此命令或事件（无论选择哪个实现），您现在可以尝试通过实际创建组中的新用户来对此“消息”异步反应。发生这种情况后，您可以发布一些用户创建的事件。
用户创建的事件可以是 订阅 ，如果您的情况有意义，则可以通过单个组件，甚至是单独的组件来通过发送确认电子邮件和来对其做出反应。通知组管理员。将其分为单独的订户可能会增加更多的实施工作，但也可以使您更加灵活地处理具有不同性能和可靠性要求的同一事件。例如，正如您提到的那样，电子邮件确认并不像在您的情况下通知管理员那样重要。
创建用户命令的实际处理 （或 分别请求的事件）和 用户创建事件 以所需的弹性程度来处理临时停电，并确保在某个时候发生一切，这使您具有 eve eve tual Constancions的特征。
我已经几次遵循了这种模式，尤其是在处理电子商务以实施订购流程时，客户（例如Web或移动前端）需要立即确认他们的请求已同步经过，但实际处理的实际处理关于完成的相同请求和通知可以在以后在异步上发生。
因此，您可以考虑创建新用户的请求类似于下订单，实际上创建了类似于处理订单的新用户，发送确认电子邮件，类似于向客户发送订单确认电子邮件以及管理员的通知类似于通知系统中其他重要的播放器有关新订单的信息，例如触发有序项目的分布的物流系统。
I can think of splitting the user creation two steps.
First, the synchronous API request to create the new user in the group is performed which immediately returns some kind of "task id" for this request. This just means, okay, we got your request to create a new user and will process it. The task id can be used to get information about the status of this request. If it is fire and forget from this point on the response to the client that the request to create the user was received at the other end might be sufficient and the task id (or request id) might only have system internal relevance, e.g. for correlation, logging and the actual processing in the background.
When this request is received at your backend you could for instance put a new command on a queue (like a create user command) or this could also be implemented as events (e.g. user creation requested Event). Note, by queueing I rather refer to the concept of queuing so this could be implemented differently, for instance, with transactional outbox or some persistent message queue solution.
If you consider having this command or event on a reliable queue (whichever implementation chosen) you could now try to react to this "message" asynchronously by actually creating the new user in the group. Once this happened you can publish some user created event.
The user created event can be subscribed by a single component or even separate components if that makes sense in your case to react to it by sending the confirmation email and by notifying the group admin. Splitting this into separate subscribers might add more implementation efforts but also gives you more flexibility in processing the same event with different performance and reliability requirements. For instance, as you mentioned, the email confirmation is not as crucial as informing your admin in your case.
The actual processing of the create user command (or user creation requested event respectively) and the user created event are then performed with the required degree of resiliency to deal with temporary outages and guaranteeing that everything is happening at some point which leaves you with the characteristics of eve tual consistency.
I've been following this pattern a couple of times already, especially when dealing with e-commerce for implementing ordering processes where clients (e.g. web or mobile front-ends) need immediate confirmation that their request has gone through synchronously but the actual processing of the same request and notification about the completion can happen later on asynchronously.
So you could consider the request to create the new user to be similar to placing an order, actually creating the new user similar to processing the order, sending the confirmation email similar to sending an order confirmation email to the customer and the notification of the admin similar to notifying some other important player in the system about the new order such as a logistics system that triggers the distribution of the ordered items.