使用Kafka和主题类型的活动采购，外国钥匙会发生什么？
我正在尝试使用Kafka设计事件来源系统。假设我们有一个非常简单的事件采购应用程序，该应用程序由四个汇总根源组成：用户，类别，产品和购买。
遵循 cancluent 我，创建了4个主题（并让我们假设每个主题为简单性），每个汇总一个：
- 用户一个主题
- 一个主题一个类别的主题一个
- 主题的产品
- 一个主题是购买我的投影的一个主题
，我的投影是一个postgressql投影，它从kafka读取事件，并且然后将每个事件变成SQL查询（根据事件的类型插入，更新或删除）。
问题在于数据库模型大约是这样的：
如果我尝试在没有类别的情况下将产品添加到数据库中，则不能保证该事件顺序数据库中的外键错误。同样，如果我尝试在没有用户或已处理的产品的情况下添加购买，我很可能会收到另一个外国错误。
我必须在读取模型中解决这个外国关键问题，并让我的系统随着更多的实体和更多的事件而不必依靠将所有内容放入一个主题以避免外国关键问题而增长？
我发生的事情是，尽管我可以在没有冲突的情况下独立处理用户和产品，但我仍然需要对购买（以某种方式延迟该主题上的事件的处理，直到确保用户和产品存在于数据库）。
I'm trying to design an event sourced system with Kafka. Let's say we have a very simple event sourced application consisting of four aggregate roots: users, categories, products and purchases.
Following some guidance from confluent, I've created 4 topics (and let's assume 1 partition per topic for simplicity), one per each aggregate:
- One topic for users
- One topic for categories
- One topic for products
- One topic for purchases
My projection is a PostgresSQL projection, which reads events from Kafka, and then turns each event into an SQL query (INSERT, UPDATE or DELETE depending on the type of event).
The problem is that the database model is approximately something like this:
Given that event order is not guaranteed between topics, if I try to add a product to the database without its category, I will most likely receive a ForeignKey error from the database. Likewise, if I try to add a purchase without the user or the product having been processed yet, I will most likely receive another ForeignKey error.
What partitioning options do I have to workaround this foreign key issue in the read model, and allow my system to grow with more entities and more events without having to rely on putting everything into a single topic to avoid foreign key problems?
Something that occurred to me is that, while I could process users and products independently without clashes, I would still need to do something with the purchases (delaying somehow the processing of events on that topic till ensuring that the user and the product exist in the database).
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
如果您的目标数据模型使用外部密钥，那么您将不得不延迟成功的插入件，直到满足约束为止。
可以想象，要这样做（假设您同时运行了消费流），就是要在失败上重试查询，直到成功为止，尽管这通常需要阻止受影响分区的进一步消息进行处理。它还取决于事件的含义（例如，如果可能发生删除或更新某些列），则可能打开各种僵局和数据竞赛，这是由于非确定性消费顺序引起的。
考虑到诸如外国钥匙限制之类的目的是验证和拒绝写入，它们通常在读取模型中没有太多目的，在此写作已经通过写作模型验证：一个将产品与产品相关联的事件无论读取模型的状态如何，都应将类别视为无可争议的证据表明每个产品和类别都存在。这表明至少一个：i）从读取模型中删除外国密钥约束，ii）在读取模型中具有最小的记录，以便允许外键验证，iiii）进行一些“待处理”）您知道存在但尚未完成的记录表（您的投影可能会检查待处理的表格，以查看更新是否可能是完成待处理记录所需的内容），或者iv）重新考虑聚合物。
If your target data model uses foreign keys, you're fundamentally going to have to delay successful inserts until the constraints can be met.
Conceivably, to do this (assuming that you're running the consuming streams concurrently), is to have the query retry on failure until it succeeds, though this will typically entail preventing further messages from the affected partition from being processed. It also, depending on what the meanings of events are (e.g. if it's possible for a delete to occur or certain columns to be updated), potentially opens various deadlocks and data races arising from non-deterministic order of consumption.
Considering that the purpose of things like foreign key constraints is to validate and reject writes, they generally don't have much purpose in a read-model, where the writes have already been validated by the write-model: an event associating a product with a category should be considered incontrovertible proof that the product and category each exist, regardless of the state of the read-model. This suggests at least one of: i) removing the foreign key constraints from the read-model, ii) having the projection create minimal records in the read-model in order to allow the foreign key to validate, iii) having some "pending" tables for records which you know exist but aren't yet complete (your projection might then check the pending table to see if an update could be what's needed to complete a pending record), or iv) rethinking your aggregates.