Kafka专题-如何保证一键一分区关系
我正在开发一个项目，我希望在我的主题中确保每个分区有 1 个且只有 1 个密钥。
问题是我不知道将产生数据的特定键的数量（可能是 1、2 或 1000！流式传输的不同键的数量随时间变化）。
在一个主题中，我知道我们必须指定初始分区数量，但我们可以在创建后添加更多分区。
确保一键一分区的最佳解决方案是什么？。
我有一些线索...
我可以提前创建一个具有 3000 个分区的主题，这样我就有了一个缓冲区，但它绝对没有优化。磁盘空间消耗怎么样？对性能的影响如何？
一旦用完可用分区，我就可以添加分区，因此我只会有与键相同数量的分区，但是会发生冲突，并且会威胁事件流的连续性，因为可能会分配一个键到不同的分区
我可以覆盖生产者服务使用的默认分区程序，以确保键之间不存在冲突并且调整大小不会影响，但是我的消费者如何知道分区程序选择的分区？如何确保没有其他生产者影响另一个密钥的相同分区号...
非常感谢您的帮助！
I am working on a project where I would like to ensure I have 1 and only 1 key per partition in my topic.
The problem is that I don't know the number of specific keys I will produce data from (could be 1, 2, or 1000 ! The number of different keys streamed vary in time).
In a topic, I know we have to specify an initial number of partitions, but we can add more after creation.
What would be the best solution to ensure 1 key 1 partition?.
I have some leads...
I could create a topic with 3000 partition in advance so I have a buffer, but it is definitely not optimized. What about disk space consumption? What about the impact on performances?
I could add partitions once I run out of available partitions, so I will only have the same number of partitions as keys, but there will be collision and it will threaten the continuity of the event stream as potentially a key will be assigned to a different partition
I could override the default Partitioner used by my Producer services to ensure there is no collision between keys and the resizing does not affect, but how my consumer will know the partition the partitioner choose? How to ensure no other Producer affect the same partition number to another key...
Many thanks for your help !
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论