当前位置： 文江博客话题详情

卡夫卡+分区副本如何大于配置的 ReplicationFactor

发布于 2025-01-11 02:24:51 字数 6038 浏览 0 评论 0原文

我们在 Linux 机器上有 Kafka 集群，有 11 个代理，

我们有主题 - my_first_car ，有 300 个分区，并且这个主题配置了复制因子 = 3

我们从

kafka-topics.sh  --zookeeper zoo01:2181 --describe

以下示例中注意到： （请查看 --> ReplicationFactor:3 ）

Topic:my_first_car  PartitionCount:300  ReplicationFactor:3 Configs:retention.ms=17280000,retention.bytes=27571641765
    Topic: my_first_car Partition: 0    Leader: 1009    Replicas: 1009,1007,1008            Isr: 1007,1008,1009
    Topic: my_first_car Partition: 1    Leader: 1001    Replicas: 1001,1011,1008,1010,1009  Isr: 1001,1011,1008,1010,1009
    Topic: my_first_car Partition: 2    Leader: 1001    Replicas: 1001,1002,1011,1010,1009  Isr: 1001,1002,1011,1010,1009
    Topic: my_first_car Partition: 3    Leader: 1001    Replicas: 1001,1002,1011,1003,1010  Isr: 1001,1002,1011,1003,1010
    Topic: my_first_car Partition: 4    Leader: 1002    Replicas: 1002,1011,1001            Isr: 1002,1001,1011
    Topic: my_first_car Partition: 5    Leader: 1005    Replicas: 1005,1001,1002,1003,1004  Isr: 1003,1005,1001,1002
    Topic: my_first_car Partition: 6    Leader: 1005    Replicas: 1005,1006,1002,1003,1004  Isr: 1005,1003,1002,1006
    Topic: my_first_car Partition: 7    Leader: 1005    Replicas: 1005,1003,1004            Isr: 1003,1005,1004
    Topic: my_first_car Partition: 8    Leader: 1005    Replicas: 1006,1004,1005            Isr: 1005,1004,1006
    Topic: my_first_car Partition: 9    Leader: 1005    Replicas: 1005,1006,1007,1008,1009  Isr: 1005,1006,1007,1008,1009
    Topic: my_first_car Partition: 10   Leader: 1007    Replicas: 1007,1009,1010            Isr: 1009,1007,1010
    Topic: my_first_car Partition: 11   Leader: 1008    Replicas: 1008,1011,1001            Isr: 1001,1008,1011
    Topic: my_first_car Partition: 12   Leader: 1009    Replicas: 1009,1001,1002            Isr: 1002,1001,1009
    Topic: my_first_car Partition: 13   Leader: 1010    Replicas: 1010,1002,1003            Isr: 1003,1002,1010
    Topic: my_first_car Partition: 14   Leader: 1011    Replicas: 1011,1003,1004            Isr: 1003,1004,1011
    Topic: my_first_car Partition: 15   Leader: 1001    Replicas: 1001,1004,1005            Isr: 1005,1001
    Topic: my_first_car Partition: 16   Leader: 1002    Replicas: 1002,1005,1006            Isr: 1005,1002,1006
    Topic: my_first_car Partition: 17   Leader: 1003    Replicas: 1003,1006,1007            Isr: 1003,1006,1007
    Topic: my_first_car Partition: 18   Leader: 1004    Replicas: 1004,1007,1008            Isr: 1004,1007,1008

正如我们在上面看到的，在某些分区上我们有 5 个副本，在 2 个分区上我们有有 4 个副本

，但配置清楚地表明 - ReplicationFactor:3

我们还检查了机器上的副本，我们看到分区确实根据描述的输出进行了复制

，那么修复此行为的方法是什么，以便只有 3 副本和 3 Isr ？

预期结果示例

Topic:my_first_car  PartitionCount:300  ReplicationFactor:3 Configs:retention.ms=17280000,retention.bytes=27571641765
    Topic: my_first_car Partition: 0    Leader: 1009    Replicas: 1009,1007,1008            Isr: 1007,1008,1009
    Topic: my_first_car Partition: 1    Leader: 1001    Replicas: 1001,1011,1008            Isr: 1001,1011,1008
    Topic: my_first_car Partition: 2    Leader: 1001    Replicas: 1001,1002,1011            Isr: 1001,1002,1011
    Topic: my_first_car Partition: 3    Leader: 1001    Replicas: 1001,1002,1011            Isr: 1001,1002,1011
    Topic: my_first_car Partition: 4    Leader: 1002    Replicas: 1002,1011,1001            Isr: 1002,1001,1011
    Topic: my_first_car Partition: 5    Leader: 1005    Replicas: 1005,1001,1002            Isr: 1003,1005,1001
    Topic: my_first_car Partition: 6    Leader: 1005    Replicas: 1005,1006,1002            Isr: 1005,1003,1002
    Topic: my_first_car Partition: 7    Leader: 1005    Replicas: 1005,1003,1004            Isr: 1003,1005,1004
    Topic: my_first_car Partition: 8    Leader: 1005    Replicas: 1006,1004,1005            Isr: 1005,1004,1006
    Topic: my_first_car Partition: 9    Leader: 1005    Replicas: 1005,1006,1007            Isr: 1005,1006,1007
    Topic: my_first_car Partition: 10   Leader: 1007    Replicas: 1007,1009,1010            Isr: 1009,1007,1010
    Topic: my_first_car Partition: 11   Leader: 1008    Replicas: 1008,1011,1001            Isr: 1001,1008,1011
    Topic: my_first_car Partition: 12   Leader: 1009    Replicas: 1009,1001,1002            Isr: 1002,1001,1009
    Topic: my_first_car Partition: 13   Leader: 1010    Replicas: 1010,1002,1003            Isr: 1003,1002,1010
    Topic: my_first_car Partition: 14   Leader: 1011    Replicas: 1011,1003,1004            Isr: 1003,1004,1011
    Topic: my_first_car Partition: 15   Leader: 1001    Replicas: 1001,1004,1005            Isr: 1005,1001,1007
    Topic: my_first_car Partition: 16   Leader: 1002    Replicas: 1002,1005,1006            Isr: 1005,1002,1006
    Topic: my_first_car Partition: 17   Leader: 1003    Replicas: 1003,1006,1007            Isr: 1003,1006,1007
    Topic: my_first_car Partition: 18   Leader: 1004    Replicas: 1004,1007,1008            Isr: 1004,1007,1008

问题 - 访问 Kafka 机器并从磁盘中删除额外副本是否是一个好的选择？

类似帖子 - 无法重新分配 kafka 主题分区

修复kafka中的复制分区

Kafka 分区在某些节点上不同步

https://www.signifytechnology.com/blog/2019/02/apache-kafkas-distributed-system-firefighter-the-controller-broker-by-stanislav-kozlovski

https://hevodata.com/learn/kafka-replication/

原文

we have Kafka cluster with 11 brokers on Linux machines

we have topic - my_first_car , with 300 partitions , and this topic configured with replication factor = 3

we notice from

kafka-topics.sh  --zookeeper zoo01:2181 --describe

the following example: ( please look on --> ReplicationFactor:3 )

Topic:my_first_car  PartitionCount:300  ReplicationFactor:3 Configs:retention.ms=17280000,retention.bytes=27571641765
    Topic: my_first_car Partition: 0    Leader: 1009    Replicas: 1009,1007,1008            Isr: 1007,1008,1009
    Topic: my_first_car Partition: 1    Leader: 1001    Replicas: 1001,1011,1008,1010,1009  Isr: 1001,1011,1008,1010,1009
    Topic: my_first_car Partition: 2    Leader: 1001    Replicas: 1001,1002,1011,1010,1009  Isr: 1001,1002,1011,1010,1009
    Topic: my_first_car Partition: 3    Leader: 1001    Replicas: 1001,1002,1011,1003,1010  Isr: 1001,1002,1011,1003,1010
    Topic: my_first_car Partition: 4    Leader: 1002    Replicas: 1002,1011,1001            Isr: 1002,1001,1011
    Topic: my_first_car Partition: 5    Leader: 1005    Replicas: 1005,1001,1002,1003,1004  Isr: 1003,1005,1001,1002
    Topic: my_first_car Partition: 6    Leader: 1005    Replicas: 1005,1006,1002,1003,1004  Isr: 1005,1003,1002,1006
    Topic: my_first_car Partition: 7    Leader: 1005    Replicas: 1005,1003,1004            Isr: 1003,1005,1004
    Topic: my_first_car Partition: 8    Leader: 1005    Replicas: 1006,1004,1005            Isr: 1005,1004,1006
    Topic: my_first_car Partition: 9    Leader: 1005    Replicas: 1005,1006,1007,1008,1009  Isr: 1005,1006,1007,1008,1009
    Topic: my_first_car Partition: 10   Leader: 1007    Replicas: 1007,1009,1010            Isr: 1009,1007,1010
    Topic: my_first_car Partition: 11   Leader: 1008    Replicas: 1008,1011,1001            Isr: 1001,1008,1011
    Topic: my_first_car Partition: 12   Leader: 1009    Replicas: 1009,1001,1002            Isr: 1002,1001,1009
    Topic: my_first_car Partition: 13   Leader: 1010    Replicas: 1010,1002,1003            Isr: 1003,1002,1010
    Topic: my_first_car Partition: 14   Leader: 1011    Replicas: 1011,1003,1004            Isr: 1003,1004,1011
    Topic: my_first_car Partition: 15   Leader: 1001    Replicas: 1001,1004,1005            Isr: 1005,1001
    Topic: my_first_car Partition: 16   Leader: 1002    Replicas: 1002,1005,1006            Isr: 1005,1002,1006
    Topic: my_first_car Partition: 17   Leader: 1003    Replicas: 1003,1006,1007            Isr: 1003,1006,1007
    Topic: my_first_car Partition: 18   Leader: 1004    Replicas: 1004,1007,1008            Isr: 1004,1007,1008

as we can see above on some partitions we have 5 replicas and on 2 partitions we have 4 replicas

but the configuration say clearly that - ReplicationFactor:3

we also check physically the replica on machines , and we saw that partition are really replicated according to described output

so what is the way to fix this bhavior , in order to have only 3 replica and 3 Isr ?

example of expected results

Topic:my_first_car  PartitionCount:300  ReplicationFactor:3 Configs:retention.ms=17280000,retention.bytes=27571641765
    Topic: my_first_car Partition: 0    Leader: 1009    Replicas: 1009,1007,1008            Isr: 1007,1008,1009
    Topic: my_first_car Partition: 1    Leader: 1001    Replicas: 1001,1011,1008            Isr: 1001,1011,1008
    Topic: my_first_car Partition: 2    Leader: 1001    Replicas: 1001,1002,1011            Isr: 1001,1002,1011
    Topic: my_first_car Partition: 3    Leader: 1001    Replicas: 1001,1002,1011            Isr: 1001,1002,1011
    Topic: my_first_car Partition: 4    Leader: 1002    Replicas: 1002,1011,1001            Isr: 1002,1001,1011
    Topic: my_first_car Partition: 5    Leader: 1005    Replicas: 1005,1001,1002            Isr: 1003,1005,1001
    Topic: my_first_car Partition: 6    Leader: 1005    Replicas: 1005,1006,1002            Isr: 1005,1003,1002
    Topic: my_first_car Partition: 7    Leader: 1005    Replicas: 1005,1003,1004            Isr: 1003,1005,1004
    Topic: my_first_car Partition: 8    Leader: 1005    Replicas: 1006,1004,1005            Isr: 1005,1004,1006
    Topic: my_first_car Partition: 9    Leader: 1005    Replicas: 1005,1006,1007            Isr: 1005,1006,1007
    Topic: my_first_car Partition: 10   Leader: 1007    Replicas: 1007,1009,1010            Isr: 1009,1007,1010
    Topic: my_first_car Partition: 11   Leader: 1008    Replicas: 1008,1011,1001            Isr: 1001,1008,1011
    Topic: my_first_car Partition: 12   Leader: 1009    Replicas: 1009,1001,1002            Isr: 1002,1001,1009
    Topic: my_first_car Partition: 13   Leader: 1010    Replicas: 1010,1002,1003            Isr: 1003,1002,1010
    Topic: my_first_car Partition: 14   Leader: 1011    Replicas: 1011,1003,1004            Isr: 1003,1004,1011
    Topic: my_first_car Partition: 15   Leader: 1001    Replicas: 1001,1004,1005            Isr: 1005,1001,1007
    Topic: my_first_car Partition: 16   Leader: 1002    Replicas: 1002,1005,1006            Isr: 1005,1002,1006
    Topic: my_first_car Partition: 17   Leader: 1003    Replicas: 1003,1006,1007            Isr: 1003,1006,1007
    Topic: my_first_car Partition: 18   Leader: 1004    Replicas: 1004,1007,1008            Isr: 1004,1007,1008

Question - is it good option to access the Kafka machines and to remove the additional replica from disk ?

similar posts - Cannot reassign kafka topic partition

Fixing under replicated partitions in kafka

Kafka partitions out of sync on certain nodes

https://www.signifytechnology.com/blog/2019/02/apache-kafkas-distributed-system-firefighter-the-controller-broker-by-stanislav-kozlovski

https://hevodata.com/learn/kafka-replication/

如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论， 你可以免费 注册 一个本站的账号。
列表为空，暂无数据

关于作者

箹锭⒈辈孓

暂无简介

文章
评论
25 人气
发私信

相关话题

更多

热门标签

操作系统 程序设计 IT运维 Linux系统管理 JavaScript 服务器应用 solaris C/C++ PHP Shell BSD Vue.js aix Oracle Python HTML 系统管理 HTML5 CSS 前端
更多

推荐作者

紫罗兰の梦幻

文章 0 评论 0

-2134

文章 0 评论 0

liuxuanli

文章 0 评论 0

意中人

文章 0 评论 0

○愚か者の日

文章 0 评论 0

xxhui

文章 0 评论 0

更多

友情链接

文江博客
    我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站，即表示您同意使用 Cookies 和您的相关数据。
    原文