为什么受监视的KAFKA消费者滞后值并不总是正确？
我正在使用 kubernetes 使用 minikube 在 kubernetes上运行apache kafka，而且我还有一个用python编写的 Confluent消费者。我正在用 JMX出口商和用 kminion消费者出口商监视KAFKA经纪人指标。这些出口商也是2个单独的豆荚。最后，我有 Prometheus 监视这两个出口商并阅读指标。
我每秒发出2条消息到某个主题。我的消费者会食用一条消息，然后执行任务。该任务需要0.4秒才能完成。因此，我也以每秒2条消息的速率消费。
我的假设是，由于我以相同的速度生产和消费，队列滞后指标应始终为零或2个。我每秒都在监视队列，这是我在5秒内获得的：
t = 0: Queue is 0.
t = 1: Queue is 3.
t = 2: Queue is 5.
t = 3: Queue is 7.
t = 4: Queue is 9.
t = 5: Queue is 0.
它重复相同的周期。因此，队列滞后的avg_over_time是5。为什么会发生这种情况？我知道消费者不能一次消耗9条消息，因为它运行了一项任务，该任务需要0.3秒才能完成，因此我的最大消费率是2次迫害。
我还尝试过将其他出口商用于消费者指标，但我仍然得到相同的结果。
I am running Apache Kafka on Kubernetes using minikube and I have also a Pod with a confluent consumer written in python. I am monitoring Kafka broker metrics with a JMX exporter and consumer metrics with kminion consumer exporter. These exporters are also 2 seperate pods. Lastly I have Prometheus monitoring both of these exporters and reading the metrics.
I am producing 2 messages per second to a certain topic. My consumer consumes a message and then runs a task. The task needs 0.4 seconds to complete. So I am also consuming with a rate of 2 messages per second.
My hypothesis is that the queue lag metric should be either zero or 2 at all times since I am producing and consuming at the same rate. I am monitoring the queue every second and this is what I get over a period of 5 seconds:
t = 0: Queue is 0.
t = 1: Queue is 3.
t = 2: Queue is 5.
t = 3: Queue is 7.
t = 4: Queue is 9.
t = 5: Queue is 0.
And it repeats the same cycle. So the avg_over_time of the queue lag is 5. Why is this happening? I know the consumer cannot consume 9 messages at once since it runs a task that takes 0.3 seconds to complete and therefore my maximum consume rate is 2 perSecond.
I have also tried using a different exporter for consumer metrics but I still get the same results.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
您的消费者什么时候承诺偏移？
如果您在处理每条消息后没有手动承诺偏移，则默认情况下，消费者每5秒提交一次： https://kafka.apache.org/documentation/#consumerconfigs_auto.commit.interval.ms
这会解释为什么您会看到这些排队值
When is your consumer committing the offsets?
If you are not manually committing offsets after processing each message, by default, a consumer commits them every 5 seconds: https://kafka.apache.org/documentation/#consumerconfigs_auto.commit.interval.ms
Which would explain why you see these queue values