Kafka Log Cleaner是否在AWS MSK上记录任何内容？

发布于 2025-01-27 09:15:25 字数 409 浏览 5 评论 0 原文

我将AWS MSK群集与经纪人登录到CloudWatch。登录作品，我可以看到经纪人日志。我们有一些主题， clearup.policy = compact 和 callicup.policy = delete 。该系统在新集群上运行大约2周。

根据我的研究（例如-Into-kafka-log-compaction-5E520F4291F0 ）我看到kafka应该运行日志清洁器（显然），并且该活动的日志中应该有一些痕迹。但是，在我的CloudWatch日志组中，我找不到一个单词“清洁器”或“清洁”，我找不到运行的日志清洁器的痕迹。

日志清洁器根本运行吗？显然应该这样做，但我在日志中找不到任何东西来确认这一点，而且我们有很多符合清理资格但仍未清理的消息，大约两个星期。

Kafka群集版本为2.8.1

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

近箐 2025-02-03 09:15:25

这些日志很可能没有在MSK中显示，因为默认情况下，它们没有从： https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-log-log-log-log-log-log-log-log-logcleaner.html

请注意，kafka带有预配置的kafka.log.logcleaner logger在config/log4j.properties中：

log4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log
log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender
log4j.additivity.kafka.log.LogCleaner=false

这意味着logCleaner的日志在信息日志记录级别转到logs/log-cleaner.log文件，并且未添加到主日志中（每个log4j.addititivity已关闭）。

不过，这有点误导，因为 logCleaner 要照顾紧凑的主题，我不确定已记录何处（或在哪个日志级别上，因为AWS MSK仅导出 info级logs logs ）带有 delete> delete delete> delete 清理政策。

我会联系AWS支持，以了解是否有办法或知道他们对这些日志有什么作用。

另外，您可以尝试设置 prometheus 将使Kafka导出的所有指标都向JMX出口。如果启用了，则应有 metric（max-clean time-sec）至少会告诉您它是否正在运行，并且您可能会获得其他一些有趣的信息来解决问题。

It is quite likely these logs are not being show in MSK since it seems that, by default, they do not go to the main log stream, from: https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-log-LogCleaner.html

Please note that Kafka comes with a preconfigured kafka.log.LogCleaner logger in config/log4j.properties:

log4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log
log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender
log4j.additivity.kafka.log.LogCleaner=false

That means that the logs of LogCleaner go to logs/log-cleaner.log file at INFO logging level and are not added to the main logs (per log4j.additivity being off).

It is a bit misleading though because the LogCleaner takes care of compacted topics, I'm not sure where is logged (or at which log level since AWS MSK only exports INFO level logs) the deletion of messages in topics with delete cleanup policy.

I would contact AWS support to know if there is a way or to know what do they do with these logs.

Alternatively, you could try to set up open monitoring with Prometheus which will get all metrics exported by Kafka to JMX. If enabled, there should be a metric (max-clean-time-sec) that, at least, will tell you if it is running and you may get some other interesting information to troubleshoot your issue.