Fluentd Regex过滤器卸下其他钥匙

发布于 2025-02-06 08:42:27 字数 1349 浏览 3 评论 0原文

我会收到Fluentd的消息，其中已经从以前的阶段中填充了一些键（另一个主机上的流利位置）。我正在尝试将日志字段的内容解析如下：

# Parse app_logs
<filter filter.app.backend.app_logs>
  @type parser
  key_name log
  <parse>
    @type regexp
    expression /^(?<module>[^ ]*) *(?<time>[\d ,-:]*) (?<severity>[^ ]*) *(?<file>[\w\.]*):(?<function>[\w_]*) (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L
  </parse>
</filter>

它可以（一种），因为它按预期提取了字段。也就是说，它消除了以前的所有其他领域。

过滤器之前的示例消息：

filter.app.backend.app_logs: {"docker.container_name":"intranet-worker","docker.container_id":"98b7784f27f93a056c05b4c5066c06cb5e23d7eeb436a6e4a66cdf8ff045d29f","time":"2022-06-10T17:00:00.248932151Z","log":"org-worker  2022-06-10 19:00:00,248 INFO     briefings.py:check_expired_registrations Checking for expired registrations\n","docker.container_image":"registry.my-org.de/org-it-infrastructure/org-fastapi-backend/backend-worker:v0-7-11","stream":"stdout","docker.container_started":"2022-06-10T14:57:27.925959889Z"}

在过滤器之后，消息看起来像这样（略有不同，但流相同）：

filter.app.backend.app_logs: {"module":"mksp-api","severity":"DEBUG","file":"authToken.py","function":"verify_token","message":"Token is valid, checking permission"}

因此，只保留了解析的字段，其余的删除。我可以以某种方式使用该过滤器将字段添加到消息中而不是替换吗？

原文

I'm getting a message into fluentd with a few keys already populated from previous stages (fluent-bit on another host). I'm trying to parse the content of the log field as follows:

# Parse app_logs
<filter filter.app.backend.app_logs>
  @type parser
  key_name log
  <parse>
    @type regexp
    expression /^(?<module>[^ ]*) *(?<time>[\d ,-:]*) (?<severity>[^ ]*) *(?<file>[\w\.]*):(?<function>[\w_]*) (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L
  </parse>
</filter>

It works (kind of), as it extracts the fields as expected. That said, it removes all the other fields that were there before.

Example message before the filter:

filter.app.backend.app_logs: {"docker.container_name":"intranet-worker","docker.container_id":"98b7784f27f93a056c05b4c5066c06cb5e23d7eeb436a6e4a66cdf8ff045d29f","time":"2022-06-10T17:00:00.248932151Z","log":"org-worker  2022-06-10 19:00:00,248 INFO     briefings.py:check_expired_registrations Checking for expired registrations\n","docker.container_image":"registry.my-org.de/org-it-infrastructure/org-fastapi-backend/backend-worker:v0-7-11","stream":"stdout","docker.container_started":"2022-06-10T14:57:27.925959889Z"}

After the filter, the message looks like this (its a slightly different one, but same stream):

filter.app.backend.app_logs: {"module":"mksp-api","severity":"DEBUG","file":"authToken.py","function":"verify_token","message":"Token is valid, checking permission"}

So only the parsed fields are kept, the rest is removed. Can I somehow use that filter to add the fields to the message, instead of replacing it?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

愛上了 2025-02-13 08:42:27

实际上，文档中描述了这种情况，其不是regexp文档的一部分，而是相应的 parser滤波器文档：

rexive_data
将原始键值对保持在分析结果中。

因此，以下配置有效：

<filter filter.app.backend.app_logs>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type regexp
    expression /^(?<module>[^ ]*) *(?<time>[\d ,-:]*) (?<severity>[^ ]*) *(?<file>[\w\.]*):(?<function>[\w_]*) (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L
  </parse>
</filter>

Actually, this scenario is described in the documentation, its not part of the regexp documentation but of the corresponding parser filter documentation:

reserve_data
Keeps the original key-value pair in the parsed result.

Therefore, the following configuration works:

<filter filter.app.backend.app_logs>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type regexp
    expression /^(?<module>[^ ]*) *(?<time>[\d ,-:]*) (?<severity>[^ ]*) *(?<file>[\w\.]*):(?<function>[\w_]*) (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L
  </parse>
</filter>

回复收藏 0 原文

~没有更多了~