如何用flutterbit分割日志（键）字段？

发布于 2025-01-18 10:11:26 字数 392 浏览 1 评论 0原文

我们使用 FluentBit 将 node.js 代码发送到 OpenSearch。我们遇到问题是因为 log 键包含嵌套值作为 message。我们需要拆分以下日志消息中提到的值 -

log-    {"level":"info","message":"\"{\"method:\" GET , \"url:\" / , \"status:\" 404 , \"responseTime:\" 0.545 ms , \"responseContentLength:\" 39}\"\n","timestamp":"2022-04-01T12:48:37.091Z"}

我们需要将每个字段拆分为单独的 -

level: info 方法：获取状态：404

原文

We are sending node.js code to OpenSearch using FluentBit. We are having issues because log key contains nested value as message. We need to split the values mentioned in the below log message -

log-    {"level":"info","message":"\"{\"method:\" GET , \"url:\" / , \"status:\" 404 , \"responseTime:\" 0.545 ms , \"responseContentLength:\" 39}\"\n","timestamp":"2022-04-01T12:48:37.091Z"}

We need to split each and every field as separate -

level: info
method: GET
status: 404

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

冷清清 2025-01-25 10:11:26

，解决方案分为两个部分：

在 Fluent-bit 配置文件中添加 Kubernetes 过滤器
更正我们的 API/微服务中的 json 日志记录

我们遇到了类似的问题看起来与 json 格式相关，特别是 message 字段（参见下面的第 2 点）

1.在 Fluent-bit 配置文件中添加 Kubernetes 过滤器

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Merge_Log_Key log_processed
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

现在，这会将 json 输出拆分到新字段中：

log = "{original dict as string}"
log_processed.level = "info"
log_processed.message = 等等

2.更正我们 API 中的 json 日志记录

json 中的 message 字段似乎输出为 String，而不是 json目的。

即你有：

{
  "level": "info",
  "message": "\"{\"method:\" GET , \"url:\" / , \"status:\" 404 , \"responseTime:\" 0.545 ms , \"responseContentLength:\" 39}\"\n",
  "timestamp": "2022-04-01T12:48:37.091Z"
}

但你可能想要这个：

{
  "level": "info",
  "message": {
    "method": "GET",
    "url": "/",
    "status": "404",
    "responseTime": "0.545 ms",
    "responseContentLength": 39
  },
  "timestamp": "2022-04-01T12:48:37.091Z"
}

请注意，我在这里假设的数据类型只是为了演示问题。

一些相关的阅读/链接：

We had a similar problem and there was two parts to the solution:

Add Kubernetes filter in the Fluent-bit config file
Correct the json logging from our APIs/microservices

Though your issue looks json format related, specifically for the message field (see point 2 below)

1. Add Kubernetes filter in the Fluent-bit config file

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Merge_Log_Key log_processed
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

Now this splits the json output in new fields:

log = "{original dict as string}"
log_processed.level = "info"
log_processed.message = etc.

2. Correct the json logging from our APIs

It looks like the message field in your json is outputting as a String, not a json object.

i.e. you have:

{
  "level": "info",
  "message": "\"{\"method:\" GET , \"url:\" / , \"status:\" 404 , \"responseTime:\" 0.545 ms , \"responseContentLength:\" 39}\"\n",
  "timestamp": "2022-04-01T12:48:37.091Z"
}

But you may want this instead:

{
  "level": "info",
  "message": {
    "method": "GET",
    "url": "/",
    "status": "404",
    "responseTime": "0.545 ms",
    "responseContentLength": 39
  },
  "timestamp": "2022-04-01T12:48:37.091Z"
}

Please note that I've assumed datatypes here to demonstrate the issue only.

Some relevant reading/links:

回复收藏 0 原文

~没有更多了~