更新：流利的位分析JSON log作为文本

Question

我正在使用Fluentd发送日志。应用日志以JSON格式。这是Kibana中的其中之一

{"@timestamp":"2022-05-06T06:02:10.669Z", "log.level": "INFO", "message":"INFO Health check ok", "ecs.version": "1.2.0","service.name":"spring-boot-application","event.dataset":"spring-boot-application","process.thread.name":"http-nio-9001-exec-7","log.logger":"com.app.designer.rules.controller.RulesController","transaction.id":"15e36b5ef6cc69dd","trace.id":"fd275718e25c061d309954773985b101"}

，我可以看到日志

JSON日志是log键的值。 但是，我们希望JSON日志键值为字段和值 请建议。 流利位配置映射是：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-designer
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    error
        Parsers_File parsers.conf

  fluent-bit-input.conf: |
    [INPUT]
        Name              tail
        Path              /app/logs/designer-logs.log.json
        Multiline         On
        Parser_Firstline  multiline_pattern
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Path_key          FileName

  fluent-bit-filter.conf: |
    [FILTER]
        Name    modify
        Match   *
        Add     PodName ${POD_NAME}
        Add     Namespace ${NAMESPACE_NAME}

  fluent-bit-output.conf: |
    [OUTPUT]   
       Name    forward
       Match   *
       Host    {{ .Values.efkCluster }}-fluentd-aggregator.{{ .Values.efkCluster }}
       Port    24224


  fluent-bit.conf: |
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf

  parsers.conf: |
    [PARSER]
        Name         multiline_pattern
        Format       regex
        Regex        ^\[(?<LogDate>\d{2}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{3}\+\d{4})\] (?<LogLevel>[^ ]*) * (?<Thread>\S+) (?<Username>[^\s]*?) (?<Logger>[A-Za-z0-9$_.]+) -(\r\n|\r|\n)*(?<Message>.*)
        time_key     LogDate
        time_format  %d-%m-%y %H:%M:%S.%L%z

Answer 1

我将解析器的配置从正则表达式更改为 JSON解析器使其正常工作。

请找到Fluentbit Confgmap：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-designer
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    error
        Parsers_File parsers.conf

  fluent-bit-input.conf: |
    [INPUT]
        Name              tail
        Path              /app/logs/app.log
        Parser            json_parser
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Path_key          FileName

  fluent-bit-filter.conf: |
    [FILTER]
        Name    modify
        Match   *
        Add     PodName ${POD_NAME}
        Add     Namespace ${NAMESPACE_NAME}

  fluent-bit-output.conf: |
    [OUTPUT]   
       Name    forward
       Match   *
       Host    {{ .Values.efkCluster }}-fluentd-aggregator.{{ .Values.efkCluster }}
       Port    24224


  fluent-bit.conf: |
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf

  parsers.conf: |
    [PARSER]
        Name         json_parser
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped_utf8    log    do_next
        Decode_Field_As   json       log

在Kibana