Fluentd -Kubernetes -ParserError错误=“模式与数据不匹配”。

Question

我正在尝试将 Kubernetes 日志从容器重定向到 OpenSearch。 但日期总是有一些错误。我做错了什么？

Docker 日志示例：

{"log":"time=\"2022-04-01T10:02:31Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker2\n","stream":"stderr","time":"2022-04-01T10:02:31.230191143Z"}
{"log":"E0401 10:02:31.230146       1 backup_controller.go:153] longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\n","stream":"stderr","time":"2022-04-01T10:02:31.230214608Z"}
{"log":"time=\"2022-04-01T10:02:31Z\" level=warning msg=\"Dropping Longhorn backup longhorn-system/backup-989764daba094e0d out of the queue\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker2\n","stream":"stderr","time":"2022-04-01T10:02:31.230218285Z"}

Fluentd 输出：

fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524389862 +0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524404952 +0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.538539106 +0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}

配置：

<source>
  @type tail
  @id tail_all_container_logs
  @label @FLUENTD.OPENSEARCH
  path /var/log/containers/longhorn*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  exclude_path "#{ENV['FLUENT_ALL_CONTAINERS_TAIL_EXCLUDE_PATHS']}"
  <parse>
      @type json
  </parse>
</source>

<filter kubernetes.**>
    @type parser
    key_name log
    <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        timezone +00:00
    </parse>
</filter>

Answer 1

日志的日期/时间是2022-04-01T09：54：33Z请注意没有毫秒。

而在您的配置中有time_format％y-％m-％dt％h：％m：％s.％n％z

％n-分数秒数字，默认值为9位数字（纳秒）。

尝试删除。％n零件以与日志中的时间格式匹配。这将是：

time_format％y-％m-％dt％h：％m：％s％z

有关时间格式语法的更多信息，请参考此 page 。