Filebeat POD 由于 OOM 问题而被杀死

Question

我们在 Kubernetes 集群中运行工作负载，为了将应用程序日志发送到集群之外，我们使用 Filebeat 作为守护进程集，该守护进程在专用命名空间上运行。

Filebeat 将数据发送到在负载均衡器后面运行的 Logstash 集群。

我们发现一些 Filebeat pod 由于 OOM 而重新启动，我们正在尝试调查该问题，但自从在 Pod 中运行以来，我们陷入了僵局。

Filebeat 配置：

filebeat.inputs:
- type: container
  paths:
    - /var/lib/docker/containers/*/*.log
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
    - drop_event.when.not.or:
        - equals.kubernetes.labels.console-log: "enabled"
  exclude_lines: ["34mdebug"]
- type: log
  paths:
    - /var/log/node-log/messages
  include_lines:
    - 'kubelet:'
  exclude_lines:
    - 'CNI Plugin version'
  tags:
    - "kubelet-logs"
- type: log
  paths:
    - /mnt/logs/*.log
  tags:
    - "additional-logs"
processors:
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.name", "agent.type", "agent.version", "container.id", "container.image.name", "container.runtime", "kubernetes.container.image", "kubernetes.container.name", "kubernetes.namespace_labels.env", "kubernetes.namespace_labels.team", "kubernetes.namespace_uid", "kubernetes.node.hostname", "kubernetes.node.labels.beta_kubernetes_io/arch", "kubernetes.node.labels.beta_kubernetes_io/instance-type", "kubernetes.node.labels.beta_kubernetes_io/os", "kubernetes.node.labels.failure-domain_beta_kubernetes_io/region", "kubernetes.node.labels.failure-domain_beta_kubernetes_io/zone", "kubernetes.node.labels.kubernetes_io/arch", "kubernetes.node.labels.kubernetes_io/hostname", "kubernetes.node.labels.kubernetes_io/os", "kubernetes.node.labels.spotinst_io/node-lifecycle", "kubernetes.node.labels.vng-group", "kubernetes.node.name", "kubernetes.node.uid", "ecs.version", "input.type", "kubernetes.labels.app_kubernetes_io/managed-by", "kubernetes.labels.console-log", "kubernetes.pod.uid", "kubernetes.labels.app_kubernetes_io/name", "kubernetes.node.labels.node_kubernetes_io/instance-type", "kubernetes.node.labels.topology_kubernetes_io/region", "kubernetes.node.labels.topology_kubernetes_io/zone"]

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}      

fields:
  ekscluster: "eks-prod"

output.logstash:
  hosts: ["elk-logstash-internal.amazonaws.com:5043"]
  timeout: 60

在检查 pod 日志时，我们看到此错误：

Mar  3 08:35:57 ip-172-20-95-255 kernel: memory: usage 204760kB, limit 204800kB, failcnt 57072
Mar  3 08:35:57 ip-172-20-95-255 kernel: memory+swap: usage 204760kB, limit 9007199254740988kB, failcnt 0
Mar  3 08:35:57 ip-172-20-95-255 kernel: kmem: usage 2252kB, limit 9007199254740988kB, failcnt 0
Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB acti
ve_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f/0575ed6294aea6f4300b85da751d0239aa4d895efba7be3bdc7e4b9cdf0ba53f: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped
_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f/cabb8b1f5425af10732fb04bee3437ad3c5d5c29cd7c2ebe2b5f9ddd3f18fc7c: cache:92KB rss:0KB rss_huge:0KB shmem:0KB mappe
d_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f/083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918: cache:116KB rss:202276KB rss_huge:20480KB shmem
:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:202408KB inactive_file:56KB active_file:0KB unevictable:0KB
Mar  3 08:35:58 ip-172-20-95-255 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mar  3 08:35:58 ip-172-20-95-255 kernel: [ 4283]     0  4283      242        1       3       2        0          -998 pause
Mar  3 08:35:58 ip-172-20-95-255 kernel: [ 1278]     0  1278     1092      160       8       3        0           975 tini
Mar  3 08:35:58 ip-172-20-95-255 kernel: [ 1303]     0  1303   275030    60940     168       4        0           975 filebeat
Mar  3 08:35:58 ip-172-20-95-255 kernel: Memory cgroup out of memory: Kill process 1303 (filebeat) score 2164 or sacrifice child
Mar  3 08:35:58 ip-172-20-95-255 kernel: Killed process 1303 (filebeat) total-vm:1100120kB, anon-rss:191432kB, file-rss:52328kB, shmem-rss:0kB
Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.151720    2946 setters.go:77] Using node IP: "172.20.95.255"
Mar  3 08:35:59 ip-172-20-95-255 dockerd: time="2022-03-03T08:35:59.249926267Z" level=info msg="ignoring event" container=083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918 module=libcontainerd namespace=moby topic=/tasks/de
lete type="*events.TaskDelete"
Mar  3 08:35:59 ip-172-20-95-255 containerd: time="2022-03-03T08:35:59.251109262Z" level=info msg="shim disconnected" id=083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918
Mar  3 08:35:59 ip-172-20-95-255 containerd: time="2022-03-03T08:35:59.251188224Z" level=error msg="copy shim log" error="read /proc/self/fd/66: file already closed"
Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.385350    2946 kubelet.go:1982] SyncLoop (PLEG): "filebeat-f4h8x_filebeat(20b62bc0-c582-41ef-876a-58df4f4e073f)", event: &pleg.PodLifecycleEvent{ID:"20b62bc0-c582-41ef-876a-58df4f4e
073f", Type:"ContainerDied", Data:"083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918"}
Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.386181    2946 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: 083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918
Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.387106    2946 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: a81a6bbc9d15d2ea0f51c7aad5eb1fa092c97b9826c86e0701ca93914ee5ce97
Mar  3 08:35:59 ip-172-20-95-255 containerd: time="2022-03-03T08:35:59.497230474Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/e956e035777aa655cb88d98efe5bb0a27208fd6ab67851
536a3cc3407d484bb3 pid=12405
Mar  3 08:36:00 ip-172-20-95-255 kubelet: I0303 08:36:00.413098    2946 kubelet.go:1982] SyncLoop (PLEG): "filebeat-f4h8x_filebeat(20b62bc0-c582-41ef-876a-58df4f4e073f)", event: &pleg.PodLifecycleEvent{ID:"20b62bc0-c582-41ef-876a-58df4f4e
073f", Type:"ContainerStarted", Data:"e956e035777aa655cb88d98efe5bb0a27208fd6ab67851536a3cc3407d484bb3"}

内存和 CPU 限制是

  limits:
            cpu: 250m
            memory: 200Mi
          requests:
            cpu: 40m
            memory: 100Mi

Filebeat 版本 7.11.2

根本原因是什么，有什么办法吗调查该问题。