Redis在K8S中，容器使用内存超出限制的问题

Question

在K8S中部署了Redis，容器的内存上限为1G

resources:
  limits:
    memory: 1Gi

同时设置Redis maxmemory 为1G，info memory可以看到RSS内存为~550M，没有超出1G。

info memory
# Memory
used_memory:577791152
used_memory_human:551.02M
used_memory_rss:611942400
used_memory_rss_human:583.59M
used_memory_peak:578877888
used_memory_peak_human:552.06M
used_memory_peak_perc:99.81%
used_memory_overhead:32108388
used_memory_startup:786608
used_memory_dataset:545682764
used_memory_dataset_perc:94.57%
total_system_memory:67555897344
total_system_memory_human:62.92G
used_memory_lua:41984
used_memory_lua_human:41.00K
maxmemory:1000000000
maxmemory_human:953.67M
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.06
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

同时配置了k8s-prometheus采集指标，得到Redis容器的使用内存超出了1G，对应的指标是sum by(container_name) (container_memory_working_set_bytes{namespace="$namespace",container_name!="",container_name=~"$container",container_name!="POD",pod_name="$pod"})，而且还没有被OOM Killed

使用MEMORY PURGE命令从Prometheus看也并没有释放内存。

现在问题是：

1. 怎样才能释放内存？
2. container_memory_working_set_bytes与redis自己报告的used_memory_rss和used_memory是啥关系？

Answer 1

这个问题和Prometheus有关，Prometheus实际上采集到了双份指标：

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",cpu="total",endpoint="https-metrics",id="/kubepods/pod149c0943-bc55-4e5d-ac9d-59fedca56138/1e2f5a63ddba49cb13465383e0de66e50ce9f084b52975f9bea8d1d77b966b3c",image="sha256:8a74675d2c029efb8c574f6097f2494d0e6acb7427aeea874227be48d96a3fb2",instance="192.168.116.119:10250",job="expose-kubelets-metrics",name="k8s_redis-server_redis-server-0_user-data-service-test_149c0943-bc55-4e5d-ac9d-59fedca56138_0",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",endpoint="https-metrics",instance="192.168.116.119:10250",job="expose-kubelets-metrics",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"} 

所以结果被乘以2了，看上去就超出了