numa机器上，docker容器中进程的cpuset设置

Question

机器配置 2个numa, 48 cpu

root@yc:/# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35
node 0 size: 32109 MB
node 0 free: 3384 MB
node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42 43 44 45 46 47
node 1 size: 32251 MB
node 1 free: 3228 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

docker run 启动一个容器，比如说 ubuntu。

root@yc:/# docker run -it  ubuntu:bionic bash

容器内查看 PID 的 status，发现这里竟然设置了 Cpus_allowed_list，同时 memory 也绑定只能从 numa 1 分配。

root@78fc8c193ef3:/# cat /proc/1/status
....省略...
Cpus_allowed:   fd70,00fd7000
Cpus_allowed_list:      12-14,16,18-23,36-38,40,42-47
Mems_allowed:   00000000,00000002
Mems_allowed_list:      1
voluntary_ctxt_switches:        82
nonvoluntary_ctxt_switches:     1

但从宿主机上用 docker inspect 看该容器，可以看到并无限制

root@yc:/# docker inspect 78fc8c193ef3
[
    ... 省略
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",              ----> 并无限制
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,               
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,

        // 省略
]

我的问题是，容器启动后的 cpuset 是谁设置的，我原本预期是全部的 cpu 0-47

docker version 是 19.03.5

补充: 确认了一个细节，刚启动时, cpu 还是 0-47 但几秒过后 被改成了"12-14,16,18-23,36-38,40,42-47"

哎，难道又要去磕内核源码了

额 破案了，是有其他服务进行了numa平衡 ，orz

Answer 1

破案了，是有其他服务进行了numa平衡 ，orz