NodeJS CheckList

Question

有一个醉汉半夜在路灯下徘徊，路过的人奇怪地问他： 你在路灯下找什么？ 醉汉回答： 我在找我的 KEY，路人更奇怪了： 找钥匙为什么在路灯下? ，醉汉说： 因为这里最亮！ 。

很多服务端的同学在说到检查服务器状态时只知道使用 top 命令，其实情况就和上面的笑话一样，因为对于他们而言 top 是最亮的那盏路灯。

对于服务端程序员而言，完整的服务器 checklist 首推 《性能之巅》 第二章中讲述的 USE 方法 。

The USE Method provides a strategy for performing a complete check of system health, identifying common bottlenecks and errors. For each system resource, metrics for utilization, saturation and errors are identified and checked. Any issues discovered are then investigated using further strategies.

This is an example USE-based metric list for Linux operating systems (eg, Ubuntu, CentOS, Fedora). This is primarily intended for system administrators of the physical systems, who are using command line tools. Some of these metrics can be found in remote monitoring tools.

Physical Resources

component	type	metric
CPU	utilization	system-wide: vmstat 1 , "us" + "sy" + "st"; sar -u , sum fields except "%idle" and "%iowait"; dstat -c , sum fields except "idl" and "wai"; per-cpu: mpstat -P ALL 1 , sum fields except "%idle" and "%iowait"; sar -P ALL , same as mpstat ; per-process: top , "%CPU"; htop , "CPU%"; ps -o pcpu ; pidstat 1 , "%CPU"; per-kernel-thread: top / htop ("K" to toggle), where VIRT == 0 (heuristic). [1]
CPU	saturation	system-wide: vmstat 1 , "r" > CPU count [2]; sar -q , "runq-sz" > CPU count; dstat -p , "run" > CPU count; per-process: /proc/PID/schedstat 2nd field (sched_info.run_delay); perf sched latency (shows "Average" and "Maximum" delay per-schedule); dynamic tracing, eg, SystemTap schedtimes.stp "queued(us)" [3]
CPU	errors	perf (LPE) if processor specific error events (CPC) are available; eg, AMD64's "04Ah Single-bit ECC Errors Recorded by Scrubber" [4]
Memory capacity	utilization	system-wide: free -m , "Mem:" (main memory), "Swap:" (virtual memory); vmstat 1 , "free" (main memory), "swap" (virtual memory); sar -r , "%memused"; dstat -m , "free"; slabtop -s c for kmem slab usage; per-process: top / htop , "RES" (resident main memory), "VIRT" (virtual memory), "Mem" for system-wide summary
Memory capacity	saturation	system-wide: vmstat 1 , "si"/"so" (swapping); sar -B , "pgscank" + "pgscand" (scanning); sar -W ; per-process: 10th field (min_flt) from /proc/PID/stat for minor-fault rate, or dynamic tracing [5]; OOM killer: dmesg | grep killed
Memory capacity	errors	dmesg for physical failures; dynamic tracing, eg, SystemTap uprobes for failed malloc()s
Network Interfaces	utilization	sar -n DEV 1 , "rxKB/s"/max "txKB/s"/max; ip -s link , RX/TX tput / max bandwidth; /proc/net/dev, "bytes" RX/TX tput/max; nicstat "%Util" [6]
Network Interfaces	saturation	ifconfig , "overruns", "dropped"; netstat -s , "segments retransmited"; sar -n EDEV , *drop and *fifo metrics; /proc/net/dev, RX/TX "drop"; nicstat "Sat" [6]; dynamic tracing for other TCP/IP stack queueing [7]
Network Interfaces	errors	ifconfig , "errors", "dropped"; netstat -i , "RX-ERR"/"TX-ERR"; ip -s link , "errors"; sar -n EDEV , "rxerr/s" "txerr/s"; /proc/net/dev, "errs", "drop"; extra counters may be under /sys/class/net/...; dynamic tracing of driver function returns 76]
Storage device I/O	utilization	system-wide: iostat -xz 1 , "%util"; sar -d , "%util"; per-process: iotop; pidstat -d ; /proc/PID/sched "se.statistics.iowait_sum"
Storage device I/O	saturation	iostat -xnz 1 , "avgqu-sz" > 1, or high "await"; sar -d same; LPE block probes for queue length/latency; dynamic/static tracing of I/O subsystem (incl. LPE block probes)
Storage device I/O	errors	/sys/devices/.../ioerr_cnt; smartctl ; dynamic/static tracing of I/O subsystem response codes [8]
Storage capacity	utilization	swap: swapon -s ; free ; /proc/meminfo "SwapFree"/"SwapTotal"; file systems: "df -h"
Storage capacity	saturation	not sure this one makes sense - once it's full, ENOSPC
Storage capacity	errors	strace for ENOSPC; dynamic tracing for ENOSPC; /var/log/messages errs, depending on FS
Storage controller	utilization	iostat -xz 1 , sum devices and compare to known IOPS/tput limits per-card
Storage controller	saturation	see storage device saturation, ...
Storage controller	errors	see storage device errors, ...
Network controller	utilization	infer from ip -s link (or /proc/net/dev) and known controller max tput for its interfaces
Network controller	saturation	see network interface saturation, ...
Network controller	errors	see network interface errors, ...
CPU interconnect	utilization	LPE (CPC) for CPU interconnect ports, tput / max
CPU interconnect	saturation	LPE (CPC) for stall cycles
CPU interconnect	errors	LPE (CPC) for whatever is available
Memory interconnect	utilization	LPE (CPC) for memory busses, tput / max; or CPI greater than, say, 5; CPC may also have local vs remote counters
Memory interconnect	saturation	LPE (CPC) for stall cycles
Memory interconnect	errors	LPE (CPC) for whatever is available
I/O interconnect	utilization	LPE (CPC) for tput / max if available; inference via known tput from iostat/ip/...
I/O interconnect	saturation	LPE (CPC) for stall cycles
I/O interconnect	errors	LPE (CPC) for whatever is available

Software Resources

component	type	metric
Kernel mutex	utilization	With CONFIG_LOCK_STATS=y, /proc/lock_stat "holdtime-totat" / "acquisitions" (also see "holdtime-min", "holdtime-max") [8]; dynamic tracing of lock functions or instructions (maybe)
Kernel mutex	saturation	With CONFIG_LOCK_STATS=y, /proc/lock_stat "waittime-total" / "contentions" (also see "waittime-min", "waittime-max"); dynamic tracing of lock functions or instructions (maybe); spinning shows up with profiling ( perf record -a -g -F 997 ... , oprofile , dynamic tracing)
Kernel mutex	errors	dynamic tracing (eg, recusive mutex enter); other errors can cause kernel lockup/panic, debug with kdump/ crash
User mutex	utilization	valgrind --tool=drd --exclusive-threshold=... (held time); dynamic tracing of lock to unlock function time
User mutex	saturation	valgrind --tool=drd to infer contention from held time; dynamic tracing of synchronization functions for wait time; profiling (oprofile, PEL, ...) user stacks for spins
User mutex	errors	valgrind --tool=drd various errors; dynamic tracing of pthread_mutex_lock() for EAGAIN, EINVAL, EPERM, EDEADLK, ENOMEM, EOWNERDEAD, ...
Task capacity	utilization	top / htop , "Tasks" (current); sysctl kernel.threads-max , /proc/sys/kernel/threads-max (max)
Task capacity	saturation	threads blocking on memory allocation; at this point the page scanner should be running (sar -B "pgscan*"), else examine using dynamic tracing
Task capacity	errors	"can't fork()" errors; user-level threads: pthread_create() failures with EAGAIN, EINVAL, ...; kernel: dynamic tracing of kernel_thread() ENOMEM
File descriptors	utilization	system-wide: sar -v , "file-nr" vs /proc/sys/fs/file-max; dstat --fs , "files"; or just /proc/sys/fs/file-nr; per-process: ls /proc/PID/fd | wc -l vs ulimit -n
File descriptors	saturation	does this make sense? I don't think there is any queueing or blocking, other than on memory allocation.
File descriptors	errors	strace errno == EMFILE on syscalls returning fds (eg, open(), accept(), ...).

ulimit

ulimit 用于管理用户对系统资源的访问。

-a   显示目前全部限制情况
-c   设定 core 文件的最大值，单位为区块
-d   <数据节区大小> 程序数据节区的最大值，单位为 KB
-f   <文件大小> shell 所能建立的最大文件，单位为区块
-H   设定资源的硬性限制，也就是管理员所设下的限制
-m   <内存大小> 指定可使用内存的上限，单位为 KB
-n   <文件描述符数目> 指定同一时间最多可开启的 fd 数
-p   <缓冲区大小> 指定管道缓冲区的大小，单位 512 字节
-s   <堆叠大小> 指定堆叠的上限，单位为 KB
-S   设定资源的弹性限制
-t   指定 CPU 使用时间的上限，单位为秒
-u   <进程数目> 用户最多可开启的进程数目
-v   <虚拟内存大小> 指定可使用的虚拟内存上限，单位为 KB

例如：

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127988
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655360
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

注意，open socket 等资源拿到的也是 fd，所以 ulimit -n 比较小除了文件打不开，还可能建立不了 socket 链接。