从僵死应用的wireshark tcp包能看出可能的问题原因吗

Question

线上有一Python应用 偶尔会有僵死的情况 即进程在但是调用Api接口却提示

curl: (56) Recv failure: Connection reset by peer

在本地通过wireshark抓包 tcp包如下

"TCP","78","0.000000","51325 → 8000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=32 TSval=736973079 TSecr=0 SACK_PERM=1"
"TCP","74","0.039896","8000 → 51325 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1452 SACK_PERM=1 TSval=1018915739 TSecr=736973079 WS=128"
"TCP","66","0.040025","51325 → 8000 [ACK] Seq=1 Ack=1 Win=132480 Len=0 TSval=736973119 TSecr=1018915739"
"HTTP","280","0.040185","POST /CRFTag/textEmotionTag HTTP/1.1  (application/json)"
"TCP","280","0.221089","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736973299 TSecr=1018915739"
"TCP","280","0.362219","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736973440 TSecr=1018915739"
"TCP","280","0.804312","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736973881 TSecr=1018915739"
"TCP","280","1.486962","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736974561 TSecr=1018915739"
"TCP","74","1.624576","[TCP Spurious Retransmission] 8000 → 51325 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1452 SACK_PERM=1 TSval=1018917328 TSecr=736974561 WS=128"
"TCP","66","1.624652","[TCP Dup ACK 3#1] 51325 → 8000 [ACK] Seq=215 Ack=1 Win=132480 Len=0 TSval=736974698 TSecr=1018915739"
"TCP","280","2.651312","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736975721 TSecr=1018915739"
"TCP","74","3.844019","[TCP Spurious Retransmission] 8000 → 51325 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1452 SACK_PERM=1 TSval=1018919528 TSecr=736975721 WS=128"
"TCP","66","3.844093","[TCP Dup ACK 3#2] 51325 → 8000 [ACK] Seq=215 Ack=1 Win=132480 Len=0 TSval=736976909 TSecr=1018915739"
"TCP","280","4.779702","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736977841 TSecr=1018915739"
"TCP","280","7.872911","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736980924 TSecr=1018915739"
"TCP","280","13.848645","[TCP Retransmission] 51325 → 8000 [PSH, ACK] Seq=1 Ack=1 Win=132480 Len=214 TSval=736986884 TSecr=1018915739"
"TCP","60","13.886028","8000 → 51325 [RST] Seq=1 Win=0 Len=0"

重启应用后 就恢复正常了 此时的wireshark日志如下

"TCP","78","0.000000","50048 → 8000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=32 TSval=837789964 TSecr=0 SACK_PERM=1"
"TCP","74","0.037032","8000 → 50048 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1397333536 TSecr=837789964 WS=128"
"TCP","66","0.037111","50048 → 8000 [ACK] Seq=1 Ack=1 Win=131744 Len=0 TSval=837790001 TSecr=1397333536"
"HTTP","280","0.037243","POST /CRFTag/textEmotionTag HTTP/1.1  (application/json)"
"TCP","66","0.073054","8000 → 50048 [ACK] Seq=1 Ack=215 Win=30080 Len=0 TSval=1397333572 TSecr=837790001"
"TCP","83","0.078404","[TCP segment of a reassembled PDU]"
"TCP","66","0.078473","50048 → 8000 [ACK] Seq=215 Ack=18 Win=131744 Len=0 TSval=837790042 TSecr=1397333578"
"HTTP","365","0.079349","HTTP/1.0 200 OK  (application/json)"
"TCP","66","0.079418","50048 → 8000 [ACK] Seq=215 Ack=318 Win=131424 Len=0 TSval=837790042 TSecr=1397333578"
"TCP","66","0.079570","50048 → 8000 [FIN, ACK] Seq=215 Ack=318 Win=131424 Len=0 TSval=837790043 TSecr=1397333578"
"TCP","66","0.114651","8000 → 50048 [ACK] Seq=318 Ack=216 Win=30080 Len=0 TSval=1397333614 TSecr=837790043"

从异常的tcp包中 能看出什么端倪吗？可能是什么原因导致应用不可用的呢？下次再有僵死的情况 该怎么确认是不是这个原因呢？

Answer 1

可以看出TCP连接在建立的时候前两个握手包都好像是正常的，但从第三个包开始，就可以发现服务器似乎“收不到”第三个包，于是服务器在中间重传了若干次SYN, ACK包，而客户端也重传了好几次对服务器的SYN, ACK包的ACK包。最终连接超时，服务器重置了连接。不是很清楚到底发生了什么，可能需要在服务器端也抓个包看看。

---

是过一段时间就会发生连接的问题吗？有可能是服务器负载太大了？

Answer 2

重启 python 应用就恢复正常

那极可能是客户端的问题，应该检查应用层的代码。比如 DNS 解析错误。