流量猛增期间keepalived失效

Question

公司两台服务器运行nginx，使用keepalived做双机。使用keepalived双master方式。

近期遇到的一个问题是，网站流量猛增时，keepalived开始不停的切换master 和 backup，虚拟ip也访问不到。重启keepalived后恢复。三分钟的时间里，VIP切换了3次，期间外部无法访问VIP。故障从forcing a new MASTER election，不知道为什么要重选MASTER……

以下是keepalived日志。

 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) forcing a new MASTER election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Transition to MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received higher prio advert
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering BACKUP STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) forcing a new MASTER election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Transition to MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received higher prio advert
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering BACKUP STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) forcing a new MASTER election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Transition to MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received higher prio advert
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering BACKUP STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) forcing a new MASTER election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Transition to MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received higher prio advert
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering BACKUP STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) forcing a new MASTER election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Transition to MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) setting protocol VIPs.
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Sending gratuitous ARPs on em1 for 192.168.1.28
 localhost Keepalived_healthcheckers[19607]: Netlink reflector reports IP 192.168.1.28 added
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received higher prio advert
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering BACKUP STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) removing protocol VIPs.
 localhost Keepalived_healthcheckers[19607]: Netlink reflector reports IP 192.168.1.28 removed
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) forcing a new MASTER election
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Transition to MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering MASTER STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) setting protocol VIPs.
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Sending gratuitous ARPs on em1 for 192.168.1.28
 localhost Keepalived_healthcheckers[19607]: Netlink reflector reports IP 192.168.1.28 added
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Received higher prio advert
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) Entering BACKUP STATE
 localhost Keepalived_vrrp[19608]: VRRP_Instance(VI_1) removing protocol VIPs.
 localhost Keepalived_healthcheckers[19607]: Netlink reflector reports IP 192.168.1.28 removed

以下是keepalived配置

! Configuration File for keepalived


global_defs {
   notification_email {
   yiwei@xx.cn
   }
   notification_email_from root@localhost.localdomain
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id node2
}
vrrp_script chk_http_port {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight 2
}


vrrp_instance VI_1 {
    state MASTER
    interface em1
    virtual_router_id 51
    mcast_src_ip 192.168.1.27
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 12345QWER
    }
    virtual_ipaddress {
        192.168.1..28
    }
   track_script {
        chk_http_port
        }
}

nginx_check.sh

A=`ps -C nginx --no-header |wc -l`

if [ $A -eq 0 ]
then
        /usr/sbin/nginx 
        sleep 3
        if [ `ps -C nginx --no-header |wc -l` -eq 0 ]
        then
                service keepalived stop 
        fi 
fi

Answer 1

多谢帮助。我后来把检测脚本改了一下。不监控端口也不检测网页，而是该做监控nginx进程，问题就解决了。可能是瞬间的高流量导致网页端口检测超时，进而keepalived就出问题了。

Answer 2

如果有什么问题再回复我看看

Answer 3

好的。我试试。

Answer 4

你好，你把权重调整一下，把主服务器调大点试试！暂时把nginx脚本暂时去掉再试试，如果没有问题在把脚本加上去！