rancher dashboard面板提示etcd组件不健康

Question

问题描述

不健康的组件蛮多的

• 警告: 组件etcd-2不健康
• 警告: 组件scheduler不健康
• 警告: 组件controller-manager不健康

看了下controller-manaer的容器日志

W0424 18:27:54.060616       1 garbagecollector.go:649] failed to discover preferred resources: Get https://127.0.0.1:6443/api?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0424 18:27:54.344478       1 garbagecollector.go:175] no resources reported by discovery, skipping garbage collector sync
E0424 18:27:54.368949       1 resource_quota_controller.go:430] failed to discover resources: Get https://127.0.0.1:6443/api?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E0424 18:27:58.743291       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: the server was unable to return a response in the time allotted, but may still be processing the request (get endpoints kube-controller-manager)
E0424 18:27:58.744595       1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'rancher-master1_6b3b4fc3-38dc-11e9-bedd-00163e0d2fc2 stopped leading'
I0424 18:27:58.749907       1 leaderelection.go:231] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition
I0424 18:27:58.758103       1 resource_quota_controller.go:297] Shutting down resource quota controller
F0424 18:27:58.758343       1 controllermanager.go:224] leaderelection lost
I0424 18:27:58.882216       1 replica_set.go:194] Shutting down replicaset controller
I0424 18:27:58.961055       1 job_controller.go:155] Shutting down job controller

etcd日志

2019-04-25 05:37:21.294182 W | rafthttp: health check for peer fb97a3b8fd75069b could not connect: dial tcp 10.27.234.189:2380: getsockopt: connection refused
2019-04-25 05:37:21.849407 W | etcdserver: failed to reach the peerURL(https://10.27.234.189:2380) of member fb97a3b8fd75069b (Get https://10.27.234.189:2380/version: dial tcp 10.27.234.189:2380: getsockopt: connection refused)
2019-04-25 05:37:21.849434 W | etcdserver: cannot get the version of member fb97a3b8fd75069b (Get https://10.27.234.189:2380/version: dial tcp 10.27.234.189:2380: getsockopt: connection refused)
2019-04-25 05:37:25.851488 W | etcdserver: failed to reach the peerURL(https://10.27.234.189:2380) of member fb97a3b8fd75069b (Get https://10.27.234.189:2380/version: dial tcp 10.27.234.189:2380: getsockopt: connection refused)
2019-04-25 05:37:25.851522 W | etcdserver: cannot get the version of member fb97a3b8fd75069b (Get https://10.27.234.189:2380/version: dial tcp 10.27.234.189:2380: getsockopt: connection refused)
2019-04-25 05:37:26.294527 W | rafthttp: health check for peer fb97a3b8fd75069b could not connect: dial tcp 10.27.234.189:2380: getsockopt: connection refused
2019-04-25 05:37:29.854396 W | etcdserver: failed to reach the peerURL(https://10.27.234.189:2380) of member fb97a3b8fd75069b (Get https://10.27.234.189:2380/version: dial tcp 10.27.234.189:2380: getsockopt: connection refused)
2019-04-25 05:37:29.854431 W | etcdserver: cannot get the version of member fb97a3b8fd75069b (Get https://10.27.234.189:2380/version: dial tcp 10.27.234.189:2380: getsockopt: connection refused)
2019-04-25 05:37:31.295014 W | rafthttp: health check for peer fb97a3b8fd75069b could not connect: dial tcp 10.27.234.189:2380: getsockopt: connection refused

不知如何排查问题?