kube-prometheus-stack / prometheus targets状态降低(只有普罗米修斯服务器实例启动)
在问题发生之前,我在没有Prometheus-operator的情况下安装Prometheus( https://github.com/prometheus-community/helm-charts/tree/main/main/charts/prometheus ),Kube度量都在起作用,并且目标显示出来。 然后, 我建立了aws eks并安装kube-prometheus-stack( https://github.com/prometheus-community/helm-charts/tree/main/main/main/charts/kube-prometheus-stack )和thanos(不可能/ [email  procearted] )在监视名称空间中,无论thanos源来自Thanos源状态>目标页面显示如下: targets
以“ servicemonitor/prometheus-kube-kube-kube-kube-kube-kube-kube-kube-proxy”为/0(1/9 UP)“例如,只有Prometheus-Server居住的实例是状态,其他实例已下降。 prometheus-kube-prometheus-kube-kube-proxy
在其他目标中,也只能刮擦Prometheus-Instance comprometheus-Instance cance ,而且我不知道为什么使用Kube-Prometheus-stack与Org Prometheus不同?我进入Prometheus-Pods查询其他实例( 。
如下:
prometheus-node-exporter:
prometheus:
monitor:
scrapeTimeout: "10m"
kube-state-metrics:
prometheus:
monitor:
scrapeTimeout: "10m"
prometheus:
serviceAccount:
create: true
name: "thanos-eks-sa"
prometheusSpec:
scrapeTimeout: "10m"
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelector: {}
serviceMonitorNamespaceSelector: {}
additionalScrapeConfigs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics
- job_name: 'kubernetes-nodes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
externalLabels:
cluster: dev-general
environment: dev
resources: {}
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp2
resources:
requests:
storage: 10Gi
thanos:
baseImage: improbable/thanos
version: v0.2.1
objectStorageConfig:
key: thanos.yaml
name: thanos-objstore-config
thanosService:
enabled: true
thanosServiceMonitor:
enabled: true
interval: 5s
kubeProxy:
metricsBindAddress: 0.0.0.0
kubeconfig:
enabled: true
prometheusOperator:
namespaces: ''
denyNamespaces: ''
prometheusInstanceNamespaces: ''
alertmanagerInstanceNamespaces: ''
thanosRulerInstanceNamespaces: ''
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
thanosImage:
repository: improbable/thanos
version: v0.2.1
admissionWebhooks:
patch:
podAnnotations:
linkerd.io/inject: disabled
serviceMonitor:
scrapeTimeout: "10m"
## alertmanager
alertmanager:
alertmanagerSpec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
## grafana
grafana:
sidecar:
dashboards:
multicluster:
global:
enabled: true
adminPassword: admin
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
所以我认为这是一个网络问题,但我不知道该如何解决?在同一K8的情况下,我不了解,为什么我使用Prometheus和Kube-Prometheus-Community的图表有所不同?
Before the issue happened, I install prometheus without prometheus-operator follow this chart(https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus), and the kube metrics is all worked and targets shows up.
Then,
I build up aws eks and install kube-prometheus-stack(https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) and thanos(improbable/[email protected]) in monitoring namespace, whatever the thanos source comes from, the prometheus-server > status > targets page shows as below:
targets
Take the "serviceMonitor/monitoring/prometheus-kube-prometheus-kube-proxy/0 (1/9 up)" as example, only the instance that prometheus-server lived is up status, others instance is down.
prometheus-kube-prometheus-kube-proxy
In other targets also only can scraped the prometheus-instance, and I don't know why using kube-prometheus-stack is different from org prometheus? And I go into the prometheus-pods to query other instance (https://xxx.xxx.xxx.xxx:10250/metrics) and it timeout return: wget: can't connect to remote host (xxx.xxx.xxx.xxx): Connection timed out
The override values of charts as below:
prometheus-node-exporter:
prometheus:
monitor:
scrapeTimeout: "10m"
kube-state-metrics:
prometheus:
monitor:
scrapeTimeout: "10m"
prometheus:
serviceAccount:
create: true
name: "thanos-eks-sa"
prometheusSpec:
scrapeTimeout: "10m"
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelector: {}
serviceMonitorNamespaceSelector: {}
additionalScrapeConfigs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics
- job_name: 'kubernetes-nodes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
externalLabels:
cluster: dev-general
environment: dev
resources: {}
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp2
resources:
requests:
storage: 10Gi
thanos:
baseImage: improbable/thanos
version: v0.2.1
objectStorageConfig:
key: thanos.yaml
name: thanos-objstore-config
thanosService:
enabled: true
thanosServiceMonitor:
enabled: true
interval: 5s
kubeProxy:
metricsBindAddress: 0.0.0.0
kubeconfig:
enabled: true
prometheusOperator:
namespaces: ''
denyNamespaces: ''
prometheusInstanceNamespaces: ''
alertmanagerInstanceNamespaces: ''
thanosRulerInstanceNamespaces: ''
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
thanosImage:
repository: improbable/thanos
version: v0.2.1
admissionWebhooks:
patch:
podAnnotations:
linkerd.io/inject: disabled
serviceMonitor:
scrapeTimeout: "10m"
## alertmanager
alertmanager:
alertmanagerSpec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
## grafana
grafana:
sidecar:
dashboards:
multicluster:
global:
enabled: true
adminPassword: admin
tolerations:
- key: "dedicated"
operator: "Equal"
value: "prometheus"
effect: "NoSchedule"
nodeSelector:
dedicated: prometheus
lifecycle: on-demand
So I think this is a networking issue, but I don't know how to fix it? I don't understand in the same k8s situation, why am I using the charts prometheus and kube-prometheus-stack of prometheus-community are different?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
这是因为Eks的代理指标暴露于127.0.0.1
This is because eks's proxy metrics expose on 127.0.0.1