The Prometheus receiver of OpenTelemetry Collector failed while attempting to capture kube state metrics.

[xxlaila]

The Prometheus receiver of OpenTelemetry Collector failed while attempting to capture kube state metrics. Indicators can be manually accessed, but OpenTelemetry Collector cannot successfully capture data. The k8s cluster network is a VPC network, and there are no issues with network communication,

配置文件

# cat cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
  namespace: skywalking
data:
  otel-collector-config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'kubernetes-cadvisor'
              scheme: https
              tls_config:                                      # 证书配置，忽略证书验证。
                insecure_skip_verify: true
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              authorization:                                   # 认证配置
                credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              kubernetes_sd_configs:
                - role: node
              relabel_configs:
                - action: labelmap
                  regex: __meta_kubernetes_node_label_(.+)
                - source_labels: []
                  target_label: cluster
                  replacement: tke-app-test
                - target_label: __address__
                  replacement: kubernetes.default.svc:443
                - source_labels: [__meta_kubernetes_node_name]
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
                - source_labels: [instance]
                  separator: ;
                  regex: (.+)
                  target_label: node
                  replacement: $$1
                  action: replace
            - job_name: kube-state-metrics
              scrape_interval: 30s
              scrape_timeout: 20s
              scheme: https
              metrics_path: /metrics
              tls_config:                                      # 证书配置，忽略证书验证。
                insecure_skip_verify: true
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              authorization:                                   # 认证配置
                credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              kubernetes_sd_configs:
              - role: endpoints
              relabel_configs:
              - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
                regex: kube-state-metrics
                replacement: $$1
                action: keep
              - action: labelmap
                regex: __meta_kubernetes_service_label_(.+)
              - source_labels: []
                target_label: cluster
                replacement: tke-app-test
                      
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            endpoint: ${env:MY_POD_IP}:4318

    processors:
      batch:
        timeout: 30s
        
    exporters:
      otlp:
        endpoint: "http://oap-svc:11800" # skywalking oap后端地址  oap-svc:11800,需要修改
        tls:
          insecure: true
        timeout: 200s
        sending_queue:  # 发送队列
          enabled: true
          num_consumers: 50 # 消费者数量
          queue_size: 10000 # 队列长度
        retry_on_failure:
          enabled: true # 是否重试
          initial_interval: 5s # 初始重试间隔
          max_interval: 30s # 最大重试间隔
          max_elapsed_time: 300s # 最大重试时间
      logging:                           # 日志输出，开启debug调试
        loglevel: info
    service:
      # extensions: [memory_ballast]
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [batch]
          exporters: [otlp, logging]

权限文件

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: skywalking-oap
  name: otel-collector
  namespace: skywalking
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: otel-collector
  namespace: skywalking
  labels:
    app: otel-collector
rules:
- apiGroups: [""]
  resources: ["pods", "endpoints", "services", "nodes", "nodes/proxy", "nodes/metrics", "namespaces", "deployment", "statefulset", "daemonset", "replicasets", "nodes/stats", "pods/status", "resourcequotas", "replicationcontrollers/status", "namespaces/status", "nodes/spec", "events"]
  verbs: ["get", "watch", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
  labels:
    app: otel-collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-collector
subjects:
- kind: ServiceAccount
  name: otel-collector
  namespace: skywalking

otel-collector 报错

2024-07-19T09:38:48.944Z        warn    internal/transaction.go:123     Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1721381928925, "target_labels": "{__name__=\"up\", app_kubernetes_io_component=\"exporter\", app_kubernetes_io_name=\"kube-state-metrics\", app_kubernetes_io_version=\"2.12.0\", cluster=\"tke-app-test\", instance=\"10.51.16.92:8080\", job=\"kube-state-metrics\"}"}
2024-07-19T09:38:52.813Z        error   exporterhelper/queue_sender.go:125      Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "max elapsed time expired rpc error: code = DeadlineExceeded desc = context deadline exceeded", "dropped_items": 23633}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).consume
        go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:125
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue[...]).Consume
        go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*QueueConsumers[...]).Start.func1
        go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/consumers.go:43