dcgm-exporter daemonset Startup error Failed to pass the health check

[guoliangmiao]

Ask your question

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dcgm-exporter
namespace: monitoring
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: dcgm-exporter
app.kubernetes.io/instance: dcgm-exporter
app.kubernetes.io/name: dcgm-exporter
template:
metadata:
labels:
app.kubernetes.io/component: dcgm-exporter
app.kubernetes.io/instance: dcgm-exporter
app.kubernetes.io/name: dcgm-exporter
namespace: monitoring
spec:
containers:
- args:
- '-f'
- /etc/dcgm-exporter/dcp-metrics-included.csv
env:
- name: DCGM_EXPORTER_KUBERNETES
value: 'true'
- name: DCGM_EXPORTER_LISTEN
value: ':9400'
- name: DCGM_EXPORTER_DEBUG
value: 'true'
image: >-
crpi-y1mch6v3dn8zzsm8.cn-hangzhou.personal.cr.aliyuncs.com/cloudsway/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 9400
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: exporter
ports:
- containerPort: 9400
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 9400
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: '1'
memory: 2Gi
securityContext:
capabilities:
add:
- SYS_ADMIN
runAsNonRoot: false
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/pod-resources
name: pod-gpu-resources
readOnly: true

=================================================================================================
2024/09/26 04:12:34 maxprocs: Leaving GOMAXPROCS=192: CPU quota undefined
2024-09-26T12:12:34.555993494+08:00 time="2024-09-26T04:12:34Z" level=info msg="Starting dcgm-exporter"
2024-09-26T12:12:34.556005237+08:00 time="2024-09-26T04:12:34Z" level=debug msg="Debug output is enabled"
2024-09-26T12:12:34.556632224+08:00 time="2024-09-26T04:12:34Z" level=debug msg="Command line: /usr/bin/dcgm-exporter -f /etc/dcgm-exporter/dcp-metrics-included.csv"
time="2024-09-26T04:12:34Z" level=debug msg="Loaded configuration" dump="&{CollectorsFile:/etc/dcgm-exporter/dcp-metrics-included.csv Address::9400 CollectInterval:30000 Kubernetes:true KubernetesGPUIdType:uid CollectDCP:true UseOldNamespace:false UseRemoteHE:false RemoteHEInfo:localhost:5555 GPUDevices:{Flex:true MajorRange:[] MinorRange:[]} SwitchDevices:{Flex:true MajorRange:[] MinorRange:[]} CPUDevices:{Flex:true MajorRange:[] MinorRange:[]} NoHostname:false UseFakeGPUs:false ConfigMapData:none MetricGroups:[] WebSystemdSocket:false WebConfigFile: XIDCountWindowSize:300000 ReplaceBlanksInModelName:false Debug:true ClockEventsCountWindowSize:300000 EnableDCGMLog:false DCGMLogLevel:NONE PodResourcesKubeletSocket:/var/lib/kubelet/pod-resources/kubelet.sock HPCJobMappingDir: NvidiaResourceNames:[]}"
time="2024-09-26T04:12:34Z" level=info msg="DCGM successfully initialized!"
2024-09-26T12:12:34.948180836+08:00 time="2024-09-26T04:12:34Z" level=info msg="Not collecting DCP metrics: This request is serviced by a module of DCGM that is not currently loaded"
2024-09-26T12:12:34.948188060+08:00 time="2024-09-26T04:12:34Z" level=info msg="Falling back to metric file '/etc/dcgm-exporter/dcp-metrics-included.csv'"
2024-09-26T12:12:34.948330476+08:00 time="2024-09-26T04:12:34Z" level=warning msg="Skipping line 20 ('DCGM_FI_PROF_GR_ENGINE_ACTIVE'): metric not enabled"
2024-09-26T12:12:34.948341428+08:00 time="2024-09-26T04:12:34Z" level=warning msg="Skipping line 21 ('DCGM_FI_PROF_PIPE_TENSOR_ACTIVE'): metric not enabled"
2024-09-26T12:12:34.948345155+08:00 time="2024-09-26T04:12:34Z" level=warning msg="Skipping line 22 ('DCGM_FI_PROF_DRAM_ACTIVE'): metric not enabled"
2024-09-26T12:12:34.948359042+08:00 time="2024-09-26T04:12:34Z" level=warning msg="Skipping line 23 ('DCGM_FI_PROF_PCIE_TX_BYTES'): metric not enabled"
time="2024-09-26T04:12:34Z" level=warning msg="Skipping line 24 ('DCGM_FI_PROF_PCIE_RX_BYTES'): metric not enabled"
2024-09-26T12:12:34.948489134+08:00 time="2024-09-26T04:12:34Z" level=info msg="Initializing system entities of type: GPU"
time="2024-09-26T04:12:35Z" level=debug msg="System entities of type GPU initialized"
2024-09-26T12:12:35.285853835+08:00 time="2024-09-26T04:12:35Z" level=info msg="Not collecting NvSwitch metrics; no fields to watch for device type: 3"
time="2024-09-26T04:12:35Z" level=info msg="Not collecting NvLink metrics; no fields to watch for device type: 6"
2024-09-26T12:12:35.285870137+08:00 time="2024-09-26T04:12:35Z" level=info msg="Not collecting CPU metrics; no fields to watch for device type: 7"
time="2024-09-26T04:12:35Z" level=info msg="Not collecting CPU Core metrics; no fields to watch for device type: 8"
time="2024-09-26T04:12:35Z" level=debug msg="Counters are initialized" dump="[{FieldID:100 FieldName:DCGM_FI_DEV_SM_CLOCK PromType:gauge Help:SM clock frequency (in MHz).} {FieldID:101 FieldName:DCGM_FI_DEV_MEM_CLOCK PromType:gauge Help:Memory clock frequency (in MHz).} {FieldID:140 FieldName:DCGM_FI_DEV_MEMORY_TEMP PromType:gauge Help:Memory temperature (in C).} {FieldID:150 FieldName:DCGM_FI_DEV_GPU_TEMP PromType:gauge Help:GPU temperature (in C).} {FieldID:155 FieldName:DCGM_FI_DEV_POWER_USAGE PromType:gauge Help:Power draw (in W).} {FieldID:156 FieldName:DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION PromType:counter Help:Total energy consumption since boot (in mJ).} {FieldID:202 FieldName:DCGM_FI_DEV_PCIE_REPLAY_COUNTER PromType:counter Help:Total number of PCIe retries.} {FieldID:203 FieldName:DCGM_FI_DEV_GPU_UTIL PromType:gauge Help:GPU utilization (in %).} {FieldID:204 FieldName:DCGM_FI_DEV_MEM_COPY_UTIL PromType:gauge Help:Memory utilization (in %).} {FieldID:206 FieldName:DCGM_FI_DEV_ENC_UTIL PromType:gauge Help:Encoder utilization (in %).} {FieldID:207 FieldName:DCGM_FI_DEV_DEC_UTIL PromType:gauge Help:Decoder utilization (in %).} {FieldID:230 FieldName:DCGM_FI_DEV_XID_ERRORS PromType:gauge Help:Value of the last XID error encountered.} {FieldID:251 FieldName:DCGM_FI_DEV_FB_FREE PromType:gauge Help:Framebuffer memory free (in MiB).} {FieldID:252 FieldName:DCGM_FI_DEV_FB_USED PromType:gauge Help:Framebuffer memory used (in MiB).} {FieldID:449 FieldName:DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL PromType:counter Help:Total number of NVLink bandwidth counters for all lanes.} {FieldID:526 FieldName:DCGM_FI_DEV_VGPU_LICENSE_STATUS PromType:gauge Help:vGPU License status} {FieldID:393 FieldName:DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS PromType:counter Help:Number of remapped rows for uncorrectable errors} {FieldID:394 FieldName:DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS PromType:counter Help:Number of remapped rows for correctable errors} {FieldID:395 FieldName:DCGM_FI_DEV_ROW_REMAP_FAILURE PromT
ype:gauge Help:Whether remapping of rows has failed} {FieldID:1 FieldName:DCGM_FI_DRIVER_VERSION PromType:label Help:Driver Version}]"
time="2024-09-26T04:12:35Z" level=info msg="Kubernetes metrics collection enabled!"
2024-09-26T12:12:35.321315631+08:00 time="2024-09-26T04:12:35Z" level=info msg="Pipeline starting"
2024-09-26T12:12:35.321320811+08:00 time="2024-09-26T04:12:35Z" level=info msg="Starting webserver"
2024-09-26T12:12:35.321682302+08:00 time="2024-09-26T04:12:35Z" level=info msg="Listening on" address="[::]:9400"
2024-09-26T12:12:35.321690668+08:00 time="2024-09-26T04:12:35Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false

=======================================================================================================
As shown above, these are the key parts of the deployment files and error logs. I have enabled debug mode. However, the root cause of the issue has not yet been analyzed. Please help me with this.

[nvvfedorov]

@guoliangmiao , According to the log the dcgm-exporter started.

Also, I see, that performance metrics aren't supported on your hardware: "Not collecting DCP metrics: This request is serviced by a module of DCGM that is not currently loaded".

[guoliangmiao]

I found that the solution to the problem is to add the runtime: nvidia field in the deployment file, but I don't quite understand this behavior because the default_container_runtime has already been specified as nvidia in the containerd configuration for each node.