Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for GPU monitoring #1601

Merged
merged 14 commits into from
Jan 30, 2025
7 changes: 7 additions & 0 deletions api/datadoghq/v2alpha1/const.go
Original file line number Diff line number Diff line change
@@ -78,6 +78,9 @@ const (
KubeServicesAndEndpointsListeners = "kube_services kube_endpoints"
EndpointsChecksConfigProvider = "endpointschecks"
ClusterAndEndpointsConfigProviders = "clusterchecks endpointschecks"

// DefaultGPUMonitoringRuntimeClass default runtime class for GPU pods
DefaultGPUMonitoringRuntimeClass = "nvidia"
gjulianm marked this conversation as resolved.
Show resolved Hide resolved
gjulianm marked this conversation as resolved.
Show resolved Hide resolved
)

// Labels
@@ -201,6 +204,10 @@ const (
FIPSProxyCustomConfigFileName = "datadog-fips-proxy.cfg"
FIPSProxyCustomConfigMapName = "%s-fips-config"
FIPSProxyCustomConfigMountPath = "/etc/datadog-fips-proxy/datadog-fips-proxy.cfg"

NVIDIADevicesMountPath = "/var/run/nvidia-container-devices/all"
NVIDIADevicesVolumeName = "nvidia-devices"
DevNullPath = "/dev/null" // used to mount the NVIDIADevicesHostPath to /dev/null in the container, it's just used as a "signal" to the nvidia runtime to use the nvidia devices
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these (including line 83) only seem to be used in the feature/gpu/feature.go code, can they be placed within the feature/gpu directory? e.g. https://github.com/DataDog/datadog-operator/blob/main/internal/controller/datadogagent/feature/kubernetesstatecore/const.go

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved back to gpu/const.go

)

// Field paths
16 changes: 16 additions & 0 deletions api/datadoghq/v2alpha1/datadogagent_types.go
Original file line number Diff line number Diff line change
@@ -82,6 +82,8 @@ type DatadogFeatures struct {
SBOM *SBOMFeatureConfig `json:"sbom,omitempty"`
// ServiceDiscovery
ServiceDiscovery *ServiceDiscoveryFeatureConfig `json:"serviceDiscovery,omitempty"`
// GPU monitoring
GPUMonitoring *GPUMonitoringFeatureConfig `json:"gpu,omitempty"`
gjulianm marked this conversation as resolved.
Show resolved Hide resolved

// Cluster-level features

@@ -498,6 +500,20 @@ type ServiceDiscoveryFeatureConfig struct {
Enabled *bool `json:"enabled,omitempty"`
}

// GPUMonitoringFeatureConfig contains the GPU monitoring configuration.
type GPUMonitoringFeatureConfig struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// GPUMonitoringFeatureConfig contains the GPU monitoring configuration.
type GPUMonitoringFeatureConfig struct {
// GPUFeatureConfig contains the GPU monitoring configuration.
type GPUFeatureConfig struct {

// Enabled enables GPU monitoring.
// Default: false
// +optional
Enabled *bool `json:"enabled,omitempty"`

// PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.
// If left empty, the runtime class will not be set.
// Default: nvidia
// +optional
PodRuntimeClassName *string `json:"requiredRuntimeClassName"`
}

// DogstatsdFeatureConfig contains the Dogstatsd configuration parameters.
// +k8s:openapi-gen=true
type DogstatsdFeatureConfig struct {
30 changes: 30 additions & 0 deletions api/datadoghq/v2alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion api/datadoghq/v2alpha1/zz_generated.openapi.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 30 additions & 0 deletions config/crd/bases/v1/datadoghq.com_datadogagents.yaml
Original file line number Diff line number Diff line change
@@ -1019,6 +1019,21 @@ spec:
Default: false
type: boolean
type: object
gpu:
description: GPU monitoring
properties:
enabled:
description: |-
Enabled enables GPU monitoring.
Default: false
type: boolean
requiredRuntimeClassName:
description: |-
PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.
If left empty, the runtime class will not be set.
Default: nvidia
type: string
type: object
helmCheck:
description: HelmCheck configuration.
properties:
@@ -7883,6 +7898,21 @@ spec:
Default: false
type: boolean
type: object
gpu:
description: GPU monitoring
properties:
enabled:
description: |-
Enabled enables GPU monitoring.
Default: false
type: boolean
requiredRuntimeClassName:
description: |-
PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.
If left empty, the runtime class will not be set.
Default: nvidia
type: string
type: object
helmCheck:
description: HelmCheck configuration.
properties:
30 changes: 30 additions & 0 deletions config/crd/bases/v1/datadoghq.com_datadogagents_v2alpha1.json
Original file line number Diff line number Diff line change
@@ -1065,6 +1065,21 @@
},
"type": "object"
},
"gpu": {
"additionalProperties": false,
"description": "GPU monitoring",
"properties": {
"enabled": {
"description": "Enabled enables GPU monitoring.\nDefault: false",
"type": "boolean"
},
"requiredRuntimeClassName": {
"description": "PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.\nIf left empty, the runtime class will not be set.\nDefault: nvidia",
"type": "string"
}
},
"type": "object"
},
"helmCheck": {
"additionalProperties": false,
"description": "HelmCheck configuration.",
@@ -7871,6 +7886,21 @@
},
"type": "object"
},
"gpu": {
"additionalProperties": false,
"description": "GPU monitoring",
"properties": {
"enabled": {
"description": "Enabled enables GPU monitoring.\nDefault: false",
"type": "boolean"
},
"requiredRuntimeClassName": {
"description": "PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.\nIf left empty, the runtime class will not be set.\nDefault: nvidia",
"type": "string"
}
},
"type": "object"
},
"helmCheck": {
"additionalProperties": false,
"description": "HelmCheck configuration.",
4 changes: 2 additions & 2 deletions config/manager/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ resources:
- manager.yaml
images:
- name: controller
newName: gcr.io/datadoghq/operator
newTag: 1.11.1
newName: 601427279990.dkr.ecr.us-east-1.amazonaws.com/guillermo.julian/sandbox
newTag: operator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind undoing these changes?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, sorry. Removed!

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
2 changes: 2 additions & 0 deletions docs/configuration.v2alpha1.md
Original file line number Diff line number Diff line change
@@ -111,6 +111,8 @@ spec:
| features.externalMetricsServer.registerAPIService | RegisterAPIService registers the External Metrics endpoint as an APIService Default: true |
| features.externalMetricsServer.useDatadogMetrics | UseDatadogMetrics enables usage of the DatadogMetrics CRD (allowing one to scale on arbitrary Datadog metric queries). Default: true |
| features.externalMetricsServer.wpaController | WPAController enables the informer and controller of the Watermark Pod Autoscaler. NOTE: The Watermark Pod Autoscaler controller needs to be installed. See also: https://github.com/DataDog/watermarkpodautoscaler. Default: false |
| features.gpu.enabled | Enables GPU monitoring. Default: false |
| features.gpu.requiredRuntimeClassName | PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature. If left empty, the runtime class will not be set. Default: nvidia |
gjulianm marked this conversation as resolved.
Show resolved Hide resolved
| features.helmCheck.collectEvents | CollectEvents set to `true` enables event collection in the Helm check (Requires Agent 7.36.0+ and Cluster Agent 1.20.0+) Default: false |
| features.helmCheck.enabled | Enables the Helm check. Default: false |
| features.helmCheck.valuesAsTags | ValuesAsTags collects Helm values from a release and uses them as tags (Requires Agent and Cluster Agent 7.40.0+). Default: {} |
2 changes: 2 additions & 0 deletions examples/datadogagent/datadog-agent-all.yaml
Original file line number Diff line number Diff line change
@@ -47,6 +47,8 @@ spec:
enabled: true
serviceDiscovery:
enabled: true
gpu:
enabled: true
eventCollection:
collectKubernetesEvents: true
orchestratorExplorer:
1 change: 1 addition & 0 deletions internal/controller/datadogagent/controller.go
Original file line number Diff line number Diff line change
@@ -34,6 +34,7 @@ import (
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/enabledefault"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/eventcollection"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/externalmetrics"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/gpu"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/helmcheck"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/kubernetesstatecore"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/livecontainer"
Original file line number Diff line number Diff line change
@@ -37,6 +37,8 @@ const (

defaultEBPFCheckEnabled bool = false

defaultGPUMonitoringEnabled bool = false

defaultServiceDiscoveryEnabled bool = false

defaultAPMEnabled bool = true
@@ -265,6 +267,12 @@ func defaultFeaturesConfig(ddaSpec *v2alpha1.DatadogAgentSpec) {
}
apiutils.DefaultBooleanIfUnset(&ddaSpec.Features.ServiceDiscovery.Enabled, defaultServiceDiscoveryEnabled)

// GPU monitoring feature
if ddaSpec.Features.GPUMonitoring == nil {
ddaSpec.Features.GPUMonitoring = &v2alpha1.GPUMonitoringFeatureConfig{}
}
apiutils.DefaultBooleanIfUnset(&ddaSpec.Features.GPUMonitoring.Enabled, defaultGPUMonitoringEnabled)

// APM Feature
// APM is enabled by default
if ddaSpec.Features.APM == nil {
Loading