[EBPF] gpu: auto-enable agent check if system-probe gpu_monitoring module is enabled #32521
Conversation
github-actions
bot
added
team/agent-shared-components
short review
|
[Fast Unit Tests Report] On pipeline 52124222 (CI Visibility). The following jobs did not run any unit tests: Jobs:
If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help |
Uncompressed package size comparisonComparison with ancestor Diff per package
Decision✅ Passed |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 1c800cc Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_logs | % cpu utilization | +2.31 | [-0.95, +5.57] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | +0.44 | [-0.03, +0.91] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | +0.06 | [-0.85, +0.96] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.03 | [-0.74, +0.81] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.00 | [-0.77, +0.78] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.12, +0.12] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.01, +0.01] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.01 | [-0.88, +0.86] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | -0.02 | [-0.85, +0.81] | 1 | Logs |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | -0.05 | [-0.70, +0.60] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | -0.14 | [-0.18, -0.11] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.24 | [-0.32, -0.16] | 1 | Logs bounds checks dashboard |
| ➖ | file_tree | memory utilization | -0.24 | [-0.37, -0.12] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.50 | [-1.28, +0.28] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.53 | [-1.21, +0.15] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.67 | [-0.73, -0.61] | 1 | Logs |
Bounds Checks: ❌ Failed
| perf | experiment | bounds_check_name | replicates_passed | links |
|---|---|---|---|---|
| ❌ | file_to_blackhole_500ms_latency | lost_bytes | 9/10 | |
| ✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
| ✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | lost_bytes | 10/10 | |
| ✅ | quality_gate_logs | memory_usage | 10/10 |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
gjulianm
changed the title
gjulianm
force-pushed
the
guillermo.julian/auto-enable-gpu-agent-check
branch
from
1db0904 to
ec443a5
Compare
gjulianm
added
changelog/no-changelog
qa/done
gjulianm
force-pushed
the
guillermo.julian/auto-enable-gpu-agent-check
branch
from
ec443a5 to
4ffac72
Compare
|
|
||
| # Match the key gpu_monitoring.enabled: true using Python's YAML parser, which is included in the base image | ||
| # and is more robust than using regexes. | ||
| gpu_monitoring_enabled=$(python -c "import yaml, sys; data=yaml.safe_load(sys.stdin); print(bool(data.get('gpu_monitoring', {}).get('enabled')))" < $sysprobe_cfg) |
There was a problem hiding this comment.
nit: Not sure if the python alias is here to stay or not (anyway we need to get rid of these cont-init.d files at some point.
Also it'd be nice to have the same way for al then.
There was a problem hiding this comment.
Changed to explicitly use python3. We can make a refactor of the previous calls in another PR, although if the plan is to remove this file entirely in the future I'm not sure if it's worth the risk to change it.
github-actions
bot
added
medium review
| if grep -Eq '^ *enable_tcp_queue_length *: *true' /etc/datadog-agent/system-probe.yaml || [[ "$DD_SYSTEM_PROBE_CONFIG_ENABLE_TCP_QUEUE_LENGTH" == "true" ]]; then | ||
| sysprobe_cfg="/etc/datadog-agent/system-probe.yaml" | ||
|
|
||
| if grep -Eq '^ *enable_tcp_queue_length *: *true' $sysprobe_cfg || [[ "$DD_SYSTEM_PROBE_CONFIG_ENABLE_TCP_QUEUE_LENGTH" == "true" ]]; then |
There was a problem hiding this comment.
This is a very brittle way to check for a config...
Eg. all the following values are considered as true by Viper: "1", "t", "T", "true", "TRUE", "True"
There was a problem hiding this comment.
I think the assumption here is that the config is set by the operator/helm chart in some reasonably sane way. In any case, if this fails is not critical, it just means that customers have to enable the check manually.
There was a problem hiding this comment.
I think it also runs when using the agent container "manually", but I definitely agree that failing to detect some config being set is not a big deal here 👍
| # Match the key gpu_monitoring.enabled: true using Python's YAML parser, which is included in the base image | ||
| # and is more robust than using regexes. | ||
| gpu_monitoring_enabled=$(python3 -c "import yaml, sys; data=yaml.safe_load(sys.stdin); print(bool(data.get('gpu_monitoring', {}).get('enabled')))" < $sysprobe_cfg) |
There was a problem hiding this comment.
Why do that only for gpu_monitoring.enabled and not the other configs in this script ?
There was a problem hiding this comment.
Also as mentioned in my other comment Viper is very liberal in what it accepts as a boolean, so this won't work for every value
There was a problem hiding this comment.
I preferred to avoid changes to this just in case, it's an implicit config change that might be hard to debug/notice if for some reason the behaviour is changed.
|
/merge |
Devflow running:
|
dd-mergequeue
bot
deleted the
guillermo.julian/auto-enable-gpu-agent-check
branch
What does this PR do?
This PR adds automatic detection of the GPU monitoring feature in
60-sysprobe-check.shconfig script for containerized environments, and enables the corresponding necessary agent-side check.Motivation
Reduce the possibility of misconfigurations: if system-probe has the GPU monitoring module enabled but the agent gpu check is not enabled, no data will be reported.
This PR also simplifies deployments in k8s environments. As the GPU monitoring feature is deployed in mixed clusters (those where some nodes have GPUs and some don't), we need to override the features based on the node GPU availability. Enabling/disabling the system-probe module is easy enough with environment variables, but enabling/disabling checks is not as simple. This PR removes the need to manually configure the agent side check.
Describe how you validated your changes
Validated by manually running the script with different input values.
Possible Drawbacks / Trade-offs
Additional Notes
This PR parses the YAML file with Python's YAML module, which is present in the container image so no extra dependencies are required. This is a more robust alternative than using regex: while for other settings (e.g.
enable_oom_kill) we're looking for a single key, here we're looking for a nested key inside another, so the regex would be more complex and more prone to errors.