Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[k8sclusterreceiver] refactoring pod status phase #24425

Closed
povilasv opened this issue Jul 21, 2023 · 18 comments
Closed

[k8sclusterreceiver] refactoring pod status phase #24425

povilasv opened this issue Jul 21, 2023 · 18 comments

Comments

@povilasv
Copy link
Contributor

Component(s)

receiver/k8scluster

Describe the issue you're reporting

We currently encode pod status phase to number:

  k8s.pod.phase:
    enabled: true
    description: Current phase of the pod (1 - Pending, 2 - Running, 3 - Succeeded, 4 - Failed, 5 - Unknown)
    unit: 1
    gauge:
      value_type: int

I think it's preferable to split into many metrics:

  k8s.pod.phase_pending:
    enabled: true
    description: Whether this pod phase is Pending (1), or not (0).
    unit: 1
    gauge:
      value_type: int
  k8s.pod.phase_running:
    enabled: true
    description: Whether this pod phase is Running (1), or not (0).
    unit: 1
    gauge:
      value_type: int

IMO it's easier to parse from the name what is happening and we already have this pattern for Nodes conditions.

Thoughts?

@povilasv povilasv added the needs triage New item requiring triage label Jul 21, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@dmitryax
Copy link
Member

I see your point, but I have a couple of counter-arguments:

  1. This will introduce 5x bigger payload to deliver the same information.
  2. When you look at the chart, it's pretty convenient to see what states the pod has been in over time using just one time series. Some UIs can let you assign the states to the values on the chart, which is very convenient. Using several metrics for that is not that handy.

If a user wants this particular representation, I believe it can be translated with the metrics transform processor. Also, most backends likely have a query language capable of representing this metric as 5 "boolean" time series.

I wouldn't like to make this change until we have strict guidelines from the OTel Spec on how to represent this kind of data

@dmitryax dmitryax added discussion needed Community discussion needed receiver/k8scluster and removed needs triage New item requiring triage labels Jul 21, 2023
@povilasv
Copy link
Contributor Author

Thanks for reply. I agree with you, the k8s.pod.phase is way more compact and in theory transformProcessor should work. I will play with it to see if it does in practice.

@sirianni
Copy link
Contributor

sirianni commented Aug 4, 2023

When you look at the chart, it's pretty convenient to see what states the pod has been in over time using just one time series. Some UIs can let you assign the states to the values on the chart, which is very convenient.

Can you provide a screenshot/example?

Also, most backends likely have a query language capable of representing this metric as 5 "boolean" time series.

Example? I can't get this to work in either of the monitoring vendors my company uses.

Since the actual numeric values are meaningless, I can't see how to make this work in a when aggregating across dimensions. Many monitoring tools also don't let you filter on metric values, just tags (i.e. you can't say WHERE value = 2). For example, if I'd like to see the number of pods that are in Running phase in a DaemonSet, how do I do that?

I think an OpenMetrics StateSet would be the more idiomatic way to model this information (and is also supported by OTel).

This is also how kube-state-metrics does it with kube_pod_status_phase. Seems to work quite well in PromQLcount(kube_pod_status_phase{phase="Running"})

@sirianni
Copy link
Contributor

sirianni commented Aug 7, 2023

I believe it can be translated with the metrics transform processor

@dmitryax can you give a rough example of how you would use metricstransform processor to do this transformation? From what I see there's no way to "match" based on values and aggregating the "numeric enums" don't make sense either.

@bertysentry
Copy link
Contributor

I agree with @sirianni.

OOT, k8s.pod.phase metric should keep the same name (don't split it in k8s.pod.phase_pending, k8s.pod.phase_running, etc.), but it should have a state attribute, to be consistent with other similar enumerations in other metrics (although I don't think state is specified anywhere in Otel's semantic conventions).

So you'll push several datapoints to represent the current "running" state of the pod:

  • k8s.pod.phase{state="running"} 1
  • k8s.pod.phase{state="pending"} 0
  • k8s.pod.phase{state="succeeded"} 0
  • k8s.pod.phase{state="failed"} 0
  • k8s.pod.phase{state="unknown"} 0

Not ultra optimal (cardinality), but it does the job, and it's easily query-able.

Also, it's a good incentive for OpenTelemetry to specify the StateSet metric type! 😉

@sirianni
Copy link
Contributor

sirianni commented Nov 6, 2023

Take a look at the hw.status metric in OTel Semantic Conventions which uses the value-as-attribute approach I'm suggesting. In particular, see this note

hw.status is currently specified as an UpDownCounter but would ideally be represented using a StateSet as defined in OpenMetrics. This semantic convention will be updated once StateSet is specified in OpenTelemetry. This planned change is not expected to have any consequence on the way users query their timeseries backend to retrieve the values of hw.status over time.

image

@TylerHelmuth
Copy link
Member

We discussed this issue in the SIG meeting today. We decided we want the Spec to make a decision on StateSet before we'd move forward with any changes. I will try to find an open issue in the Spec repo and create one to start the conversation if I can't find one.

@TylerHelmuth
Copy link
Member

There is also an open question on where this metric should live. @sirianni can you provide more details on the scaling issue you've experienced?

@sirianni
Copy link
Contributor

There is also an open question on where this metric should live. @sirianni can you provide more details on the scaling issue you've experienced?

Sure. We run several large Kubernetes clusters (over 1000 nodes, over 12,000 pods) where using a single cluster-wide collector runs into scalability issues (scrape timeouts, etc.). Here is the Datadog Agent PR I referenced in today's SIG call where Airbnb mentions similar scalability issues using a cluster-wide collector (i.e. kube-state-metrics) for this data.

Collecting this information from the kubelet via a DaemonSet scales incrementally as the cluster grows.

@dashpole
Copy link
Contributor

Link to slack discussion mentioned in the meeting: https://cloud-native.slack.com/archives/C01NP3BV26R/p1623952435074700

@TylerHelmuth
Copy link
Member

Reading through this issue again is there actually anything we need from the spec? This link (https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#stateset) feels like it is pretty clear about how we handle statesets in otel metrics.

@dashpole
Copy link
Contributor

dashpole commented Nov 15, 2023

That is for how to convert from openmetrics stateset metrics to OTLP. The idea discussed in the slack thread was to make the stateset pattern (or something close to it) a recommended pattern for OTel instrumentation. I would like to see, at minimum, the specification recommend using labels, rather than values to represent enums (i.e. foo{state=bar} 1, rather than foo_state_bar 1 or foo 4, where value 4 implies state==bar)

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Mar 18, 2024
@jaronoff97
Copy link
Contributor

Ran into another example of some confusion/frustration recently

@github-actions github-actions bot removed the Stale label Apr 27, 2024
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Jun 27, 2024
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

No branches or pull requests

7 participants