[exporter/otlp] Report runtime status

[mwear]

Description

This PR adds runtime status reporting for the otlp and otlphttp exporters. It's an updated version of #8788 which was one of several approaches experimented with to add this functionality. This work was paused to allow the consumererror work to evolve, as it appeared as though there might be a way to uniformly apply the same logic to all exporters (via the exporterhelper), but that work has taken a different direction, and it looks like a uniform approach will not be possible.

This PR implements runtime status reporting as discussed in #9957. The choices for which statuses represent permanent errors are up for debate, but the key point is that a permanent error is an error discovered at runtime that will require user intervention to fix.

This implementation makes use of the finite state machine that underlies the status reporting system. The finite state machine will ensure that:

• Only changes in status are reported. Repeat reports of the same status will no-op.
• If a component transitions into a PermanentError all further status reports will no-op.

This means the exporter does not have to reason about current or previous statuses. It can report status based on its current view of the world and the status reporting system will handle the rest. Flapping between recoverable and ok is meant to be handled by watchers consuming status events. The healthcheckv2extension handles this by using a time based approach (e.g. recovery interval). Other watchers may choose to handle this situation differently.

For more information on status reporting, the state machine, etc see: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-status.md

While all components report status during start and shutdown (via automation), this is the first component to report runtime status. This will allow the healthcheckv2extension to replace the currently non-functioning check_collector_pipeline capability of the original healthcheckextension and should serve as an example for other components that wish to report runtime status moving forward.

Link to tracking issue

Implements #9957 for the otlp and otlphttp exporters

Testing

units/manual

Documentation

code comments

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.64%. Comparing base (f74890a) to head (2043eba).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11366      +/-   ##
==========================================
+ Coverage   91.61%   91.64%   +0.03%     
==========================================
  Files         443      443              
  Lines       23770    23828      +58     
==========================================
+ Hits        21776    21837      +61     
+ Misses       1620     1618       -2     
+ Partials      374      373       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[bogdandrutu]

Before implementation, can we document somewhere (ideal in RFC) or in https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-status.md when an exporter is consider permanent error? I feel like we have different understandings of this and I don't want you to make lots of code changes until we agree on the basic definition.

[mwear]

Before implementation, can we document somewhere (ideal in RFC) or in https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-status.md when an exporter is consider permanent error?

I have these documented here: #9957. Is it ok to discuss on that issue, or would you prefer it to be an RFC?

[mwear]

If you're suggesting that the errors identified as Permanent in this PR (and #9957) are in many cases recoverable, we can identify all errors as recoverable and skip permanent for now.

[bogdandrutu]

I am not the biggest expert, but I feel at least some are recoverable, and some are contextual (example auth when passing auth tokens from the client).

If you're suggesting that the errors identified as Permanent in this PR (and #9957) are in many cases recoverable, we can identify all errors as recoverable and skip permanent for now.

I think would be safer correct?

[mwear]

I think going with recoverable for all errors, at least for now, makes sense. We can discuss what errors should be permanent on #9957, but I suspect we'll be able to come up with counter arguments for most of the cases.

[mwear]

I updated this PR to handle all errors as recoverable. If we are happy with this approach, we could consider making this an option for all exporters via exporter helper similar to: #8684 likely as a followup.

[bogdandrutu]

Are we not concerned for every request to report the status? Should we do it periodically, or only when changes?

[mwear]

The finite state machine handles this for us. It will report status only if there has been a change, otherwise it will no-op.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[mwear]

Would anyone be interested in reviewing this, or at least removing the stale label?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[atoulme]

this might create a lot of events reporting OK, is that really what you want? Is it there to recover from the error status?