RFC - Pipeline Component Telemetry #11406
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #11406 +/- ##
=======================================
Coverage 91.61% 91.61%
=======================================
Files 443 443
Lines 23770 23770
=======================================
Hits 21776 21776
Misses 1620 1620
Partials 374 374 ☔ View full report in Codecov by Sentry. |
djaglowski
force-pushed
the
component-telemetry-rfc
branch
from
99e3086 to
5df52e1
Compare
djaglowski
added
Skip Changelog
codeboten
reviewed
Oct 10, 2024
bogdandrutu
reviewed
Oct 10, 2024
bogdandrutu
reviewed
Oct 16, 2024
bogdandrutu
reviewed
Oct 16, 2024
djaglowski
changed the title
|
Based on some offline feedback, I've broadened the scope of the RFC, while simultaneously clarifying that it is intended to evolve as we identify additional standards. |
jaronoff97
reviewed
Oct 23, 2024
dmathieu
reviewed
Oct 24, 2024
jaronoff97
reviewed
Oct 24, 2024
mx-psi
approved these changes
Oct 29, 2024
wildum
reviewed
Oct 30, 2024
wildum
reviewed
Oct 30, 2024
wildum
approved these changes
Oct 30, 2024
jaronoff97
approved these changes
Oct 30, 2024
evan-bradley
approved these changes
Nov 4, 2024
jpkrohling
reviewed
Nov 4, 2024
|
Some of my comments might have been discussed before, in which case, feel free to ignore me and just mark the items as resolved. |
dmitryax
reviewed
Nov 6, 2024
dmitryax
reviewed
Nov 17, 2024
djaglowski
force-pushed
the
component-telemetry-rfc
branch
from
7d7a75b to
a7a15e5
Compare
djaglowski
added
the
rfc:final-comment-period
codeboten
added a commit
to codeboten/opentelemetry-collector
that referenced
this pull request
Nov 21, 2024
This sets the level of all metrics that where not previously stabilized as alpha. Since many of these metrics will change as a result of open-telemetry#11406, it made sense to me to set their stability as alpha. Signed-off-by: Alex Boten <[email protected]>
|
This has enough approvals and has entered the 'final comment period'. I will merge this on 2024-11-27 if nobody blocks before. cc @open-telemetry/collector-approvers |
jpkrohling
approved these changes
Nov 22, 2024
codeboten
added a commit
that referenced
this pull request
Nov 22, 2024
This sets the level of all metrics that where not previously stabilized as alpha. Since many of these metrics will change as a result of #11406, it made sense to me to set their stability as alpha. --------- Signed-off-by: Alex Boten <[email protected]>
|
I believe all feedback has been addressed. #11743 represents two followup items raised by @jpkrohling, but I believe the RFC is clear that some changes are anticipated. |
jpkrohling
approved these changes
Nov 26, 2024
|
Per #11406 (comment) I am merging this 🎉 |
3 tasks
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 16, 2024
## Description This PR defines observability requirements for components at the "Stable" stability levels. The goal is to ensure that Collector pipelines are properly observable, to help in debugging configuration issues. #### Approach - The requirements are deliberately not too specific, in order to be adaptable to each specific component, and so as to not over-burden component authors. - After discussing it with @mx-psi, this list of requirements explicitly includes things that may end up being emitted automatically as part of the Pipeline Instrumentation RFC (#11406), with only a note at the beginning explaining that not everything may need to be implemented manually. Feel free to share if you don't think this is the right approach for these requirements. #### Link to tracking issue Resolves #11581 ## Important note regarding the Pipeline Instrumentation RFC I included this paragraph in the part about error count metrics: > The goal is to be able to easily pinpoint the source of data loss in the Collector pipeline, so this should either: > - only include errors internal to the component, or; > - allow distinguishing said errors from ones originating in an external service, or propagated from downstream Collector components. The [Pipeline Instrumentation RFC](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/component-universal-telemetry.md) (hereafter abbreviated "PI"), once implemented, should allow monitoring component errors via the `outcome` attribute, which is either `success` or `failure`, depending on whether the `Consumer` API call returned an error. Note that this does not work for receivers, or allow differentiating between different types of errors; for that reason, I believe additional component-specific error metrics will often still be required, but it would be nice to cover as many cases as possible automatically. However, at the moment, errors are (usually) propagated upstream through the chain of `Consume` calls, so in case of error the `failure` state will end up applied to all components upstream of the actual source of the error. This means the PI metrics do not fit the first bullet point. Moreover, I would argue that even post-processing the PI metrics does not reliably allow distinguishing the ultimate source of errors (the second bullet point). One simple idea is to compute `consumed.items{outcome:failure} - produced.items{outcome:failure}` to get the number of errors originating in a component. But this only works if output items map one-to-one to input items: if a processor or connector outputs fewer items than it consumes (because it aggregates them, or translates to a different signal type), this formula will return false positives. If these false positives are mixed with real errors from the component and/or from downstream, the situation becomes impossible to analyze by just looking at the metrics. For these reasons, I believe we should do one of four things: 1. Change the way we use the `Consumer` API to no longer propagate errors, making the PI metric outcomes more precise. We could catch errors in whatever wrapper we already use to emit the PI metrics, log them for posterity, and simply not propagate them. Note that some components already more or less do this, such as the `batchprocessor`, but this option may in principle break components which rely on downstream errors (for retry purposes for example). 3. Keep propagating errors, but modify or extend the RFC to require distinguishing between internal and propagated errors (maybe add a third `outcome` value, or add another attribute). This could be implemented by somehow propagating additional state from one `Consume` call to another, allowing us to establish the first appearance of a given error value in the pipeline. 5. Loosen this requirement so that the PI metrics suffice in their current state. 6. Leave everything as-is and make component authors implement their own somewhat redundant error count metrics. --------- Co-authored-by: Pablo Baeyens <[email protected]> Co-authored-by: Pablo Baeyens <[email protected]>
HongChenTW
pushed a commit
to HongChenTW/opentelemetry-collector
that referenced
this pull request
Dec 19, 2024
…ry#11729) This sets the level of all metrics that where not previously stabilized as alpha. Since many of these metrics will change as a result of open-telemetry#11406, it made sense to me to set their stability as alpha. --------- Signed-off-by: Alex Boten <[email protected]>
HongChenTW
pushed a commit
to HongChenTW/opentelemetry-collector
that referenced
this pull request
Dec 19, 2024
This PR adds a RFC for normalized telemetry across all pipeline components. See open-telemetry#11343 edit by @mx-psi: - Announced on #otel-collector-dev on 2024-10-23: https://cloud-native.slack.com/archives/C07CCCMRXBK/p1729705290741179 - Announced on the Collector SIG meeting from 2024-10-30 --------- Co-authored-by: Alex Boten <[email protected]> Co-authored-by: Damien Mathieu <[email protected]> Co-authored-by: William Dumont <[email protected]> Co-authored-by: Evan Bradley <[email protected]>
HongChenTW
pushed a commit
to HongChenTW/opentelemetry-collector
that referenced
this pull request
Dec 19, 2024
…ry#11772) ## Description This PR defines observability requirements for components at the "Stable" stability levels. The goal is to ensure that Collector pipelines are properly observable, to help in debugging configuration issues. #### Approach - The requirements are deliberately not too specific, in order to be adaptable to each specific component, and so as to not over-burden component authors. - After discussing it with @mx-psi, this list of requirements explicitly includes things that may end up being emitted automatically as part of the Pipeline Instrumentation RFC (open-telemetry#11406), with only a note at the beginning explaining that not everything may need to be implemented manually. Feel free to share if you don't think this is the right approach for these requirements. #### Link to tracking issue Resolves open-telemetry#11581 ## Important note regarding the Pipeline Instrumentation RFC I included this paragraph in the part about error count metrics: > The goal is to be able to easily pinpoint the source of data loss in the Collector pipeline, so this should either: > - only include errors internal to the component, or; > - allow distinguishing said errors from ones originating in an external service, or propagated from downstream Collector components. The [Pipeline Instrumentation RFC](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/component-universal-telemetry.md) (hereafter abbreviated "PI"), once implemented, should allow monitoring component errors via the `outcome` attribute, which is either `success` or `failure`, depending on whether the `Consumer` API call returned an error. Note that this does not work for receivers, or allow differentiating between different types of errors; for that reason, I believe additional component-specific error metrics will often still be required, but it would be nice to cover as many cases as possible automatically. However, at the moment, errors are (usually) propagated upstream through the chain of `Consume` calls, so in case of error the `failure` state will end up applied to all components upstream of the actual source of the error. This means the PI metrics do not fit the first bullet point. Moreover, I would argue that even post-processing the PI metrics does not reliably allow distinguishing the ultimate source of errors (the second bullet point). One simple idea is to compute `consumed.items{outcome:failure} - produced.items{outcome:failure}` to get the number of errors originating in a component. But this only works if output items map one-to-one to input items: if a processor or connector outputs fewer items than it consumes (because it aggregates them, or translates to a different signal type), this formula will return false positives. If these false positives are mixed with real errors from the component and/or from downstream, the situation becomes impossible to analyze by just looking at the metrics. For these reasons, I believe we should do one of four things: 1. Change the way we use the `Consumer` API to no longer propagate errors, making the PI metric outcomes more precise. We could catch errors in whatever wrapper we already use to emit the PI metrics, log them for posterity, and simply not propagate them. Note that some components already more or less do this, such as the `batchprocessor`, but this option may in principle break components which rely on downstream errors (for retry purposes for example). 3. Keep propagating errors, but modify or extend the RFC to require distinguishing between internal and propagated errors (maybe add a third `outcome` value, or add another attribute). This could be implemented by somehow propagating additional state from one `Consume` call to another, allowing us to establish the first appearance of a given error value in the pipeline. 5. Loosen this requirement so that the PI metrics suffice in their current state. 6. Leave everything as-is and make component authors implement their own somewhat redundant error count metrics. --------- Co-authored-by: Pablo Baeyens <[email protected]> Co-authored-by: Pablo Baeyens <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a RFC for normalized telemetry across all pipeline components. See #11343
edit by @mx-psi: