Integrate with AWS Distro for OpenTelemetry

[eduardomourar]

• Why AWS Distro for OpenTelemetry?

OpenTelemetry provides open source APIs, libraries, and agents to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) of distributed applications.

Because there is a few overlap between the lambda power tools and OpenTelemetry, it would be nice to see how the layer provided can be reused or merged:
https://github.com/open-telemetry/opentelemetry-lambda/tree/main/python

Another important aspect, although it has not been fully finalized yet, is to follow the Log Data Model defined in the OpenTelemetry spec so that the context can be later related to both metric and trace.

[heitorlessa]

Hey @eduardomourar thanks a lot for the suggestion. This is something we'd like to consider for v2. I signalled this to the OpenTelemetry team to have a discussion on how to integrate in the future - I'll leave as Ideas until we have an initial chat with the OTel team on this in H2.

Following Log Data Model until it's finalized, stable, and in use for some time would be a good measure too besides customers +1 this feature request. In the medium term (v2, Q4), I can see us providing a better integration but not using it as the default yet.

[go4real]

It would be nice if we could change the logging backend between the open telemetry agent and the x-ray agent by setting some configuration values without any code changes.

There are currently two options for tracking data, each with pros and cons. And our main concern that is hesitating to do more custom instrumentation is code changes between logging backend changes. If there is a wrapper sdk, we will use it without hesitation.

[gbisaga]

I agree, this would be great. We want to be committed to Otel in the long run, but the native integration with lambdas is a winner. Having a switchable backend would let large organizations commit to this library knowing it could be used inline with longer term plans.

[heitorlessa]

Sharing updates as we're discussing the Observability Provider feature in Python: aws-powertools/powertools-lambda-python#1433

As of now, we no longer intend to move entirely to OTel (ADOT included) for two main reasons:

• Cold start increase is too significant because it requires a Daemon to be spun up (Java, .NET being the most impacted)
• Logging is still in Draft status

Instead, we will focus on making it easier to bring any observability SDK/agent through observability providers (e.g., we have an undocumented TracingProvider today).

[Dilski]

@heitorlessa
The x-ray python SDK docs say:

AWS X-Ray recommends using AWS Distro for OpenTelemetry (ADOT) to instrument your application instead of this X-Ray SDK due to its wider range of features and instrumentations.

It would be good if either X-Ray's recommendation/docs were updated to show/explain these reasons, or if power tools docs were updated to explain the decision to use x-ray SDK instead of otel.

I understand the reasoning you've got for not incorporating otel/adot into power tools just now, but it may be confusing for folks just looking for the best practice for tracing in python lambdas

[willfarrell]

@Dilski Take a read of a similar thread with a more recent update on this: aws-powertools/powertools-lambda-typescript#665 (comment)

... we won't be making any plans for moving to OTel at least in 2023.
We'll communicate with ample time if anything changes.

[heitorlessa]

hey @Dilski thank you for flagging this inconsistency. I've pinged the X-Ray team as there is no confirmed plan to deprecate X-Ray SDK.

From our side, do you have ideas on where we can make this clearer? any specific place in the docs? FAQ?

Thanks a lot!! (@willfarrell too!)

[willfarrell]

Related to Tracing, there is a similar confusion with Metrics

• aws-embedded-metrics
• AWS Powertools TypeScript
• Otel

I was planning to deprecate using aws-embedded-metrics within middy (https://github.com/middyjs/middy/tree/main/packages/cloudwatch-metrics). Then a major release happened.

The dream from my perspective is that powertools is an easier to use wrapper around otel.

[heitorlessa]

The dream from my perspective is that powertools is an easier to use wrapper around otel.

OTEL brings some difficulties (latency, logs not being prod ready), but we're gonna set a good foundation with the Observability Provider discussion (I'll write a RFC after re:Invent).

From there, if you want to use OTEL we should have an OTEL provider built-in, or if you want to use Provider X Y Z, you can bring their dependencies/config and plug that into our Tracer/Metrics/Logger pieces, keeping the same UX -- a thin-wrapper with our opinionated approach to handle 80% of use cases (20% being overrides, etc.).

I've just had a discussion with one popular provider, and I think now this will be a great stepping stone for when OTEL is ready to be an all-in replacement - we want to support people going the OTEL route, but we also acknowledge the reality that the vast majority are using native tooling (AWS or from provider X) as there's many peculiar parts.

We'll get there ;) I'm more hopeful than I was a year ago when looking at OTEL alone.

[Dilski]

@heitorlessa In my opinion, one of the blue "Note" boxes on the tracer page, with a quick explanation that the SDK was chosen over otel because of the performance in lambda + incompleteness of otel - although it would be better if the X-Ray team were to update their warning/docs so we don't get the impression that we should just use adot/otel.

As for the thin-wrapper design approach - I'm a huge fan. Currently I've got a lot of python lambda functions instrumented using the X-Ray SDK that i'd love to move to opentelemetry, i just don't think opentelemetry is there quite yet. Tweaking those to use powertools' tracer util will make my otel switch easier in the future. So a big +1 for that.

[heitorlessa]

Awesome, consider it done by EOW. If you don't mind, I'd appreciate a docs issue to help track this work: https://github.com/awslabs/aws-lambda-powertools-python/issues/new?assignees=&labels=documentation%2Ctriage&template=documentation_improvements.yml&title=Docs%3A+TITLE

Once we do in Python, we can easily transfer to other languages as the wording will be the same.

although it would be better if the X-Ray team were to update their warning/docs so we don't get the impression that we should just use adot/otel.

We'll pass that feedback along!

[Dilski]

@heitorlessa aws-powertools/powertools-lambda-python#1675 - hope this helps.

[niko-achilles]

Happy to read this issue today. Some comments from my side after 21 days.

From there, if you want to use OTEL we should have an OTEL provider built-in, or if you want to use Provider X Y Z, you can bring their dependencies/config and plug that into our Tracer/Metrics/Logger pieces, keeping the same UX -- a thin-wrapper with our opinionated approach to handle 80% of use cases (20% being overrides, etc.).

Awesome @heitorlessa .
At the company i work for , the Teams in the modernization phase Serverless first strategy are now experimenting with Powertools (Python, Typescript, .NET) and in parallel with OTEL.

The Developer experience of powertools (best practices built-in) is great.
ADOT collector deployment strategy for serverless as the state of reality is today is not so developer friendly and has the domino effect in quality ;) Deploying a modified runtime as internal extension and and external extension for moving data from point A to point B gets complicated very fast.

The telemetry API announced recently is a better deployment approach in comparison to collector, however it is about moving telemetry data from Point A to Point B. ;)

(I'll write a RFC after re:Invent).

I am happy for the RFC moment after reinvent to read and also share experiences if needed.

We'll get there ;) I'm more hopeful than I was a year ago when looking at OTEL alone.

I like this statement