Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add observability for lambda-to-lambda invocations #3623

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add observability for lambda-to-lambda invocations #3623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion lib/instrumentation/http-shared.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,10 @@ exports.traceOutgoingRequest = function (agent, moduleName, method) {
return function wrapHttpRequest(orig) {
return function wrappedHttpRequest(input, options, cb) {
const parentRunContext = ins.currRunContext();
var span = ins.createSpan(null, 'external', 'http', { exitSpan: true });
var span =
input?.headers?.['x-amz-invocation-type'] === 'RequestResponse'
? null
: ins.createSpan(null, 'external', 'http', { exitSpan: true });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this change to avoid the extra span necessary for the Service Map connection to work? Or was this just to avoid having an addition span in the trace waterfall?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both I guess. If this span is created when a lambda is invoked, I can see two entries on Service Map: one empty (created by this span) and other with the full tracing (created from the code send in lib/lambda.js.

This code just removes the empty one.

var id = span && span.transaction.id;
agent.logger.debug('intercepted call to %s.%s %o', moduleName, method, {
id,
Expand Down
8 changes: 8 additions & 0 deletions lib/lambda.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -705,6 +705,14 @@ function elasticApmAwsLambda(agent) {
// Look for trace-context info in headers or messageAttributes.
let traceparent;
let tracestate;
if (
triggerType === TRIGGER_GENERIC &&
event.traceparent &&
event.tracestate
) {
traceparent = event.traceparent;
tracestate = event.tracestate;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fdaciuk Thanks for the PR.

Can you show some simple Lambda function code that uses this? That would help me understand a bit better.
Does your code that is doing the Lambda Invoke API call (https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html) manually pass in traceparent and tracestate in the payload?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @trentm. Exactly! I don't know if there is better way to do this, but that update worked for my use case.

I'm invoking the lambda like this:

  const command = new InvokeCommand({
    FunctionName: "my-lambda",
    InvocationType: "RequestResponse",
    Payload: new TextEncoder().encode(JSON.stringify({ traceparent, tracestate })),
  })

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't a current way to do this ... except possibly Amazon's X-Ray support might enable making the connection. We aren't currently doing anything with X-Ray in our Lambda support.

Providing some way for this manual distributed tracing in Lambda is interesting. I like the idea.

I wonder about instead using ClientContext in the InvokeCommand call. Theoretically, if the APM agent automatically instrumented @aws-sdk/client-lambda -- which I'm guessing you are using here -- then we could (perhaps optionally) support automatically adding the trace-context info (traceparent et al) to the ClientContext or Payload.

I'm not sure if ClientContext and Payload, IIUC, are unstructured. Or does the provided ClientContext need to be a JSON object that can be added to the context object passed to the Lambda function? If these are unstructured, in general, then it would be a hard sell to have a default feature added to the APM agent that tried to impose some structure on it.

Copy link
Author

@fdaciuk fdaciuk Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if ClientContext and Payload, IIUC, are unstructured. Or does the provided ClientContext need to be a JSON object that can be added to the context object passed to the Lambda function?

That's a good question. I'm not sure, but love the idea that this might be automatic. We're creating a library on our company to abstract the use of InvokeCommand and inject traceparent and tracestate into Payload, so if we had an option to do this from elastic-apm-node, even manually, would be great =)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a heads up that I won't be able to get to this soon. :/

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem. For now, we're using a forked version of this repo with these updates. I hope it's not a problem until you decide whether or not it's a valuable feature in this lib =)

if (
(triggerType === TRIGGER_API_GATEWAY || triggerType === TRIGGER_ELB) &&
event.headers
Expand Down