Logging and diagnostics experience improvements

[fabiocav]

Creating this issue to track the work to improve logging and diagnostics in the worker. The work will address issues like:

• Dependency tracking/distributed tracing
• Filtering/category mapping

We'll continue to update this with details and link to additional issues as we make progress here.

[rhythmnewt]

Any timeline available for this?

[TheSpivack]

Is this still an open issue even with the release of Azure Functions 4.0?

[hossmous]

Any idea if this is available now (function 4.0)?

[oatsoda]

Just to confirm - at the moment, Dependencies in AppInsights are not working in Isolated Process functions?

Because that's what I am seeing. The only dependencies that get logged are when I have an Output Binding. But a dependency like an Http call inside the function logic is not being logged in AppInsights.

Is it actually not tracking them, or is it a similar to this Log Category issue?

The documentation could do with updating as I've been round in circles trying to figure out if I have misconfigured the host log levels.

[hagagps]

We are currently working around this by adding the insights worker service to get dependencies and implementing custom middleware (IFunctionsWorkerMiddleware) to get distributed tracing.

serviceCollection
    .AddApplicationInsightsTelemetryWorkerService()
    .AddSingleton(new TelemetryClient(TelemetryConfiguration.CreateDefault()))
    ...

public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
{
    string? operationId = null;
    string? parentOperationId = null;

    [pull operationId and parentOperationId from 'traceparent' header]

    using (var operation = _telemetryClient.StartOperation<RequestTelemetry>(
            context.FunctionDefinition.Name, operationId, parentOperationId))
        {
        operation.Telemetry.Success = true;
        ....

        try
        {
            await next.Invoke(context);
        }
        catch (Exception ex)
        {
            operation.Telemetry.Success = false;
            _telemetryClient.TrackException(ex);
            throw;
        }
    }
}

We also set ResponseCode appropriately.

You can also remove duplicate request entries in host.json:

"logging": {
  "logLevel": {
    "Host.Results": "None"
  }
}

In order to get cosmos dependencies, we had to change connection mode to Gateway as per:
https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-dependencies#manually-tracking-dependencies.

Also not getting sql command text despite enabling legacy headers mentioned in 882.

[sebhaub]

[pull operationId and parentOperationId from 'traceparent' header]

hi @hagagps and thanks for your post, anyway i have some issues getting this to work.

Based on the post here 822#issuecomment iadded the ApplicationInsightsTelemetryWorkerService and enabled the legacy correlationHeaders, with this i see the dependencies in appinsights but still miss the correlation, so i tried to follow your approach here.

    serviceCollection.AddApplicationInsightsTelemetryWorkerService(opts =>
    {
        opts.DependencyCollectionOptions.EnableLegacyCorrelationHeadersInjection = true;
    });

can you explain this in more detail howyou achieved the dependency correlation to work properly?

thanks for any help.

[hagagps]

``> > [pull operationId and parentOperationId from 'traceparent' header]

hi @hagagps and thanks for your post, anyway i have some issues getting this to work.

Based on the post here 822#issuecomment iadded the ApplicationInsightsTelemetryWorkerService and enabled the legacy correlationHeaders, with this i see the dependencies in appinsights but still miss the correlation, so i tried to follow your approach here.

    serviceCollection.AddApplicationInsightsTelemetryWorkerService(opts =>
    {
        opts.DependencyCollectionOptions.EnableLegacyCorrelationHeadersInjection = true;
    });

can you explain this in more detail howyou achieved the dependency correlation to work properly?

thanks for any help.

We have ceased moving to NET6 Isolated for most of our applications with the many issues that are still open, especially around insights. For the application that was updated, we implemented the custom middleware shown above (StartOperation) to manually trace and pulled the traceparent header as follows:

    var data = context.BindingContext.BindingData;
        if (data.TryGetValue("Headers", out object? headers) && headers != null)
        {
            var headerDict = JsonConvert.DeserializeObject<Dictionary<string, string>>(
                headers.ToString() ?? string.Empty);
            if (headerDict != null && headerDict.TryGetValue("traceparent", out string? value))
            {
                var traceSegments = value?.Split("-");
                if (traceSegments?.Length == 4)
                {
                    operationId = traceSegments[1];
                    parentOperationId = traceSegments[2];
                }
            }
        }

The header should automatically be included when called from another azure application.

[cosminstirbu]

Did install the latest preview https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.ApplicationInsights/1.0.0-preview2 and most things are working as expected.

The only issue that we're still seeing is that the logging categories are not honoured (all of our own traces, or from EF Core, or from HttpClient have Function.<YOUR_FUNCTION_NAME>.User category).

[martinjt]

It would be really awesome if you could consider how telemetry works without ApplicationInsights by making OpenTelemetry a first class citizen like AWS Lambda have with ADOT lambda layer.

I'm more than happy for you to reach out to discuss what that could look like.

[SeanFeldman]

Linking this issue to #1182 to surface up issues. The Functions team should note that this is not just about "experience improvement"; it's also about getting the Azure bill under control. This issue should shift from "nice to have" to a "must have".

[mirekkukla]

Isolated function dependency tracking is now supported, although it doesn't work out-of-the-box (announcement here, for more details see #822).

Note that you'll have to configure things such that workers emit logs directly as detailed here.

[jviau]

The host OTel work will also cover improving the telemetry experience in workers.

Azure/azure-functions-host#9273

[fabiocav]

The core capabilities tracked by this issue have been added (announcement here.

The issue linked above by @jviau tracks additional enhancements related to OTel