How to use a middleware pattern for Span#makeCurrent?

[mikelane]

In our backend service we have a middleware pattern in which we register handlers to a handler chain. These handlers all implement a Handler interface that consists of a before and after method. The before methods are called in the order that the handlers are registered before the request is handled and the after methods are called in reverse order.

In the otel documentation and code, we see the pattern for building a span and registering that span as the current span is something like the following:

  public doTheFoo():
    Span span = tracer.spanBuilder("OurRelevant:Name:Here")
        .setParent(Context.current().with(Span.current())
        .startSpan();

    try (Scope scope = span.makeCurrent()) {
      // do whatever work
    } catch (Exception e) {
      // handle exceptions
    } finally {
      span.end();
    }

The Span#makeCurrent method has an @MustBeClosed annotation which means the span.makeCurrent() call must happen as a try-with-resource block. That's fine, except that doesn't work in our middleware pattern. Looking at the code, it seems like the only method being called is ContextStorage.get().attach(span), so we tried something like this in our before method:

    Context incoming =
        openTelemetry
            .getPropagators()
            .getTextMapPropagator()
            .extract(Context.current(), headers, textMapGetterInstance);

    scope = ContextStorage.get().attach(incoming);

But this begs the question, how do we store the scope in a thread safe way that allows us to call

    scope.close()

in our after method. Merely naively setting scope up as a class field and doing the above did not seem to do anything at all.

What's the best way for us to proceed with this?

[jkwatson]

This is definitely the hardest part of writing instrumentation. If you create a Scope, you must close it on the same thread that it was opened on. How you accomplish that is definitely going to depend very strongly on the threading model of your application server (or whatever framework you're using).

There definitely isn't a one-size-fits-all answer to this question, unfortunately. If you can, I recommend putting together an example that could be reviewed by the instrumentation experts.

One thing that I will say about your code snippet, though, is that you shouldn't ever need to interact with ContextStorage directly under normal circumstances. Everything you need should be available on the Context interface itself.

[mikelane]

Thank you for your answer. We were ultimately able to solve this by making use of InheritableThreadLocal to hold the span in the before method and then to access the span again in the after method.

[anuraaga]

Yes you can think of the MustBeClosed annotation as you need to know what you are doing. If the before and after are guaranteed to be executed on the same thread synchronously, you can ignore that, probably leaving a comment that you verified the threading behavior.

[mikelane]

I thought I had read that the MustBeClosed annotation would result in a compilation-time error if the object was not implemented in a try-with-resources block. But I've read so much in the past couple of days. I might be mis-remembering that.

[jkwatson]

That'll depend on your compiler and settings, I think. I don't think a plain javac would enforce that, but errorprone might.