Streaming: Tracing for logical groups of IoT data sequences

[jrauschenbusch]

I'm facing an issue with an IoT stream processing system which receives data from remote sensors. The sensor data is sent in individual sequences (one measurement per HTTP request), but there are logical groups of individual measurements which belong together. We want to have the ability to trace the latency in our system not only on individual requests but also on basis of the logic groups.

So my question is:
Is there a way to aggregate these individual traces into a single trace (artificial root span or anything) or is there an easy way to overwrite the trace id dynamically via a unique identifier for each sequence contained inside the HTTP request?

I'm using the OpenTelemetry Java agent. So basically currently i'm using the auto instrumentation and only the API dependency, but not the SDK dependency. Creating an Agent Extension and a custom IdGenerator seems not to work, as the initialization must be done before e.g. the HTTP request is handled. Maybe the Propagate API is a possibility, to inject a Trace Header before it is evaluated by the Java Agent.

[trask]

Would span links work for you to link the related spans together?

[jrauschenbusch]

This could work, if i would have a parent (root) span for all of the traces. But it seems that such a parent span cannot be created artificially. So there needs to be a component which export this parent trace. But with OpenTelemetry it seems that one is not able to create an artificial span and to export it. At least when you are using the Java Agent and API only instead of the SDK. Maybe with the SDK it would be possible.

Another way would be to link other existing traces of the same sequence. But then i would need to manage some kind of state to be able to lookup already existing spans for the same sequence id. This could be an option, but from an visual analysis point of view it would be better to set a new parent/root span to have a combined view for the whole sequence.

[trask]

But it seems that such a parent span cannot be created artificially.

I didn't follow why not, e.g. using OpenTelemetry API SpanBuilder?

[jrauschenbusch]

Ok. I've made it. I've used an internal cache in the web app w/ configurable retention to keep the SpanContext of the first span of a sequence. All subsequent spans are then attached to this initial span via a parent:child relation using SpanContext.createFromRemoteParent() together with spanBuilder.setParent(Context.root().with(remoteSpan)). I did no used links as i wanted to have a holistic view on Jaeger UI.

Another option would be to have a combined view. But this is currently not available: jaegertracing/jaeger#2544