How to perform cross-service tracing

[AH-dark]

Feature Request

Crates

• volo
• volo-grpc
• tracing
• tracing-opentelemetry
• opentelemetry

Motivation

In a multi-microservice architecture, link tracing cannot be connected across services. I am using tracing and tracing-opentelemetry, but they do not automatically inject the trace ID into the request metadata.

Proposal

I hope that Volo can implement multi-service tracing, similar to Kitex and Hertz.

Alternatives

Yes, I can manually modify the request metadata, but that’s not an ideal solution.

[PureWhiteWu]

Hi, sorry I'm not familiar with opentelemetry, is there any rfc that describes how to implement multi-service tracing?

[AH-dark]

Hi, sorry I'm not familiar with opentelemetry, is there any rfc that describes how to implement multi-service tracing?

Kitex extracts trace IDs and span IDs from ctx via middleware, inserts them into the client request metadata, and then the server retrieves this metadata to inject back into ctx, thus enabling continuous cross-service tracing. In Rust, there’s no direct ctx equivalent, so typically a thread-local global variable is used for context handling. However, because the Layer in Rust does not offer a method to retrieve metadata, I’m uncertain how to effectively “inject into ctx”. Furthermore, since Rust’s tracing system relies on annotations, injecting trace IDs outside the service function requires operations at the Layer level.

[PureWhiteWu]

For context handling, you may use the metainfo task local variable. You can retrieve it in the layer.

This task local is initialized by the volo framework.

But I don't know what to inject, is there any rfc about this?

[CoderPoet]

Hi, sorry I'm not familiar with opentelemetry, is there any rfc that describes how to implement multi-service tracing?

Kitex extracts trace IDs and span IDs from ctx via middleware, inserts them into the client request metadata, and then the server retrieves this metadata to inject back into ctx, thus enabling continuous cross-service tracing. In Rust, there’s no direct ctx equivalent, so typically a thread-local global variable is used for context handling. However, because the Layer in Rust does not offer a method to retrieve metadata, I’m uncertain how to effectively “inject into ctx”. Furthermore, since Rust’s tracing system relies on annotations, injecting trace IDs outside the service function requires operations at the Layer level.

In Rust, thanks to the presence of the task-local mechanism, there’s no need to explicitly pass context as a parameter in method calls. We can simply perform Inject & Extract operations on the context within the appropriate Layer by accessing it from task-local storage. Metadata transmission can be efficiently handled using the cloudwego/metainfo library.

Additional Notes on Layer Integration. For the server side, you can implement the Layer trait directly, where OpenTelemetry-related operations can be integrated. By implementing this trait, you’ll be able to incorporate OpenTelemetry’s tracing capabilities, allowing seamless context propagation and distributed tracing across services without needing to manually pass metadata.

[CoderPoet]

For context handling, you may use the metainfo task local variable. You can retrieve it in the layer.

This task local is initialized by the volo framework.

But I don't know what to inject, is there any rfc about this?

OpenTelemetry provides a default RFC implementation following the latest W3C specification, which you can find here: W3C Trace Context Propagation.

Additionally, OpenTelemetry in Rust allows flexibility to extend and implement other RFCs. For example, it supports the Jaeger context specification, as seen here: Jaeger Propagator.

[AH-dark]

For context handling, you may use the metainfo task local variable. You can retrieve it in the layer.
This task local is initialized by the volo framework.
But I don't know what to inject, is there any rfc about this?

OpenTelemetry provides a default RFC implementation following the latest W3C specification, which you can find here: W3C Trace Context Propagation.

Additionally, OpenTelemetry in Rust allows flexibility to extend and implement other RFCs. For example, it supports the Jaeger context specification, as seen here: Jaeger Propagator.

Perhaps you could implement the OpenTelemetry Extractor & Injector trait for MetadataMap? I’d be happy to help.

[Millione]

I don't think this needs to be implemented in volo itself, you can do it in your application with a wrapper type that implements Extractor & Injector?

[AH-dark]

I don't think this needs to be implemented in volo itself, you can do it in your application with a wrapper type that implements Extractor & Injector?

Why can’t Volo, like Kitex, integrate a wide array of external services to enhance usability and ecosystem richness? Perhaps this could be achieved through feature flags?

[PureWhiteWu]

@AH-dark In our design, those ecosystem integrations are supposed to be placed under github.com/volo-rs. We just don't have manpower to implement them by ourselves.

All contributions are welcomed!

[AH-dark]

@AH-dark In our design, those ecosystem integrations are supposed to be placed under github.com/volo-rs. We just don't have manpower to implement them by ourselves.

All contributions are welcomed!

But I’m unable to create a new repository. 😢

[PureWhiteWu]

@AH-dark Hi, which component do you want to implement? I can create a repo for you.

[AH-dark]

@AH-dark Hi, which component do you want to implement? I can create a repo for you.

Just an implementation similar to #527.

[PureWhiteWu]

Great! So maybe we can call it volo-opentelemetry?

[AH-dark]

Maybe you need to consider whether to merge PR or create a new repository? 🤔

[PureWhiteWu]

I think creating a new repo for this is more suitable?

[AH-dark]

I think creating a new repo for this is more suitable?

Well, I mean, you ByteDance need to discuss whether to accept my PR or open a separate Repo to implement it.

[PureWhiteWu]

Oh, sorry maybe I didn't explain it clearly, I mean we can open a separate repo to implement it, and you can just move your code there. What do you think?

[AH-dark]

Oh, sorry maybe I didn't explain it clearly, I mean we can open a separate repo to implement it, and you can just move your code there. What do you think?

I think this is great.

[PureWhiteWu]

Hi, I've created an repo https://github.com/volo-rs/volo-opentelemetry and invited you as an maintainer~

You may move your code there~

Thanks very much for your contribution!