Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chained spans for services-dependent tracing not working #97

Open
bangbaew opened this issue Aug 6, 2023 · 3 comments
Open

Chained spans for services-dependent tracing not working #97

bangbaew opened this issue Aug 6, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@bangbaew
Copy link

bangbaew commented Aug 6, 2023

Motivations

In Jaeger, we can look up on spans of services that depend on another services like this.
image
image

In this example, I made the requests from chant-service (using Rust - Actix Web Client) to dish-service, from dish-service (using Go - net/http) to similar-dish, from similar-dish (using Python - requests) to alert-service, and from alert-service (using Go - net/http) to an external service (Line Notify), which can be visualized in acyclic graph using Zipkin like this.
image

But when I use Reqwest wrapped in reqwest-tracing from this repo, to send a request from distancematrix-api (Rust) to dish-service, I only get individual spans, it doesn't chain to the depended services, and the chain span started in dish-service instead of distancematrix-api.
image
image

Solution

Make tracing spans created with Reqwest chainable if it's connected to another local services, and show remote hostname in Zipkin graph if it's connected to external services (like the notify-api.line.me case)

Alternatives

This is the working Go net/http code for creating chained spans,
image

This is the working Python requests code for creating chained spans,
image

This is the working Rust AWC code for creating chained spans,
image

Additional context

This is the Rust reqwest code that doesn't create chained spans,
image
image

@bangbaew bangbaew added the enhancement New feature or request label Aug 6, 2023
@tl-rodrigo-gryzinski
Copy link
Contributor

Hey! You're supposed to be seeing the traces across service boundaries, did you enable any of the otel features?

@alexhumphreys
Copy link

Hi! I think I'm experiencing the same issues. I've tried to make a minimal example in the repo reqwest-middleware-example. It basically stands up 2 axum servers (server-a and server-b), and a jaeger instance for traces, and sends requests from server-a to server-b using various methods.

If I explicitly set the traceparent header (as done with the /proxy_add_headers endpoint), then the spans cross the two servers correctly. However if I just use reqwest-middleware (as with /proxy_middleware endpoint) then I only see the spans from server-a and server-b separately.

In the following screenshot, you can see the traces appearing to cross server-a and server-b for the /proxy_add_headers endpoint, but not for the /proxy_middleware endpoint.

Screenshot 2023-11-22 at 20 37 02

As per the last comment, I have the feature opentelemetry_0_18 enabled in the Cargo.toml for server-a here.

Is there something else I'm missing? Any help on this would be appreciated.

@vdebergue
Copy link

vdebergue commented Feb 1, 2024

I had the same issue at first, the trace context was not propagated in the headers. I then realized that in my Cargo.toml the version of the feature was not matching the opentelemetry one. After fixing this, it worked correctly.

So make sure that the opentelemetry version is aligned:

opentelemetry = { version = "^0.21" }
reqwest-tracing = { version = "^0.4", features = ["opentelemetry_0_21"] }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

4 participants