Support Datadog's propagation headers

[dvgica]

Hi!

Datadog has their own style of propagation headers, poorly documented here. I would, frankly, prefer to ignore them because they're non-standard. But, if you use Datadog Synthetics, and wish to connect your Synthetic tests to your Datadog traces, it's necessary to read the trace ID from the incoming Synthetics request, namely from the x-datadog-trace-id header.

Kamon doesn't currently have support for reading or writing Datadog's headers. What do the maintainers think about adding it? We may be willing to take on that work at some point.

[SimunKaracic]

If you will implement it, we'll have no problem accepting the PR :)
It's not really something used widely enough for us to implement it ourselves, but contributions are more than welcome

[dvgica]

Awesome. I'm unsure if/when it will become a priority for us, but I'll update here if we start on it.

[dvgica]

My colleague @seglo has done some work on this, hopefully we can contribute this support soon!

[ivantopo]

Hey @dvgica, @seglo, just to understand something here... does this mean that Datadog sends a synthetic request to your applications and Kamon is expected to just adopt that same trace ID? I find it weird that they show x-datadog-parent-id: 0 in the document you linked, and it makes me feel like we only need to make sure the trace ID is the same.

I'm asking because we already have a setting for turning any HTTP header into a context tag and another setting for using any context tag as the preferred trace id.

Maybe, just maybe, this can be solved with two configuration settings:

kamon {
  propagation.http.default.tags.mappings {
    ddSyntheticsTraceID = "x-datadog-trace-id"
  }

  instrumentation.http-server.default.tracing {
    preferred-trace-id-tag = "ddSyntheticsTraceID"
  }
}

[dvgica]

does this mean that Datadog sends a synthetic request to your applications and Kamon is expected to just adopt that same trace ID?

Yeah I think that's the right interpretation. That said, Synthetics is not the only use case - we could have a downstream service calling us with properly set x-datadog-parent-id and x-datadog-trace-id. There's also x-datadog-sampling-priority which should be propagated.

[seglo]

Yes, we implemented a general HTTP propagation reader/writer for DataDog trace headers. Our environment is kind of weird. We have lots of services that will read and transmit both B3 and DataDog trace information in headers. It complicates things when we try to figure out places where distributed tracing fails, so I'm working on our internal implementation to sort that mess out. However, I also implemented a simple Propagation.EntryReader/Writerfor DataDog headers exclusively that I think would be worthwhile contributing to Kamon and could be configured like the other builtin implementations.

kamon.propagation.http.default.entries {
  incoming.span = "datadog"
  outgoing.span = "datadog"
}

[dvgica]

For reference this was contributed in #1145.