json: ser/de bytes as base64 strings not an array of bytes

[adriangb]

I believe this is how the proto <-> json spec defines that binary data should be handled: https://protobuf.dev/programming-guides/json/

I confirmed by exporting data from Python:

from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
from opentelemetry.proto.common.v1.common_pb2 import AnyValue, KeyValue
from opentelemetry.proto.resource.v1.resource_pb2 import Resource
from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans
from google.protobuf.json_format import MessageToJson


export = ExportTraceServiceRequest(
    resource_spans=[
        ResourceSpans(
            resource=Resource(
                attributes=[
                    KeyValue(key="key", value=AnyValue(bytes_value=b"value"))
                ]
            )
        )
    ]
)

json = MessageToJson(export, use_integers_for_enums=True)

print(json)

with open("export.json", "w") as f:
    f.write(json)

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "key",
            "value": {
              "bytesValue": "dmFsdWU="
            }
          }
        ]
      }
    }
  ]
}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.1%. Comparing base (775f1f9) to head (42807da).
Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #2471   +/-   ##
=====================================
  Coverage   79.1%   79.1%           
=====================================
  Files        118     118           
  Lines      22508   22508           
=====================================
  Hits       17821   17821           
  Misses      4687    4687           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[adriangb]

I'm not familiar with the codebase so I took a guess as to where to apply the fix. I tried to add an integration test but I could not get a failure case to fail a test so I'm not sure if that's even in the right place either. Some pointers would be appreciated.

[adriangb]

Hi @lalitb @TommyCpp would one of you mind reviewing or pinging an appropriate reviewer (sorry if this is not relevant to you, I just went off of git history)? Thanks

[adriangb]

@lalitb I can't seem to get tests to agree on the order of items in attributes arrays. I'm not sure if they're getting sorted in one place but not another or something. Otherwise all tests seem to be passing. I'll point out that I had to implement bytes support for the rust sdk itself, maybe that should be it's own PR but if so it will block this PR.

[adriangb]

Hi, just checking if there's anything else I can do to move this along. Thanks!

[adriangb]

We've got some conflicts now. I'm happy to resolve them, or split this PR up, but I need a bit of guidance on how you'd like to proceed. Thanks!

[adriangb]

Hi folks pinging again. I'm willing to do the work here and this seems like an important issue (invalid data being generated) but I need some input from the dev team, this is my first PR to the repo. cc @cijothomas as main commiter.

[lalitb]

Hi folks pinging again. I'm willing to do the work here and this seems like an important issue (invalid data being generated) but I need some input from the dev team, this is my first PR to the repo. cc @cijothomas as main commiter.

@adriangb Sorry on delay on this. I believe the CI failure is for the ordering of resource attributes. But see my comment here if that makes sense.

[cijothomas]

@adriangb Thanks for helping!
I am curious to know if you are impacted by this or you are pro-actively addressing this? Given the appenders maintained in this repo does not today produce value of type array of bytes, I am curious - are there other ways people are using the crates?

[adriangb]

I use the crate to load telemetry exported as JSON by a browser SDK that follows the OTLP JSON spec, so yes I was hitting this in production.

The bigger picture is that I use the crate as part of the ingestion portion of OTEL backend written in Rust: https://pydantic.dev/logfire

[adriangb]

And thank you both for getting this across the line!

[cijothomas]

https://pydantic.dev/logfire

Thanks for sharing more context. We were earlier considering removing publishing the opentelemetry-proto crate itself, as it was considered an internal dependency for the opentelemetry-otlp itself! But later decided to keep it!

[adriangb]

I'm glad you did!

[lalitb]

Yes, interestingly, this crate is being utilized more than we initially expected. Another example: #2487.

[cijothomas]

Created an issue to track getting it to 1.0
#2615

@adriangb If you have bandwidth, please do check and see any gaps before we can make it a stable crate. (ofcource, the opentelemetry and opentelemetry_sdk crates must be 1.0 as a pre-req, and that is scheduled by mid year.)