How to send OPA Decision Logs to OpenTelemetry

[awalmsley]

Hello, I want to send OPA Decision logs to AWS CloudWatch via OpenTelemetry, looking for suggestions to implement this. Is there any support built in to OPA, or would it be necessary to implement a custom plugin. Are there any examples of doing this? any suggestions welcome

[anderseknert]

Hi @awalmsley! There's no support for that in OPA currently. If you'd want to implement that you could either write a custom plugin as you suggest, or write a proxy server that receives the decision logs from OPAs HTTP client, and then forwards (and possibly transforms) the messages to whatever backend you desire. The latter has the benefit of not having to maintain a custom version of OPA, but OTOH it obviously adds another component too.

[agardnerIT]

This is interesting. I believe the flow here would be:

1. Write a plugin (I have no idea how) which lets us configure an endpoint (ie. http://OTEL-COLLECTOR-URL:4318)
2. Logs are pushed to the OTEL collector (which I believe supports gzip)
3. Something needs to unzip the gzip and translate this:

{
    "labels": {
      "app": "my-example-app",
      "id": "1780d507-aea2-45cc-ae50-fa153c8e4a5a",
      "version": "v0.13.5"
    },
    "decision_id": "4ca636c1-55e4-417a-b1d8-4aceb67960d1",
    "bundles": {
      "authz": {
        "revision": "W3sibCI6InN5cy9jYXRhbG9nIiwicyI6NDA3MX1d"
      }    
    }, 
    "path": "http/example/authz/allow",
    "input": {
      "method": "GET",
      "path": "/salary/bob"
    },
    "result": "true",
    "requested_by": "[::1]:59943",
    "timestamp": "2018-01-01T00:00:00.000000Z"
  }

to this.

I think this would require the creation of another custom OpenTelemetry collector plugin, which would take the incoming gzip, parse and transform it, Then the standard otlphttp output could send it on it's way.

---

I'm starting to think it'll just be easier to write a new standalone API that:

1. OPA is configured to send gzipped output to this new service
2. This new service unpacks the gzip, transforms it to OTEL compatible log entries (we can leverage the logpusher logic for this)
3. This new service sends the logs to the OTEL collector endpoint

Of course, as an alternative to both of those, OPA could just choose to adjust the decision log format to be OTEL compatible (as the standard, this should be a future-proof decision).

The decision log endpoint would then be set at the OTEL collector (Which already supports gzip) and no plugins needed.

Using this third option, OPA is "almost automatically" OpenTelemetry (and any backend) compatible "out of the box".

[anderseknert]

Yep, the second option would be the proxy server I mentioned.
OPA already has the option to leverage OTEL for.. well, telemetry 🙂
Is it common to use it as a logging backend as well?

[awalmsley]

Thanks a lot for the advice. I'm trying the Custom Decision Log Plugin approach. I'm using the OpenTelemetry-Go SDK to send the decision log events to OTel. I couldn't find a way to send the logs without attaching the log entries to trace spans. There must be a simpler way (??). The spans add a lot of unnecessary detail..

import (
  ...
	"go.opentelemetry.io/otel"
)
  ...
  // Log is called by the decision logger when a record (event) should be emitted. The logs.EventV1 fields
  // map 1:1 to those described in https://www.openpolicyagent.org/docs/latest/management-decision-logs
  func (d *DecisionLogger) Log(ctx context.Context, event logs.EventV1) error {

	jsonData, err := json.MarshalIndent(event, "", "  ")
	if err != nil {
		fmt.Printf("Could not marshal json: %s\n", err)
		return err
	}
    ...
  	tracer := otel.Tracer("decision-log-tracer")

  	ctx, span := tracer.Start(ctx, "decision-log-span")
  	defer span.End()

  	attrs := []attribute.KeyValue{
  		attribute.String("decision-log", string(jsonData)),
  	}

  	span.AddEvent("decision-log-event", trace.WithAttributes(attrs...))
    ...

The traces are sent to an OpenTelemetry collector, using an OLTP receiver. Here's what the decsion logs look like in the OTel container logs. Haven't had to add any translation logic yet..

2023-06-15T22:34:38.759Z        info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
2023-06-15T22:34:38.759Z        info    ResourceSpans #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(decision-log-service)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope decision-log-tracer 
Span #0
    Trace ID       : 5b23c9c13aa506680ed1f57357c30428
    Parent ID      : 
    ID             : 00055abd26ff99d5
    Name           : decision-log-span
    Kind           : Internal
    Start time     : 2023-06-15 22:34:38.564398 +0000 UTC
    End time       : 2023-06-15 22:34:38.564418937 +0000 UTC
    Status code    : Unset
    Status message : 
Events:
SpanEvent #0
     -> Name: decision-log-event
     -> Timestamp: 2023-06-15 22:34:38.564413 +0000 UTC
     -> DroppedAttributesCount: 0
     -> Attributes::
          -> decision-log: Str({
  "labels": {
    "id": "816259e6-a550-4d67-85f1-1929af0af4e4",
    "version": "0.53.1"
  },
  "decision_id": "5cee3f82-677a-49b6-a740-4a879f75a2cd",
  "path": "foo/allow",
  "input": {
    "user": "john"
  },
  "result": true,
  "erased": [
    "/input/password"
  ],
  "requested_by": "127.0.0.1:59803",
  "timestamp": "2023-06-15T22:34:38.54376Z",
  "metrics": {
    "counter_server_query_cache_hit": 0,
    "timer_rego_input_parse_ns": 121772,
    "timer_rego_module_parse_ns": 1679,
    "timer_rego_query_compile_ns": 146283,
    "timer_rego_query_eval_ns": 89953,
    "timer_rego_query_parse_ns": 79053,
    "timer_server_handler_ns": 558795
  },
  "req_id": 1
})
        {"kind": "exporter", "data_type": "traces", "name": "logging"}

[srenatus]

Heya, thanks for providing details.

I think this is the wall you've been hitting: the go.opentelemetry.io packages have no support for logs, but only traces and metrics. And I think DL maps best to logs. There are a couple of options:

1. write a custom DL plugin that uses gRPC or HTTP to feed logs into opentelemetry-collector (another process, adequately configured). See here for an example payload
2. configure opentelemetry-collector to read the DLs from OPA's console output (configuring its decision logger to output to the console)