lokiexporter: Support dynamic attributes for labels

[gramidt]

Is your feature request related to a problem? Please describe.
The Loki exporter by design forces a user to explicitly configure the attributes they want to allow as labels within Loki. This is a safety net to help prevent accidentally adding dynamic labels that may significantly increase cardinality, thus having a performance impact on their Loki instances. However, users may trust their downstream configurations, such as Fluentd or Stanza, to keep labels to a minimum.

Describe the solution you'd like
As a Loki user, I want to be able to filter attributes and have the exporter dynamically transform them into valid Loki labels, so that downstream systems do not have to sync their attributes within the exporter config.

Describe alternatives you've considered

• Keep the exporter as is.
• Just use expressions to match attribute names and not dynamically convert them. This would require the 'attributeprocessor' to be used in the pipeline for transforming attributes into valid Loki labels. This option still requires users to sync all attributes.

[gramidt]

@andrewhsu - Could you assign this to me?

[gramidt]

@bogdandrutu - Could you assign this to me?

[gillg]

Hello, I'm pretty according with you it could be an non default option.
But we should at least define some WhiteListed patterns to prevent missconfigs.
How do you imagine the config structure ?

Maybe it can be also a workaround for another bug I faced with lokiexporters and attributes with uppercase chars impossible to whitelist (more details here : open-telemetry/opentelemetry-collector#2594, the root cause seems dispatched in 2 new other issues)

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[jpkrohling]

cc @mar4uk

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

@mar4uk, @kovrus, could this be accomplished with the transform processor? Like, add a new Loki exporter hints as attributes based on other data?

[mar4uk]

probably this configuration should work:

processors:
  transform:
    log_statements:
      - context: log
        statements:
          - keep_keys(resource.attributes, ["service.name", "service.namespace"])
  resource:
    attributes:
      - action: insert
        key: service_name
        from_attribute: service.name

      - action: insert
        key: service_namespace
        from_attribute: service.namespace

      - action: insert
        key: loki.resource.labels
        value: service_namespace, service_name
exporters:
  loki:
    endpoint: "http://localhost:3100/loki/api/v1/push"
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [transform, resource]
      exporters: [loki]

transform processor can be used to filter attributes, resource processor can be used to promote attributes to loki labels.
Do you find this sufficient @gramidt? Please let me know if my example config doesn't solve an issue and if this issue is still actual :)

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

@gramidt will fixing this issues/19215 solve your case?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

I'm closing, as this doesn't seem to be relevant anymore. If the issue @mar4uk linked isn't enough to satisfy this use-case and if there's still interest in this, please leave a comment and I'll reopen.