Invalid characters in Prometheus metrics (Jenkins)

[ramki88]

Bug Report

Describe the bug
Invalid characters gets added to the metric names [vm.memory.non-heap.usage.x100.window.5m & vm.memory.non-heap.used.window.15m]. Also fluentbit automatically adds a _sum to some of these metrics

metrics from fluentbit

# HELP +�����?y_����?y_non_heap_usage_x100_window_5m Generated from Dropwizard metric import (metric=vm.memory.non-heap.usage.x100.window.5m, type=jenkins.metrics.util.AutoSamplingHistogram)
# TYPE +�����?y_����?y_non_heap_usage_x100_window_5m summary
+�����?y_����?y_non_heap_usage_x100_window_5m{quantile="0.5"} 96
+�����?y_����?y_non_heap_usage_x100_window_5m{quantile="0.75"} 96
+�����?y_����?y_non_heap_usage_x100_window_5m{quantile="0.95"} 96
+�����?y_����?y_non_heap_usage_x100_window_5m{quantile="0.98"} 96
+�����?y_����?y_non_heap_usage_x100_window_5m{quantile="0.99"} 96
+�����?y_����?y_non_heap_usage_x100_window_5m_sum 0
+�����?y_����?y_non_heap_usage_x100_window_5m_count 432
# HELP �        ʡAy_non_heap_used_window_15m Generated from Dropwizard metric import (metric=vm.memory.non-heap.used.window.15m, type=jenkins.metrics.util.AutoSamplingHistogram)
# TYPE �        ʡAy_non_heap_used_window_15m summary
�       ʡAy_non_heap_used_window_15m{quantile="1.49227e+08"} 146495552
�       ʡAy_non_heap_used_window_15m{quantile="0.75"} 146678174
�       ʡAy_non_heap_used_window_15m{quantile="0.95"} 149220680
�       ʡAy_non_heap_used_window_15m{quantile="0.98"} 149226442.56
�       ʡAy_non_heap_used_window_15m{quantile="0.99"} 149226696
�       ʡAy_non_heap_used_window_15m_sum 0
�       ʡAy_non_heap_used_window_15m_count 432

Addition of _sum

+�����?y_����?y_non_heap_usage_x100_window_5m_sum 0

metrics from jenkins Prometheus

# HELP vm_memory_non_heap_usage_x100_window_5m Generated from Dropwizard metric import (metric=vm.memory.non-heap.usage.x100.window.5m, type=jenkins.metrics.util.AutoSamplingHistogram)
# TYPE vm_memory_non_heap_usage_x100_window_5m summary
vm_memory_non_heap_usage_x100_window_5m{quantile="0.5",} 97.0
vm_memory_non_heap_usage_x100_window_5m{quantile="0.75",} 97.0
vm_memory_non_heap_usage_x100_window_5m{quantile="0.95",} 97.0
vm_memory_non_heap_usage_x100_window_5m{quantile="0.98",} 97.0
vm_memory_non_heap_usage_x100_window_5m{quantile="0.99",} 97.0
vm_memory_non_heap_usage_x100_window_5m{quantile="0.999",} 97.0
vm_memory_non_heap_usage_x100_window_5m_count 568.0
# HELP vm_memory_non_heap_used_window_15m Generated from Dropwizard metric import (metric=vm.memory.non-heap.used.window.15m, type=jenkins.metrics.util.AutoSamplingHistogram)
# TYPE vm_memory_non_heap_used_window_15m summary
vm_memory_non_heap_used_window_15m{quantile="0.5",} 1.47329584E8
vm_memory_non_heap_used_window_15m{quantile="0.75",} 1.47381384E8
vm_memory_non_heap_used_window_15m{quantile="0.95",} 1.474519312E8
vm_memory_non_heap_used_window_15m{quantile="0.98",} 1.4748106272E8
vm_memory_non_heap_used_window_15m{quantile="0.99",} 1.47481992E8
vm_memory_non_heap_used_window_15m{quantile="0.999",} 1.47481992E8
vm_memory_non_heap_used_window_15m_count 568.0

• Steps to reproduce the problem:
  • Deploy jenkins in a k3s cluster using Jenkins Helm Chart
  • Install and enable the jenkins prometheus plugin
  • Configure fluentbit to scrape and export the jenkins metrics
    Expected behavior
    The output of fluent bit prometheus exporter and jenkins prometheus exporter should be the same

Your Environment

• Version used: 3.2.4
• Configuration:

    service: |
      [SERVICE]
          Daemon Off
          Flush {{ .Values.flush }}
          Log_Level {{ .Values.logLevel }}
          Parsers_File /fluent-bit/etc/parsers.conf
          Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
          HTTP_Server On
          HTTP_Listen 0.0.0.0
          HTTP_Port {{ .Values.metricsPort }}
          Health_Check On

    ## https://docs.fluentbit.io/manual/pipeline/inputs
    inputs: |
      [INPUT]
          name prometheus_scrape
          host cicd-apps-jenkins.jio-cloudxp-cicd
          port 8080
          tag apps.jenkins.*
          metrics_path /jenkins/prometheus/
          scrape_interval 10s

    ## https://docs.fluentbit.io/manual/pipeline/outputs
    outputs: |
      [OUTPUT]
          name prometheus_exporter
          match apps.jenkins.*
          host 0.0.0.0
          port 9195

• Environment name and version (e.g. Kubernetes? What version?): k3s v1.30.3+k3s1
• Server type and version: Linux 5.15.0-124-generic
• Operating System and version: Ubuntu 22.04.5 LTS
• Filters and plugins: None

Additional context