Skip to content

Latest commit

 

History

History
57 lines (37 loc) · 3.17 KB

custom-prom.md

File metadata and controls

57 lines (37 loc) · 3.17 KB
Raw

Custom Prometheus

Integrating Kubecost with an existing Prometheus installation can be nuanced. We recommend first installing Kubecost with a bundled Prometheus (instructions) as a dry run before integrating with your own Prometheus. We also recommend getting in touch ([email protected]) for assistance.

Note: integrating with an existing Prometheus is officially supported under all Kubecost paid plans.

Required Steps

  1. Copy values.yaml and update the following parameters:

    promtheus.fqdn to match your local Prometheus with this format http://<prometheus-server-service-name>.<prometheus-server-namespace>.svc.cluster.local
    prometheus.enabled set to false

    Pass this updated file to the Kubecost helm install command with --values values.yaml

  2. Have your Prometheus scrape the cost-model /metrics endpoint. These metrics are needed for reporting accurate pricing data. Here is an example scrape config:

- job_name: kubecost
      honor_labels: true
      scrape_interval: 1m
      scrape_timeout: 10s
      metrics_path: /metrics
      scheme: http
      dns_sd_configs:
      - names:
        - kubecost-cost-analyzer.<namespace-of-your-kubecost>
        type: 'A'
        port: 9003

This config needs to be added under extraScrapeConfigs in Prometheus configuration. Example

You can confirm that this job is successfully running with the Targets view in Prometheus.

Prometheus Targets

Recording Rules

Kubecost uses Prometheus recording rules to enable certain product features and to help improve product performance. These are recommended additions, especially for medium and large-sized clusters using their own Prometheus installation.

Troubleshooting Issues

Common issues include the following:

  • Wrong Prometheus FQDN: evidenced by the following pod error message No valid prometheus config file at .... We recommend running curl <your_prometheus_url>/api/v1/status/config from a pod in the cluster to confirm that your Prometheus config is returned. If not, this is an indication that an incorrect Prometheus Url has been provided. If a config file is returned, then the Kubecost pod likely has it's access restricted by a cluster policy, service mesh, etc.

  • Prometheus throttling -- ensure Prometheus isn't being CPU throttled due to a low resource request.

  • Required dependancy versions (node-exporter - v0.16, kube-state-metrics - v1.3, cadvisor)

  • Missing scrape configs -- visit Prometheus Target page (screenshot above)

You can visit Settings in Kubecost to see basic diagnostic information on these Prometheus metrics:

Prometheus status diagnostic