Using pyrra with thanos multi cluster setup

[BradErz]

Hey there,

We are using Thanos for our metrics and we use it to have long-term storage and a single data source in grafana with a selector by cluster. You can see a diagram of our setup here:

flowchart TB

    subgraph infra[prod-infra-us-east4]
        grafana-infra[grafana]-->thanos-query-frontend-infra[thanos-query-frontend]

        thanos-query-frontend-infra[thanos-query-frontend]-->thanos-query-infra[thanos-query]

        thanos-query-frontend-infra[thanos-query-frontend]-->redis-thanos

        thanos-query-infra[thanos-query]-->thanos-store

        thanos-store-->redis-thanos

        thanos-query-infra[thanos-query]-->thanos-sidecar-x-infra[thanos-sidecar]

        subgraph prom-x-infra[prometheus-x]
          thanos-sidecar-x-infra[thanos-sidecar]-->promx-x-infra[prometheus]
        end
    end

    subgraph all-other-clusters[All other clusters]
        subgraph prom-x[prometheus-x]
          thanos-sidecar-x[thanos-sidecar]-->prom-xx[prometheus]
        end
    end

   thanos-sidecar-x-infra-->gcs[Google cloud storage]
   thanos-sidecar-x[thanos-sidecar]-->gcs
   thanos-store-->gcs
   thanos-query-infra-->thanos-sidecar-x[thanos-sidecar]

Loading

We were looking to see if we could deploy the pyrra component like we do grafana and have a selector for "cluster" in the pyrra UI.

Is this possible in the current version of pyrra? From looking at the way the UI component is currently set up it doesn't seem like it.

Ideally, I would like to deploy pyrra in kubernetes mode in our leaf clusters to do the recording rule generation from CRDs. And then deploy pyrra in api mode in our prod-infra-cluster. It seems the UI currently tries to speak directly pyrra kubernetes component but it can also only speak with one of them.

I guess in this case we should probably just try to use the grafana dashboards etc instead of using the pyrra UI? (it's a shame though the UI is pretty and I like it haha)

Kind Regards,
Bradley.

[ArthurSens]

You can deploy pyrra and set -prometheus-url to the address of your thanos-query.

When setting up your SLOs, be sure to add grouping: cluster to them. The grouping will split your SLOs, creating for each label value that you have stored for the cluster label

[BradErz]

Thanks for the reply @ArthurSens!

Yep we are already setting the PrometheusURL to the thanos-query so we get longer metric visibility. We are not currently setting the grouping. Will investigate that thanks!

In that case though what would you set for the value https://github.com/pyrra-dev/pyrra/blob/main/main.go#L56?

[metalmatze]

I actually run Pyrra with a Thanos cluster myself. ☺️
--api-url should still point to the Pyrra Backend container. Most likely you're running the Kubernetes backend, so something like --api-url=http://pyrra-kubernetes.monitoring.svc.cluster.local:9444

The prometheus-url you want to set to the Thanos Query Frontend:
--prometheus-url=http://thanos-query-frontend.monitoring.svc.cluster.local:9090

Little additional info: Pyrra actually sets Thanos specific request headers to make the best of Thanos' architecture:

pyrra/main.go

Lines 351 to 390 in ffb5f03

	func (c *thanosClient) Do(ctx context.Context, r *http.Request) (*http.Response, []byte, error) {
	if r.Body == nil {
	return c.client.Do(ctx, r)
	}
	body, err := io.ReadAll(r.Body)
	if err != nil {
	return nil, nil, fmt.Errorf("reading body: %w", err)
	}
	query, err := url.ParseQuery(string(body))
	if err != nil {
	return nil, nil, fmt.Errorf("parsing body: %w", err)
	}
	// We don't want partial responses, especially not when calculating error budgets.
	query.Set("partial_response", "false")
	r.ContentLength += 23
	if strings.HasSuffix(r.URL.Path, "/api/v1/query_range") {
	start, err := strconv.ParseFloat(query.Get("start"), 64)
	if err != nil {
	return nil, nil, fmt.Errorf("parsing start: %w", err)
	}
	end, err := strconv.ParseFloat(query.Get("end"), 64)
	if err != nil {
	return nil, nil, fmt.Errorf("parsing end: %w", err)
	}
	if end-start >= 28*24*60*60 { // request 1h downsamples when range > 28d
	query.Set("max_source_resolution", "1h")
	r.ContentLength += 25
	} else if end-start >= 7*24*60*60 { // request 5m downsamples when range > 1w
	query.Set("max_source_resolution", "5m")
	r.ContentLength += 25
	}
	}
	encoded := query.Encode()
	r.Body = io.NopCloser(strings.NewReader(encoded))
	return c.client.Do(ctx, r)
	}

[PabloPie]

I think the problem is that with the setup shown by @BradErz, if we want to create SLOs in the two clusters we need to deploy twice all of the components from Pyrra, including the UI. What would be interesting to do is having a single UI pointing to multiple clusters so that we can centralize the visualization of all SLOs.

[metalmatze]

Yes, agreed. For that purpose I would run a single Thanos querier entrypoint and then run the Thanos Ruler against that too. That way the Ruler can work with a global view on all metrics too. Let me know if you need more details for that.