Series limits in queries are difficult to understand

[damnever]

Describe the bug

Users have no knowledge of whether vertical sharding works for certain queries. However, since series limits are counted independently, sharded queries might have a lower likelihood of triggering the series limit compared to non-sharded queries. Another issue is that the ruler lacks vertical sharding capability, unlike the query-frontend.

To Reproduce

When vertical sharding is enabled, the following query might work fine:

sum by (cluster) (rate(api_total[5m]))

However, this query will fail with the error 'the query hit the max number of series limit':

rate(api_total[5m])
# or sum(rate(api_total[5m]))

Expected behavior

Series limits should be easy to understand and have a clear explanation about which level of the series they are limiting or counting.

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
• Deployment tool: [e.g., helm, jsonnet]

Additional Context

[yeya24]

Hi @damnever, thanks for the issue.

I agree it is something to improve. We have thought about having a global limit that count limit across shards instead of doing it per shard but that's not a trivial change.

Do you have any suggestions?

[damnever]

I personally prefer to use the existing one as the global limit, however, this might break compatibility.

[damnever]

@yeya24 another issue is that the series limit in the store-gateway is a simple sum of the block level counts(thanos code), not the unique series count since each block may contain a portion of the same series.

thanos-io/thanos#8011

[damnever]

@yeya24 another issue is that the series limit in the store-gateway is a simple sum of the block level counts(thanos code), not the unique series count since each block may contain a portion of the same series.

thanos-io/thanos#8011

It might hurt the performance, but a solution could be to compare the limit independently with each block without summing them, and then allow the querier to count the unique series.