Alerting on cache / memcached getting full for blocks storage

[shybbko]

In https://cortexmetrics.io/docs/chunks-storage/caching/#chunks-cache it is suggested to alert for possible Memcached overload based on cortex_ingester_chunk_size_bytes_sum and cortex_ingester_chunk_size_bytes_count metrics. This however is for chunk storage, which is deprecated and for blocks storage those metrics are not exposed / are equal zero.

Is there a similar query / alert / rule that would be applicable for blocks storage?

Thanks

[friedrich-at-adobe]

yes, if you are getting timeouts, you should scale up the memcached replicas

https://github.com/cortexproject/cortex-jsonnet/blob/978fe497e328c78a41fc710410cd479dd38abb5d/cortex-mixin/alerts/alerts.libsonnet#L175-L190

https://github.com/cortexproject/cortex-jsonnet/blob/main/cortex-mixin/docs/playbooks.md#cortexmemcachedrequesterrors

[shybbko]

Gotcha. Thanks!