How to use k8s to dynamically expand capacity?

[gaecoli]

Is there a way to use k8s expand capacity? While a large of users to use redash, the workers will stuck at queue statu.

[rcheatham-q]

This is something I have toyed with and I believe it's possible using KEDA. You should be able to configure KEDA to read from your Celery broker (Redis or RabbitMQ) and scale your workers appropriately. You may need to create one deployment per queue to get full scalability, though. That will depend on the relative load of ad-hoc and scheduled queries.

[rcheatham-q]

Two seconds after writing this I discovered Scaling Modifiers which will allow a single KEDA ScaledObject to set the number of workers based on the total length of all queues.

[justinclift]

Bear in mind that modern Redash uses RQ, not Celery. No idea if that really changes things though. 😉

[rcheatham-q]

It doesn't, RQ uses Redis lists which is a builtin scaler to KEDA