Splunk Operator: Need ability to restart a percentage of Indexers on updates instead of all indexer pods at once.

[cderocco5]

Please select the type of request

Feature Request

Tell us more

Describe the request

• When making image or affinity changes or other CR changes that require pod recreation. I need the ability to specify a percentage of the indexer pods to recreate. Right now, these operations cause all the indexer pods to recreate at once. When you have over 450 indexer pods, having them restart at once puts a lot of pressure on Azure Kubernetes Service (AKS) API server which causes requests to get throttled by AKS.

Expected behavior

• I need to ability to specify a percentage in the IndexerCluster CR to say update 10% or 33% of the indexers at a time. Until the IndexerCluster reconciles and all the indexer pods are recreated in batches.

Splunk setup on K8S

• 15 Searchheads. 455 Indexers running on Azure Kubernetes Service. Running Splunk 9.1.2

Reproduction/Testing steps

• Change affinity rules, specify new docker image, or any changes you make in the IDXC CR that cause indexer pods to recreate.

K8s environment

• Running on Azure Kubernetes Service. K8s version 1.28.0

Proposed changes(optional)

• Allow ability to update indexer pods in batches by specifying a percentage of indexers to recreate on updates in the IDXC CR.

[vivekr-splunk]

@cderocco5 thank you for your request, we will look into this issue and get back to you.

[cderocco5]

It will be nice to have this functionality on searchheads too. Allowing users to update multiple searchheads at once instead of 1 at a time.