Periodic restart of SolrCloud pods

[mogul]

User Story

In order to implement an alternative to NIST control SI-3, the SSB should do rolling restarts of SolrCloud pods every 15 minutes.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I have administrative access to a provisioned EKS instance
  WHEN I check the ages of all the running SolrCloud pods
  THEN I see that no pod is older than 15 minutes
  AND I can see that the deployment had rolling restarts about 15 minutes apart.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]
See control SI-3 in our SSP.

Security Considerations (required)

This is an alternative to NIST control SI-3. By restarting pods periodically we're ensuring that any compromised pod is only useful for the attacker for a short window of time. An attacker would also have to generate more log traffic to keep compromising pods when they get restarted.

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

Solr-operator specific solution

• Use the solr-operator's native support for restarting pods

More general solution for any EKS workload

• Make sure everything critical is running more than one container in the pod
• Several options are discussed here.
  • A description of the options for a batch CronJob resource option. Such a job should be able to invoke the equivalent of kubectl -n {NAMESPACE} rollout restart deploy.
  • Terraform can manage the deployment of a CronJob.
  • The container used must have kubectl installed and be using a service account with enough permission to do the job.
  • Caution: Check the CronJob API version carefully as it stabilized in k8s 1.21, and at the time of writing we're using 1.19. 1.19's CronJob v1beta1 spec is here.

[mogul]

Nick had turned this on using the solr-operator's native support, but we turned it off while we were figuring out the persistent volume issues. It needs to be turned back on.

[mogul]

@nickumia-reisys this is back on, right? If so can we mark this done?