Ensure High/Critical vulnerabilities in Managed Node Groups are mitigated ASAP

[mogul]

User Story

In order to ensure that High or Critical findings on live EC2 instances are addressed, the data.gov team want to automatically install patched packages that address findings whenever they are available.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I am logged into the AWS Console
  AND I am looking at AWS Inspector
  AND I see a CVE detected for which an updated package is available
  WHEN the next maintenance window passes
  THEN I see that the CVE is no longer detected

Background

Amazon periodically releases new AWS Linux 2 AMIs for EKS that incorporate updates. However, some updates may be available between AMI versions that address CVEs, etc. We want to make sure that these updates, if they address a High or Critical finding, are installed on running instances even between the AMI update schedule. AWS Systems Manager provides a subset of functionality called Patch Manager that does this. We just need to turn it on and make sure it's running the update task on all of our EC2 instances.

Out of scope:

• Ensuring that our Managed Node Groups are restarted with the latest AMI for our EKS version as soon as a new one becomes available.
• Ensuring that our EKS control plane is running the latest k8s version available.

Security Considerations (required)

This work will make our compliance with the SI family of controls much tighter. No real downside identified.

Sketch

• In datagov-brokerpak-eks
  • configure an aws_ssm_maintenance_window_target
  • configure an aws_ssm_maintenance_window_task that will run the default patch baseline task provided by AWS SSM Patch Manager on all images tagged with eks:cluster-name equal to the cluster name during that window

[mogul]

A nice example module for doing this:
https://github.com/cloudposse/terraform-aws-ssm-patch-manager

[mogul]

If we do end up having to define a policy for the task, there's an example of what it needs to look like here;
https://github.com/osodevops/aws-terraform-module-ssm-automated-patching/blob/master/aws_patching_iam_policy_document.tf

[mogul]

Iterated a bit on an SSM-focused IAM policy in the AWS Console (GSA/datagov-ssb#133) and tag-based target selection (GSA-TTS/datagov-brokerpak-eks#85) until the branch that @nickumia-reisys pushed was working locally.

[mogul]

@hkdctol: This is the issue where we set up automatic remediation of Critical and High vulnerabilities detected in our managed nodes, in which EKS runs. (The initial scans turned up just one CVE, which has already been remediated.)

Since we never install custom code on these nodes, we expect that AWS' continuing release of patches and our ongoing adoption of new AMI baselines (which we will also automate) mean we may never have to touch them beyond this.

[mogul]

Confirmed working... We saw an OpenSSL CVE in the manually-deployed EKS. You can see the "41" here referring to how many hosts had the finding.

We triggered the update manually (it's automatically triggered during a maintenance window at 1pm ET), and we were able to watch the number go down gradually....

Inspector now reports they're all clean.

QED

[hkdctol]

@nickumia-reisys on Bret's points in the above comments -- do you have screenshots or something else to export that I can set to Ryan?

[mogul]

I made sure Ryan saw the posts and screenshots above. From DM with him:

[hkdctol]

@mogul does Ryan need anything else from us before we cut over catalog to cloud.gov version?

[mogul]

He wants to see us use the GSA CIS-hardened machine images for our clusters.

He's also asking about container-scanning, but AFAIK he grudgingly agrees that was not part of the original ATO conditions. I've copied you on those DMs privately.

[rpalmer-gsa]

@mogul I think you indicated that you were going to use the hardened images provided by ISE. Container scanning isn't part of original ATO conditions, but its not a sustainable gap long term.

[hkdctol]

@rpalmer-gsa we've done that at #3668 and will cut over to cloud.gov unless we hear otherwise. Thanks!

[nickumia-reisys]

Note: a copy of the findings closed from this issue were exported via the process outlined here.