kees - Kubernetes External Environment Secrets

https://hub.docker.com/r/bsycorp/kees/

In our current draft of Kubernetes Secret Management, we have three components: Init Container, Creator and Exporter.

The functionalities of each component are outlined as following:

Creator:

• Watches for relevant pods
• Read secret annotations
• Create secrets that do not exist yet

Init Container:

• Read annotations of requested secrets and resources
• Retrieve secret from Parameter Store
• Write secrets to EmptyDir for app container to consume

Exporter:

• Is called during build/packaging stage, not a runtime component
• Generates a terraform IAM policy that matches the secret/resource annotations in the manifest to maintain a single source of truth
• IAM policy changes will be reviewed as part of code review (in the manifest) and reviewed again during terraform plan

How does Init Container work?

Init container are used to initialise a pod before an application container runs. In our case, it will read the secrets from Parameter Store, and make it available to the app container before it spins up.

Step 1:

Before an application is run, the build/package step adds an init container to all relevant pods as part of the gradle build task.

spec:
  containers:
  ...
  initContainers:
  - name: init-container
    image: bsycorp/kees/init:latest
    volumeMounts:
    - mountPath: /secret
      name: secret-volume
    - mountPath: /annotations
      name: annotations

During the package stage we also generate a terraform IAM policy. The application needs to be given some permissions to the secret storage (SSM / Parameter Store) this is done via the Exporter. We want the manifest annotations to be the source of truth for defining what secrets an application needs access too, but this needs to be enforced by IAM, so the exporter is run at build/package time to generate a policy that ensures the two are aligned.

Step 2:

Then it will use the downwardAPI to read annotations from the pod spec.

volumes:
- name: secret-volume
  emptyDir: {}
- name: annotations
  downwardAPI:
    items:
      - path: "annotations"
        fieldRef:
          fieldPath: metadata.annotations

This will load the annotations into a file for init container to consume.

spec:
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: cluster-app-role
        secret.bsycorp.com/signingKey.v1_public: "kind=DYNAMIC,type=RSA,size=2048,foo=bar"
        secret.bsycorp.com/signingKey.v1_private: "kind=DYNAMIC,type=RSA,size=2048,foo=bar"
        secret.bsycorp.com/db.password: "kind=REFERENCE,type=PASSWORD"

The above spec will be read into /annotations/annotations as following:

secret.bsycorp.com/signingKey.v1_public="kind=DYNAMIC,type=RSA,size=2048,foo=bar"
secret.bsycorp.com/signingKey.v1_private="kind=DYNAMIC,type=RSA,size=2048,foo=bar"
secret.bsycorp.com/db.password="kind=REFERENCE,type=PASSWORD"

Step 3:

The init image will then

• Process the annotations and extract secret keys
• Query Parameter Store to retrieve the secret values (as restricted by its IAM role)
• Save the secret key value pairs to a file in EmptyDir

An example output /secret/secret would be as following:

db.password=fakepassword
signingKey.v1_private=privatekeydummyvalue
signingKey.v1_public=publickeydummyvalue

Step 4:

When the init container completes, app container will spin up, and will be able to consume the secrets from /secret/secret in the EmptyDir volume.

Reference

• Init containers: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/