kubeaudit ☁️ 🔒 💪

kubeaudit is a command line tool to audit Kubernetes clusters for various different security concerns: run the container as a non-root user, use a read only root filesystem, drop scary capabilities, don't add new ones, don't run privileged, ... You get the gist of it and more on that later. Just know:

kubeaudit makes sure you deploy secure containers!

• Installation
• General instructions
• Autofix
• Audits
• Override Labels
• Audit Configuration
• Contribute!

Installation

Download a binary

Kubeaudit has official releases that are blessed and stable here: Official releases

DIY build

Master will have newer features than the stable releases. If you need a newer feature not yet included in a release you can do the following to get kubeaudit:

go get -v github.com/Shopify/kubeaudit
make
make install

Now you can just call kubeaudit with one of commands from here

Kubectl Plugin

Prerequisite: kubectl v1.12.0 or later

With kubectl v1.12.0 introducing easy pluggability of external functions, kubeaudit can be invoked as kubectl audit just by

• running make plugin and having $GOPATH/bin available in your path.

or

• renaming the binary to kubectl-audit and having it available in your path.

General instructions

kubeaudit has three different modes for its audits:

1. Cluster mode If kubeaudit detects that it's running in a container, kubeaudit cmd will attempt to audit the cluster it's running in.
2. Local config mode If kubeaudit is not running in a container, kubeaudit cmd will audit the resources specified by your local kubeconfig ($HOME/.kube/config) file. You can force kubeaudit to use a specific local config file with the switch -c/--kubeconfig /config/path
3. Manifest mode If you wish to audit a manifest file, use the command kubeaudit -f/--manifest /path/to/manifest.yml

kubeaudit supports two different output types:

1. just running kubeaudit will log human readable output
2. if run with -j/--json it will log output json formatted so that its output can be used by other programs easily

kubeaudit supports using manual audit configuration provided by the user, use the command kubeaudit -f/--manifest /path/to/manifest.yml -k/--auditConfig /path/to/config.yml For more details on audit config check out Audit Configuration.

kubeaudit has four different log levels INFO, WARN, ERROR controlled by -v/--verbose LEVEL and for those who counted and want to work on kubeaudit DEBUG

1. by default the debug level is set to ERROR and will log INFO, WARN and ERROR
2. if you only care about ERROR set it to ERROR
3. if you care about ERROR and WARN set it to WARN

But wait! Which version am I actually running? kubeaudit version will tell you.

I need help! Run kubeaudit help every audit has its own help so you can run kubeaudit help sc

Last but not least before we look at the audits: kubeaudit -a/--allPods audits against pods in all the phases (default Running Phase)

Autofix

As humans we are lazy and kubeaudit knows that so it comes with the functionality to autofix workload manifests. Point it at your workload manifests and it will automagically fix everything so that manifests are as secure as it gets.

kubeaudit autofix -f path/to/manifest.yml

The manifest might end up a little too secure for the work it is supposed to do. If that is the case check out labels to opt out of certain checks.

Audits

kubeaudit has multiple checks:

• Audit all
• Audit security context
  • Audit readOnlyRootFilesystem
  • Audit runAsNonRoot
  • Audit allowPrivilegeEscalation
  • Audit privileged
  • Audit capabilities
• Audit image
• Audit Service Accounts
• Audit network policies
• Audit resources
• Audit AppArmor
• Audit Seccomp
• Audit namespaces

Audit all

Runs all the above checks.

kubeaudit all
ERRO[0000] RunAsNonRoot is not set, which results in root user being allowed!
ERRO[0000] Default serviceAccount with token mounted. Please set automountServiceAccountToken to false
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay.
ERRO[0000] Capability not dropped     CapName=AUDIT_WRITE

Audit security contexts

The security context holds a couple of different security related configurations. For convenience, kubeaudit will always log the following information when it creates a log:

kubeaudit command
LOG[0000] KubeType=deployment Name=THEdeployment Namespace=deploymentNS

And for brevity, the information will not be shown in the commands below.

Currently, kubeaudit is able to check for the following fields in the security context:

Audit readOnlyRootFilesystem

kubeaudit will detect whether readOnlyRootFilesystem is either not set nil or explicitly set to false

kubeaudit rootfs
ERRO[0000] ReadOnlyRootFilesystem not set which results in a writable rootFS, please set to true
ERRO[0000] ReadOnlyRootFilesystem set to false, please set to true

Audit runAsNonRoot

kubeaudit will detect whether the container is to be run as root:

kubeaudit nonroot
ERRO[0000] RunAsNonRoot is set to false (root user allowed), please set to true!
ERRO[0000] RunAsNonRoot is not set, which results in root user being allowed!

Audit allowPrivilegeEscalation

kubeaudit will detect whether allowPrivilegeEscalation is either set to nil or explicitly set to false

kubeaudit allowpe
ERRO[0000] AllowPrivilegeEscalation set to true, please set to false
ERRO[0000] AllowPrivilegeEscalation not set which allows privilege escalation, please set to false

Audit privileged

kubeaudit will detect whether the container is to be run privileged:

kubeaudit priv
ERRO[0000] Privileged set to true! Please change it to false!

Since we want to make sure everything is intentionally configured correctly kubeaudit warns about privileged not being set:

kubeaudit priv
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay.

Audit capabilities

Docker comes with a couple of capabilities that shouldn't be needed and therefore should be dropped. kubeaudit will also complain about added capabilities.

If the capabilities field doesn't exist within the security context:

kubeaudit caps
ERRO[0000] Capabilities field not defined!

When capabilities were added:

kubeaudit caps
ERRO[0000] Capability added  CapName=NET_ADMIN

config/caps holds a list of capabilities that we recommend be dropped, change it if you want to keep some of the capabilities otherwise kubeaudit will complain about them not being dropped:

kubeaudit caps
ERRO[0000] Capability not dropped  CapName=AUDIT_WRITE

Audit container image tags

kubeaudit can check for image names and image tags:

1. If the image tag is incorrect an ERROR will issued

   kubeaudit image -i gcr.io/google_containers/echoserver:1.7
   ERRO[0000] Image tag was incorrect

2. If the image doesn't have a tag but an image of the name was found a WARNING will be created:

   kubeaudit image -i gcr.io/google_containers/echoserver:1.7
   WARN[0000] Image tag was missing

3. If the image was found with correct tag kubeaudit notifies with an INFO message:

   kubeaudit image -i gcr.io/google_containers/echoserver:1.7
   INFO[0000] Image tag was correct

Audit Service Accounts

It audits against the following scenarios:

1. A default serviceAccount mounted with a token:

   kubeaudit sat
   ERRO[0000] Default serviceAccount with token mounted. Please set AutomountServiceAccountToken to false

2. A deprecated service account:

   kubeaudit sat
   WARN[0000] serviceAccount is a deprecated alias for ServiceAccountName, use that one instead  DSA=DeprecatedServiceAccount

Audit network policies

It checks that every namespace should have a default deny network policy installed. See Kubernetes Network Policies for more information:

kubeaudit np
WARN[0000] Default allow mode on test/testing

Audit resources limits

It checks that every resource has a CPU and memory limit. See Kubernetes Resource Quotas for more information:

kubeaudit limits
WARN[0000] CPU limit not set, please set it!
WARN[0000] Memory limit not set, please set it!

With the --cpu and --memory parameters, kubeaudit can check the limits not to be exceeded.

kubeaudit limits --cpu 500m --memory 125Mi
WARN[0000] CPU limit exceeded, it is set to 1 but it must not exceed 500m. Please adjust it! !
WARN[0000] Memory limit exceeded, it is set to 512Mi but it must not exceed 125Mi. Please adjust it!

Audit AppArmor

It checks that AppArmor is enabled for all containers by making sure the following annotation exists on the pod. There must be an annotation for each container in the pod:

container.apparmor.security.beta.kubernetes.io/<container name>: <profile>

where profile can be "runtime/default" or start with "localhost/" to be considered valid.

If the AppArmor annotation is missing:

kubeaudit apparmor
ERRO[0000] AppArmor annotation missing. Container=myContainer KubeType=pod Name=myPod Namespace=myNamespace

When AppArmor annotations are misconfigured:

kubeaudit apparmor
ERRO[0000] AppArmor disabled. Annotation=container.apparmor.security.beta.kubernetes.io/myContainer
  Container=myContainer KubeType=pod Name=myPod Namespace=myNamespace Reason=badval

Audit Seccomp

It checks that Seccomp is enabled for all containers by making sure one or both of the following annotations exists on the pod. If no pod annotation is used, then there must be an annotation for each container. Container annotations override the pod annotation:

# pod annotation
seccomp.security.alpha.kubernetes.io/pod: <profile>

# container annotation
container.seccomp.security.alpha.kubernetes.io/<container name>: <profile>

where profile can be "runtime/default" or start with "localhost/" to be considered valid. "docker/default" is deprecated and will show a warning. It should be replaced with "runtime/default".

If the Seccomp annotation is missing:

kubeaudit seccomp
ERRO[0000] Seccomp annotation missing. Container=myContainer KubeType=pod Name=myPod Namespace=myNamespace

When Seccomp annotations are misconfigured for a container:

kubeaudit seccomp
ERRO[0000] Seccomp disabled for container. Annotation=container.seccomp.security.alpha.kubernetes.io/myContainer
  Container=myContainer KubeType=pod Name=myPod Namespace=myNamespace Reason=badval

When Seccomp annotations are misconfigured for a pod:

kubeaudit seccomp
ERRO[0000] Seccomp disabled for pod. Annotation=seccomp.security.alpha.kubernetes.io/pod Container= KubeType=pod
  Name=myPod Namespace=myNamespace Reason=unconfined

Audit namespaces

kubeaudit will detect whether hostNetwork,hostIPC or hostPID is either set to true in podSpec for Pod workloads

kubeaudit namespaces
ERRO[0000] hostNetwork is set to true  in podSpec, please set to false!
ERRO[0000] hostIPC is set to true  in podSpec, please set to false!
ERRO[0000] hostPID is set to true  in podSpec, please set to false!

Override Labels

Override labels give you the ability to have kubeaudit allow certain audits to fail. For example, if you want kubeaudit to ignore the fact that AllowPrivilegeEscalation was set to true, you can add the following label:

spec:
  template:
    metadata:
      labels:
        apps: YourAppNameHere
        container.audit.kubernetes.io/<container-name>/allow-privilege-escalation: "YourReasonForOverrideHere"

Any label with a non-nil reason string will prevent kubeaudit from throwing the corresponding error and issue a warning instead. Reasons matching "true" (not case sensitive) will be displayed as Unspecified.

kubeaudit can skip certain audits by applying override labels to containers. If you want skip an audit for a specific container inside a pod, you can add an container override label. For example, if you use kubeaudit to ignore allow-run-as-root check for container "MyContainer1", you can add the following label:

spec:
  template:
    metadata:
      labels:
        apps: YourAppNameHere
        container.audit.kubernetes.io/MyContainer1/allow-run-as-root: "YourReasonForOverrideHere"

Similarly, you can have kubeaudit to skip a specific audit for all containers inside the pod by adding a pod override label. For example, if you use kubeaudit to ignore allow-run-as-root check for all containers inside the pod, you can add the following label:

spec:
  template:
    metadata:
      labels:
        apps: YourAppNameHere
        audit.kubernetes.io/pod/allow-run-as-root: "YourReasonForOverrideHere"

kubeaudit can also skip a specific audit for all network policies associated with a namespace resource by adding a namespace override label. For example, if you use kubeaudit to ignore the allow-non-default-deny-egress-network-policy check for the namespace namespaceName1 you can add the following label to the namespace:

metadata:
  name: namespaceName1
  labels:
    audit.kubernetes.io/namespaceName1/allow-non-default-deny-egress-network-policy: "YourReasonForOverrideHere"

kubeaudit supports many labels on pod, namespace or container level:

• audit.kubernetes.io/pod/allow-privilege-escalation
• container.audit.kubernetes.io/<container-name>/allow-privilege-escalation
• audit.kubernetes.io/pod/allow-privileged
• container.audit.kubernetes.io/<container-name>/allow-privileged
• audit.kubernetes.io/pod/allow-capability
• container.audit.kubernetes.io/<container-name>/allow-capability
• audit.kubernetes.io/pod/allow-run-as-root
• container.audit.kubernetes.io/<container-name>/allow-run-as-root
• audit.kubernetes.io/pod/allow-automount-service-account-token
• audit.kubernetes.io/pod/allow-read-only-root-filesystem-false
• container.audit.kubernetes.io/<container-name>/allow-read-only-root-filesystem-false
• audit.kubernetes.io/<namespace-name>/allow-non-default-deny-egress-network-policy
• audit.kubernetes.io/<namespace-name>/allow-non-default-deny-ingress-network-policy
• audit.kubernetes.io/pod/allow-namespace-host-network
• audit.kubernetes.io/pod/allow-namespace-host-IPC
• audit.kubernetes.io/pod/allow-namespace-host-PID

container.audit.kubernetes.io/<container-name>/allow-privilege-escalation

Allow allowPrivilegeEscalation to be set to true to a specific container.

audit.kubernetes.io/pod/allow-privilege-escalation

Allows allowPrivilegeEscalation to be set to true to all the containers in a pod.

kubeaudit.allow.privilegeEscalation: "Superuser privileges needed"

WARN[0000] Allowed setting AllowPrivilegeEscalation to true  Reason="Superuser privileges needed"

container.audit.kubernetes.io/<container-name>/allow-privileged

Allow privileged to be set to true to a specific container.

audit.kubernetes.io/pod/allow-privileged

Allows privileged to be set to true to all the containers in a pod.

kubeaudit.allow.privileged: "Privileged execution required"

WARN[0000] Allowed setting privileged to true                Reason="Privileged execution required"

container.audit.kubernetes.io/<container-name>/allow-capability

Allows adding a capability or keeping one that would otherwise be dropped to a specific container.

audit.kubernetes.io/pod/allow-capability

Allows adding a capability or keeping one that would otherwise be dropped to all the containers in a pod.

kubeaudit.allow.capability.chown: "true"

WARN[0000] Capability allowed                                CapName=CHOWN Reason=Unspecified

container.audit.kubernetes.io/<container-name>/allow-run-as-root

Allows setting runAsNonRoot to false to a specific container.

audit.kubernetes.io/pod/allow-run-as-root

Allows setting runAsNonRoot to false to all the containers in a pod.

kubeaudit.allow.runAsRoot: "Root privileges needed"

WARN[0000] Allowed setting RunAsNonRoot to false             Reason="Root privileges needed"

audit.kubernetes.io/pod/allow-automount-service-account-token

Allows setting automountServiceAccountToken to true to a pod.

kubeaudit.allow.autmountServiceAccountToken: "True"

WARN[0000] Allowed setting automountServiceAccountToken to true  Reason=Unspecified

container.audit.kubernetes.io/<container-name>/allow-read-only-root-filesystem-false

Allows setting readOnlyRootFilesystem to false to a specific container.

audit.kubernetes.io/pod/allow-read-only-root-filesystem-false

Allows setting readOnlyRootFilesystem to false to all containers in a pod.

kubeaudit.allow.readOnlyRootFilesystemFalse: "Write permissions needed"

WARN[0000] Allowed setting readOnlyRootFilesystem to false Reason="Write permissions needed"

audit.kubernetes.io/<namespace-name>/allow-non-default-deny-egress-network-policy

Allows absense of default-deny egress network policy for that specific namespace.

audit.kubernetes.io/<namespace-name>/allow-non-default-deny-ingress-network-policy

Allows absense of default-deny ingress network policy for that specific namespace.

audit.kubernetes.io/default/allow-non-default-deny-egress-network-policy: "Egress is allowed"

WARN[0000] Allowed Namespace without a default deny egress NetworkPolicy  KubeType=namespace Name=default Reason="Egress is allowed"

audit.kubernetes.io/pod/allow-namespace-host-network

audit.kubernetes.io/pod/allow-namespace-host-network: "hostNetwork is allowed"

WARN[0000] Allowed setting hostNetwork to true           KubeType=pod Name=Pod Namespace=PodNamespace Reason="hostNetwork is allowed"

audit.kubernetes.io/pod/allow-namespace-host-IPC

audit.kubernetes.io/pod/allow-namespace-host-IPC: "hostIPC is allowed"

WARN[0000] Allowed setting hostIPC to true               KubeType=pod Name=Pod Namespace=PodNamespace Reason="hostIPC is allowed"

audit.kubernetes.io/pod/allow-namespace-host-PID

audit.kubernetes.io/pod/allow-namespace-host-PID: "hostPID is allowed"

WARN[0000] Allowed setting hostPID to true               KubeType=pod Name=Pod Namespace=PodNamespace Reason="hostPID is allowed"

Drop capabilities list

Allows configuring the audit against drop capabilities. Sane defaults are as follows:

# SANE DEFAULTS:
capabilitiesToBeDropped:
  # https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
  - SETPCAP #Modify process capabilities.
  - MKNOD #Create special files using mknod(2).
  - AUDIT_WRITE #Write records to kernel auditing log.
  - CHOWN #Make arbitrary changes to file UIDs and GIDs (see chown(2)).
  - NET_RAW #Use RAW and PACKET sockets.
  - DAC_OVERRIDE #Bypass file read, write, and execute permission checks.
  - FOWNER #Bypass permission checks on operations that normally require the file system UID of the process to match the UID of the file.
  - FSETID #Don’t clear set-user-ID and set-group-ID permission bits when a file is modified.
  - KILL #Bypass permission checks for sending signals.
  - SETGID #Make arbitrary manipulations of process GIDs and supplementary GID list.
  - SETUID #Make arbitrary manipulations of process UIDs.
  - NET_BIND_SERVICE #Bind a socket to internet domain privileged ports (port numbers less than 1024).
  - SYS_CHROOT #Use chroot(2), change root directory.
  - SETFCAP #Set file capabilities.

This can be overridden by using -k flag and providing your own defaults in the yaml format as shown below.

Audit Configuration

Allows configuring your own audit settings for kubeaudit. By default following configuration is used:

apiVersion: v1
kind: kubeauditConfig
audit: true  # Set to false if you want kubeaudit to not audit your k8s manifests
spec:
  capabilities: # List of all supported capabilities
    NET_ADMIN: drop         # Set to `keep` to keep capability
    SETPCAP: drop           # Set to `keep` to keep capability
    MKNOD: drop             # Set to `keep` to keep capability
    AUDIT_WRITE: drop       # Set to `keep` to keep capability
    CHOWN: drop             # Set to `keep` to keep capability
    NET_RAW: drop           # Set to `keep` to keep capability
    DAC_OVERRIDE: drop      # Set to `keep` to keep capability
    FOWNER: drop            # Set to `keep` to keep capability
    FSETID: drop            # Set to `keep` to keep capability
    KILL: drop              # Set to `keep` to keep capability
    SETGID: drop            # Set to `keep` to keep capability
    SETUID: drop            # Set to `keep` to keep capability
    NET_BIND_SERVICE: drop  # Set to `keep` to keep capability
    SYS_CHROOT: drop        # Set to `keep` to keep capability
    SETFCAP: drop           # Set to `keep` to keep capability
  overrides: # List of all supported overrides
    privilege-escalation: deny                      # Set to `allow` to skip auditing potential vulnerability
    privileged: deny                                # Set to `allow` to skip auditing potential vulnerability
    run-as-root: deny                               # Set to `allow` to skip auditing potential vulnerability
    automount-service-account-token: deny           # Set to `allow` to skip auditing potential vulnerability
    read-only-root-filesystem-false: deny           # Set to `allow` to skip auditing potential vulnerability
    non-default-deny-ingress-network-policy: deny   # Set to `allow` to skip auditing potential vulnerability
    non-default-deny-egress-network-policy: deny    # Set to `allow` to skip auditing potential vulnerability
    namespace-host-network: deny                    # Set to `allow` to skip auditing potential vulnerability
    namespace-host-IPC: deny                        # Set to `allow` to skip auditing potential vulnerability
    namespace-host-PID: deny                        # Set to `allow` to skip auditing potential vulnerability

Contributing

If you'd like to fix a bug, contribute a feature or just correct a typo, please feel free to do so as long as you follow our Code of Conduct.

1. Create your own fork!
2. Get the source: go get github.com/Shopify/kubeaudit
3. Go to the source: cd $GOPATH/src/github.com/Shopify/kubeaudit
4. Add your forked repo as a fork: git remote add fork https://github.com/you-are-awesome/kubeaudit
5. Create your feature branch: git checkout -b awesome-new-feature
6. Run the tests to see everything is working as expected: make test
7. Commit your changes: git commit -am 'Adds awesome feature'
8. Push to the branch: git push fork
9. Submit a PR (All PR must be labeled with 🐛 (Bug fix), ✨ (New feature), 📖 (Documentation update), or ⚠️ (Breaking changes) )
10. ???
11. Profit