Merge pull request kata-containers#7146 from BbolroC/ibm-se-howto-doc

docs: provide a guide for how to use IBM Secure Execution
Amulyam24 · Jan 23, 2024 · f429068 · f429068
2 parents 0f67a26 + 25ecca9
commit f429068
Show file tree

Hide file tree

Showing 2 changed files with 390 additions and 0 deletions.
diff --git a/docs/how-to/README.md b/docs/how-to/README.md
@@ -48,6 +48,7 @@
 - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md)
 - [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md)
 - [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md)
+- [How to run Kata Containers with IBM Secure Execution](how-to-run-kata-containers-with-SE-VMs.md)
 - [How to use EROFS to build rootfs in Kata Containers](how-to-use-erofs-build-rootfs.md)
 - [How to run Kata Containers with kinds of Block Volumes](how-to-run-kata-containers-with-kinds-of-Block-Volumes.md)
 - [How to use the Kata Agent Policy](how-to-use-the-kata-agent-policy.md)
diff --git a/docs/how-to/how-to-run-kata-containers-with-SE-VMs.md b/docs/how-to/how-to-run-kata-containers-with-SE-VMs.md
@@ -0,0 +1,389 @@
+# Kata Containers with IBM Secure Execution VMs
+
+This document assumes a trusted environment with a functioning kata container, as per the
+[developer guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md).
+The term "trusted" implies that the system is authorized, authenticated and attested to use your artifacts and secrets safely.
+
+
+## Tested Environment Specifications
+
+1. Machine: IBM z16 LPAR
+2. OS: Ubuntu 22.04.1 LTS
+3. CPU: 16 vCPU
+4. Memory: 16G
+
+## Manual configuration
+
+### Prerequisite
+
+1. Host capable of Secure Execution
+
+To take advantage of the IBM Secure Execution capability, the host machine on which
+you intend to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III
+(or a newer model). In addition to the hardware requirement, you need to verify the CPU
+facility and kernel configuration, as outlined below:
+
+```
+$ # To check the protected virtualization support from kernel
+$ cat /sys/firmware/uv/prot_virt_host
+1
+$ # To check if an ultravisor reserves memory for the current boot
+$ sudo dmesg | grep -i ultravisor
+[    0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage
+$ # To check a facility bit for Secure Execution
+$ cat /proc/cpuinfo | grep 158
+facilities      :  ... numbers ... 158 ... numbers ...
+```
+
+If any of the results are not identifiable, please reach out to the responsible cloud
+provider to enable the Secure Execution capability. Alternatively, if you possess
+administrative privileges and the facility bit is set, you can enable the Secure Execution
+capability by adding `prot_virt=1` to the kernel parameters and performing a system reboot like:
+
+```
+$ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf
+$ sudo zipl -V
+$ sudo systemctl reboot
+```
+
+Please note that the method of enabling the Secure Execution capability may vary among Linux distributions.
+
+2. Artifacts from Kata Containers
+
+A secure image is constructed using the following artifacts
+
+- A raw kernel
+- An initial RAM disk
+
+The most straightforward approach to obtain these artifacts is by reusing kata-containers:
+
+```
+$ export PATH="$PATH:/opt/kata/bin"
+$ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path'))
+config-6.1.62-121
+kata-containers.img
+kata-containers-initrd.img
+kata-ubuntu-20.04.initrd
+kata-ubuntu-latest.image
+vmlinux-6.1.62-121
+vmlinux.container
+vmlinuz-6.1.62-121
+vmlinuz.container
+```
+
+The output indicates the deployment of the kernel (`vmlinux-6.1.62-121`, though the version
+may vary at the time of testing), rootfs-image (`kata-ubuntu-latest.image`), and rootfs-initrd (`kata-ubuntu-20.04.initrd`).
+In this scenario, the available kernel and initrd can be utilized for a secure image.
+However, if any of these components are absent, they must be built from the
+[project source](https://github.com/kata-containers/kata-containers) as follows:
+
+```
+$ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ sudo -E PATH=$PATH make kernel-tarball
+$ sudo -E PATH=$PATH make rootfs-initrd-tarball
+$ tar -tf build/kata-static-kernel.tar.xz | grep vmlinuz
+./opt/kata/share/kata-containers/vmlinuz.container
+./opt/kata/share/kata-containers/vmlinuz-6.1.62-121
+$ tar -tf build/kata-static-rootfs-initrd.tar.xz | grep initrd
+./opt/kata/share/kata-containers/kata-containers-initrd.img
+./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ mkdir artifacts
+$ tar -xvf build/kata-static-kernel.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-6.1.62-121
+$ tar -xvf build/kata-static-rootfs-initrd.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ ls artifacts/opt/kata/share/kata-containers/
+kata-ubuntu-20.04.initrd  vmlinuz-6.1.62-121
+```
+
+3. Secure Image Generation Tool
+
+`genprotimg` is a utility designed to generate an IBM Secure Execution image. It can be
+installed either from the package manager of a distribution or from the source code.
+The tool is included in the `s390-tools` package. Please ensure that you have a version
+of the tool equal to or greater than `2.17.0`. If not, you will need to specify
+an additional argument, `--x-pcf '0xe0'`, when running the command.
+Here is an example of a native build from the source:
+
+```
+$ sudo apt-get install gcc libglib2.0-dev libssl-dev libcurl4-openssl-dev
+$ tool_version=v2.25.0
+$ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git
+$ pushd s390-tools/genprotimg && make && sudo make install && popd
+$ rm -rf s390-tools
+```
+
+4. Host Key Document
+
+A host key document is a public key employed for encrypting a secure image, which is
+subsequently decrypted using a corresponding private key during the VM bootstrap process.
+You can obtain the host key document either through IBM's designated
+[Resource Link](http://www.ibm.com/servers/resourcelink) or by requesting it from the
+cloud provider responsible for the IBM Z and LinuxONE instances where your workloads are intended to run.
+
+To ensure security, it is essential to verify the authenticity and integrity of the host key document
+belonging to an authentic IBM machine. To achieve this, please additionally obtain the following
+certificates from the Resource Link:
+
+- IBM Z signing key certificate
+- `DigiCert` intermediate CA certificate
+
+These files will be used for verification during secure image construction in the next section.
+
+### Build a Secure Image
+
+
+Assuming you have placed a host key document at `$HOME/host-key-document`:
+
+- Host key document as `HKD-0000-0000000.crt`
+
+and two certificates at `$HOME/certificates`:
+
+- `DigiCert` intermediate CA certificate as `DigiCertCA.crt`
+- IBM Z signing-key certificate as `ibm-z-host-key-signing.crt`
+
+you can construct a secure image using the following procedure:
+
+```
+$ # Change a directory to the project root
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
+$ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-6.1.62-121
+$ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile
+$ genprotimg --host-key-document=${host_key_document} \
+--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
+--parmfile=parmfile --no-verify
+WARNING: host-key document verification is disabled. Your workload is not secured.
+$ file kata-containers-se.img
+kata-containers-se.img: data
+$ sudo cp kata-containers-se.img /opt/kata/share/kata-containers/
+```
+
+It is important to note that the `--no-verify` parameter, which allows skipping
+the key verification process, is intended to be used solely in a development or
+testing environment.
+In production, the image construction should incorporate the verification
+in the following manner:
+
+```
+$ cacert=$HOME/certificates/DigiCertCA.crt
+$ signcert=$HOME/certificates/ibm-z-host-key-signing.crt
+$ genprotimg --host-key-document=${host_key_document} \
+--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
+--cert=${cacert} --cert=${signcert} --parmfile=parmfile
+```
+
+The steps with no verification, including the dependencies for the kernel and initrd,
+can be easily accomplished by issuing the following make target:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ mkdir hkd_dir && cp $host_key_document hkd_dir
+$ sudo -E PATH=$PATH HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \
+make boot-image-se-tarball
+$ ls build/kata-static-boot-image-se.tar.xz
+build/kata-static-boot-image-se.tar.xz
+```
+
+`SE_KERNEL_PARAMS` could be used to add any extra kernel parameters. If no additional kernel configuration is required, this can be omitted.
+
+In production, you could build an image by running the same command, but with two
+additional environment variables for key verification:
+
+```
+$ export SIGNING_KEY_CERT_PATH=$HOME/certificates/ibm-z-host-key-signing.crt
+$ export INTERMEDIATE_CA_CERT_PATH=$HOME/certificates/DigiCertCA.crt
+```
+
+To build an image on the `x86_64` platform, set the following environment variables together with the variables above before `make boot-image-se-tarball`:
+
+```
+CROSS_BUILD=true TARGET_ARCH=s390x ARCH=s390x
+```
+
+### Adjust the configuration
+
+There still remains an opportunity to fine-tune the configuration file:
+
+```
+$ runtime_config_path=$(kata-runtime kata-env --json | jq -r '.Runtime.Config.Path')
+$ cp ${runtime_config_path} ${runtime_config_path}.old
+$ # Make the following adjustment to the original config file
+$ diff ${runtime_config_path}.old ${runtime_config_path}
+16,17c16,17
+< kernel = "/opt/kata/share/kata-containers/vmlinux.container"
+< image = "/opt/kata/share/kata-containers/kata-containers.img"
+---
+> kernel = "/opt/kata/share/kata-containers/kata-containers-se.img"
+> # image = "/opt/kata/share/kata-containers/kata-containers.img"
+41c41
+< # confidential_guest = true
+---
+> confidential_guest = true
+544c544
+< dial_timeout = 45
+---
+> dial_timeout = 90
+679c679
+< #service_offload = true
+---
+> service_offload = true
+```
+
+### Verification
+
+To verify the successful decryption and loading of the secure image within a test VM,
+please refer to the following commands:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path')
+$ secure_kernel=kata-containers-se.img
+$ sudo $hypervisor_command -machine confidential-guest-support=pv0 \
+-object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \
+--nographic --nodefaults --kernel "${secure_kernel}"
+[    0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023                                                                     [    0.110279] setup: Linux is running under KVM in 64-bit mode
+
+... log skipped ...
+
+[    1.467228] Run /init as init process
+{"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent
+","version":"0.1.0"}
+
+... log skipped ...
+
+$ # Press ctrl + a + x to exit
+```
+
+If the hypervisor log does not indicate any errors, it provides assurance that the image
+has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime
+will function properly.
+
+Let us proceed with the final verification by running a test container in a Kubernetes
+cluster. Please make user you have a running cluster like:
+
+```
+$ kubectl get node
+NAME           STATUS   ROLES                  AGE     VERSION
+test-cluster   Ready    control-plane,master   7m28s   v1.23.1
+```
+
+Please execute the following command to run a container:
+
+```
+$ cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx-kata
+spec:
+  runtimeClassName: kata-qemu
+  containers:
+  - name: nginx
+    image: nginx
+EOF
+pod/nginx-kata created
+$ kubectl get po
+NAME         READY   STATUS    RESTARTS   AGE
+nginx-kata   1/1     Running   0          29s
+$ kubectl get po -oyaml | grep "runtimeClassName:"
+    runtimeClassName: kata-qemu
+$ # Please make sure if confidential-guest-support is set and a secure image is used
+$ $ ps -ef | grep qemu | grep -v grep
+root       76972   76959  0 13:40 ?        00:00:02 /opt/kata/bin/qemu-system-s390x
+... qemu arguments ...
+-machine s390-ccw-virtio,accel=kvm,confidential-guest-support=pv0
+... qemu arguments ...
+-kernel /opt/kata/share/kata-containers/kata-containers-se.img
+... qemu arguments ...
+```
+
+Finally, an operational kata container with IBM Secure Execution is now running.
+
+## Using Kata-Deploy with Confidential Containers Operator
+
+It is reasonable to expect that the manual steps mentioned above can be easily executed.
+Typically, you can use
+[kata-deploy](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md)
+to install Kata Containers on a Kubernetes cluster. However, when leveraging IBM Secure Execution,
+you need to employ the confidential container's
+[operator](https://github.com/confidential-containers/operator).
+During this process, a `kata-deploy` container image serves as a payload image in a custom
+resource `ccruntime` for confidential containers, enabling the operator to install Kata
+binary artifacts such as kernel, shim-v2, and more.
+
+This section will explain how to build a payload image
+(i.e., `kata-deploy`) for confidential containers. For the remaining instructions,
+please refer to the
+[documentation](https://github.com/confidential-containers/operator/blob/main/docs/how-to/INSTALL-CC-WITH-IBM-SE.md)
+for confidential containers.
+
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
+$ mkdir hkd_dir && cp $host_key_document hkd_dir
+$ # kernel and rootfs-initrd are built automactially by the command below
+$ sudo -E PATH=$PATH HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \
+make boot-image-se-tarball
+$ sudo -E PATH=$PATH make qemu-tarball
+$ sudo -E PATH=$PATH make virtiofsd-tarball
+$ # shim-v2 should be built after kernel due to dependency
+$ sudo -E PATH=$PATH make shim-v2-tarball
+$ mkdir kata-artifacts
+$ build_dir=$(readlink -f build)
+$ cp -r $build_dir/*.tar.xz kata-artifacts
+$ ls -1 kata-artifacts
+kata-static-boot-image-se.tar.xz
+kata-static-kernel.tar.xz
+kata-static-qemu.tar.xz
+kata-static-rootfs-initrd.tar.xz
+kata-static-shim-v2.tar.xz
+kata-static-virtiofsd.tar.xz
+$ ./tools/packaging/kata-deploy/local-build/kata-deploy-merge-builds.sh kata-artifacts
+```
+
+In production, the environment variables `SIGNING_KEY_CERT_PATH` and
+`INTERMEDIATE_CA_CERT_PATH` should be exported like the manual configuration.
+If a rootfs-image is required for other available runtime classes (e.g. `kata` and `kata-qemu`)
+without the Secure Execution functionality, please run the following command
+before running `kata-deploy-merge-builds.sh`:
+
+```
+$ sudo -E PATH=$PATH make rootfs-image-tarball
+```
+
+At this point, you should have an archive file named `kata-static.tar.xz` at the project root,
+which will be used to build a payload image. If you are using a local container registry at
+`localhost:5000`, proceed with the following:
+
+```
+$ docker run -d -p 5000:5000 --name local-registry registry:2.8.1
+```
+
+Build and push a payload image with the name `localhost:5000/build-kata-deploy` and the tag
+`latest` using the following:
+
+```
+$ sudo -E PATH=$PATH ./tools/packaging/kata-deploy/local-build/kata-deploy-build-and-upload-payload.sh kata-static.tar.xz localhost:5000/build-kata-deploy latest
+... logs ...
+Pushing the image localhost:5000/build-kata-deploy:latest to the registry
+The push refers to repository [localhost:5000/build-kata-deploy]
+76c6644d9790: Layer already exists
+2413aff53bb1: Layer already exists
+91462f44bb06: Layer already exists
+2ad49fac591a: Layer already exists
+5c75aa64ef7a: Layer already exists
+test: digest: sha256:25825c7a4352f75403ee59a683eb122d5518e8ed6a244aacd869e41e2cafd385 size: 1369
+```
+
+## Considerations for CI
+
+If you intend to integrate the aforementioned procedure with a CI system,
+configure the following setup for an environment variable.
+The setup helps speed up CI jobs by caching container images used during the build:
+
+```
+$ export BUILDER_REGISTRY=$YOUR_PRIVATE_REGISTRY_FOR_CI
+$ export PUSH_TO_REGISTRY=yes
+```