From 3d9367526848c585223e6a6f0373931e55a922e0 Mon Sep 17 00:00:00 2001 From: AndreyKoltsov1997 Date: Wed, 11 Oct 2023 15:40:43 +0200 Subject: [PATCH 1/4] * Implement custom Dockerfile for Podman-in-Docker support, including its adjustment to rootless and rootful modes. * Include configuration for container storage. * Add extensive documentation on building and executing Agent container with Podman. --- .../linux/Agent/Ubuntu/Ubuntu-sudo.Dockerfile | 2 +- custom/README.md | 104 +++++++++++++++++- .../configs/podman/rootful.containers.conf | 12 ++ .../configs/podman/rootless.containers.conf | 5 + dockerhub/teamcity-agent/README.md | 7 +- 5 files changed, 122 insertions(+), 8 deletions(-) create mode 100644 custom/linux/agent/configs/podman/rootful.containers.conf create mode 100644 custom/linux/agent/configs/podman/rootless.containers.conf diff --git a/configs/linux/Agent/Ubuntu/Ubuntu-sudo.Dockerfile b/configs/linux/Agent/Ubuntu/Ubuntu-sudo.Dockerfile index ecdec8ff3..bff8fd572 100644 --- a/configs/linux/Agent/Ubuntu/Ubuntu-sudo.Dockerfile +++ b/configs/linux/Agent/Ubuntu/Ubuntu-sudo.Dockerfile @@ -30,7 +30,7 @@ RUN apt-get update && \ apt-get install -y --no-install-recommends sudo && \ # https://github.com/goodwithtech/dockle/blob/master/CHECKPOINT.md#dkl-di-0005 apt-get clean && rm -rf /var/lib/apt/lists/* && \ - echo 'buildagent ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \ + echo 'buildagent ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \ rm -rf /var/lib/apt/lists/* USER buildagent diff --git a/custom/README.md b/custom/README.md index 5539c2569..2cc8dcd3a 100644 --- a/custom/README.md +++ b/custom/README.md @@ -1,9 +1,21 @@ # Custom TeamCity Agent Images - The folder includes Dockerfiles that you can utilize to create custom TeamCity Agent images. -# 1. .NET SDK +# Content + +* [1. .NET SDK](#1-net-sdk) + * [1.1. Building Images](#11-building-images) +* [1.2 .NET End of Support Dates](#12-net-end-of-support-dates) +* [2. Podman](#2-podman) + * [2.1 Building Images](#21-building-images) + * [2.2 Execution](#22-execution) + * [2.2.1 Rootless Podman in Docker (no '--privileged')](#221-rootless-podman-in-docker-no---privileged) + * [2.2.2 Rootful Podman in Docker ('--privileged')](#222-rootful-podman-in-docker---privileged) + * [2.3 Podman - troubleshooting](#23-podman---troubleshooting) + * [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman) + +# 1. .NET SDK | OS | Arch | .NET SDK | Dockerfile | `dotnetSdkVersion` | `dotnetSdkChecksum` | |---------|---------|-------------------------------------------------------------------------|------------------------------------------------------------|--------------------|------------------------------------------------------------------------------------------------------------------------------------| | Linux | `AMD64` | [.NET Core 3.1](https://dotnet.microsoft.com/en-us/download/dotnet/3.1) | [link](linux/agent/amd/custom.dotnet.sdk.amd.Dockerfile) | `3.1.426` | `6c3f9541557feb5d5b93f5c10b28264878948e8540f2b8bb7fb966c32bd38191e6b310dcb5f87a4a8f7c67a7046fa932cde3cce9dc8341c1365ae6c9fcc481ec` | @@ -17,14 +29,13 @@ The folder includes Dockerfiles that you can utilize to create custom TeamCity A | Windows | `AMD64` | [.NET 7.0](https://dotnet.microsoft.com/en-us/download/dotnet/7.0) | [link](windows/agent/custom.dotnet.sdk.win.amd.Dockerfile) | `7.0.401` | `02a4ecc05d0b9dfa0c9e32f8a3d288f329e7338b2430fcbc1276ae356f9d8e14920f91382f3f141842bf1e6e6cd331e532b301edc71c26de9d9e5ad2371afbe0` | -The .NET SDK version bundled within TeamCity Docker Images is aligned with [Microsoft's Long Term Support (LTS) release](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) -at the moment of a TeamCity release. Since it is sometimes necessary to use STS (which can be newer than LTS) or +The .NET SDK version bundled within TeamCity Docker Images is aligned with [Microsoft's Long Term Support (LTS) release](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) +at the moment of a TeamCity release. Since it is sometimes necessary to use STS (which can be newer than LTS) or older versions, we provide examples of building images with custom .NET SDK versions inside. The folder contains Dockerfiles that simplify this process, allowing you to easily replace any .NET SDK version within the image with a pre-defined one. These Dockerfiles can also be used as templates for installing any custom .NET version. ## 1.1. Building Images - The table above references multiple versions of .NET framework. To build a custom image, specify the required SDK version (`dotnetSdkVersion`) and a checksum for it (`dotnetSdkChecksum`): ``` docker build \ @@ -64,7 +75,6 @@ docker run teamcity-agent:windows-custom-dotnet-7 dotnet --version ``` # 1.2 .NET End of Support Dates - In the [.NET and .NET Core Support Policy](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) article, Microsoft states the following end of support dates for .NET: * **.NET Core 3.1** - December 13th, 2022; * **.NET 5.0** - May 10th, 2022; @@ -72,3 +82,85 @@ In the [.NET and .NET Core Support Policy](https://dotnet.microsoft.com/en-us/pl * **.NET 7.0** (STS) - May 14, 2024; We strongly encourage replacing your current .NET versions to newer ones if the support for your current version is nearing its end. + +# 2. Podman +This section provides instructions for building and executing TeamCity Docker Images with Podman, suitable for use in both rootless and rootful modes. + +Please, note that the latest version of Podman for Ubuntu 20.04 is `Podman 3.4.2`, as indicated by the [libcontainers](https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/amd64/). + +In order to use Podman as a default container runtime in TeamCity, please, set `teamcity.container.wrapper.use.podman=true`. + +## 2.1 Building Images +Rootless: +``` +$ docker build \ + --build-arg teamCityAgentImage=jetbrains/teamcity-agent:2023.05.4 \ + -f linux/agent/amd/podman.amd.Dockerfile \ + -t jebrains/teamcity-agent:2023.05.4-podman . +``` + +Rootful (based on `sudo` image): +``` +$ docker build \ +--build-arg teamCityAgentImage=jetbrains/teamcity-agent:2023.05.4-linux-sudo\ +-f linux/agent/amd/podman.amd.Dockerfile \ +-t jebrains/teamcity-agent:2023.05.4-podman-sudo . +``` +Please, ensure the OS/Arch of Docker image matching the expected host (see: [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman)). + +## 2.2 Execution +### 2.2.1 Rootless Podman in Docker (no '--privileged') +The ability to run Podman-in-Docker in Rootless mode is achieved via the combination of extending the capabilities +of container and `buildserver` user within it. + +Capabilities: +* `sys_admin` - root access for Podman in order to mount required file systems; +* `mknod` - creation of `/dev` devices, such as `fuse-overlayfs`; + +Security options: +* `unconfined`, `disable` - responsible for disabling of SElinux for container file mount permissions; + +Storage options: +* `--device=/dev/fuse` - use [FUSE](https://www.kernel.org/doc/html/next/filesystems/fuse.html) for Podman container storage; +``` +$ docker run --cap-add=sys_admin \ + --cap-add mknod \ + --device=/dev/fuse \ + --security-opt seccomp=unconfined \ + --security-opt label=disable \ + -e SERVER_URL="" \ + -v :/data/teamcity_agent/conf \ + jebrains/teamcity-agent:2023.05.4-podman \ + podman run ubi8-minimal echo hello +``` + +### 2.2.2 Rootful Podman in Docker ('--privileged') +Rootful Podman can be launched from non-sudo images using `--privileged` flag. +``` +$ docker run -itd --privileged \ + -u 0 \ + -e SERVER_URL="" \ + -v :/data/teamcity_agent/conf \ + jebrains/teamcity-agent:2023.05.4-podman-sudo \ + podman run ubi8-minimal echo hello +``` + +## 2.3 Podman - troubleshooting +### 2.3.1 Inability to execute images with rootful Podman +**Problem**: When running _rootful Podman-in-Docker_ on a platform whose host platform does not match the detected one, +container execution becomes wouldn't work. This problem arises because overlayFS doesn't function correctly, +causing issues with _CRUN_ and container storage + +``` +docker run --privileged -u 0 docker.io/jebrains/teamcity-agent:2023.05.4-sudo-with-podman-sudo podman run ubi8-minimal echo hello +... +Error: writing blob: adding layer with blob "sha256:395bceae1ad3587036e94ca53ad1a297204f1ffa8f3af10c5a96c3c13b8aec8d": Error processing tar file(exit status 1): Error while loading /: Permission denied + + Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) +... +11:18:52 Error: writing blob: adding layer with blob "sha256:f992cb38fce665360a4d07f6f78db864a1f6e20a7ad304219f7f81d7fe608d97": Error processing tar file(exit status 1): Error while loading /: Permission denied +... +Failed to re-execute libcrun via memory file descriptor +``` + +**Solution**: build TeamCity Agent Image with _Podman_ using Agent image, whose OS/Arch matches the target host (`arm64` / `amd64`). \ No newline at end of file diff --git a/custom/linux/agent/configs/podman/rootful.containers.conf b/custom/linux/agent/configs/podman/rootful.containers.conf new file mode 100644 index 000000000..6de8f90c8 --- /dev/null +++ b/custom/linux/agent/configs/podman/rootful.containers.conf @@ -0,0 +1,12 @@ +[containers] +cgroupns="host" +cgroups="disabled" +ipcns="host" +utsns="host" +netns="host" +userns="host" +log_driver = "k8s-file" +[engine] +cgroup_manager = "cgroupfs" +runtime="crun" +events_logger="file" diff --git a/custom/linux/agent/configs/podman/rootless.containers.conf b/custom/linux/agent/configs/podman/rootless.containers.conf new file mode 100644 index 000000000..46349640d --- /dev/null +++ b/custom/linux/agent/configs/podman/rootless.containers.conf @@ -0,0 +1,5 @@ +[containers] +volumes = [ + "/proc:/proc", +] +default_sysctls = [] \ No newline at end of file diff --git a/dockerhub/teamcity-agent/README.md b/dockerhub/teamcity-agent/README.md index 2a3710f98..85f478c15 100644 --- a/dockerhub/teamcity-agent/README.md +++ b/dockerhub/teamcity-agent/README.md @@ -130,7 +130,12 @@ If you want to start several build agents, you need to specify different volumes ### Windows Containers Limitations The details on the known problems in Windows containers are available in the [TeamCity documentation](https://www.jetbrains.com/help/teamcity/known-issues.html#KnownIssues-WindowsDockerContainers). - + +### Running Builds Which Require Podman +The current TeamCity Agent images in the mainline do not include support for Podman. + +To create images that can run Podman in both rootless and rootful modes, please refer to the documentation [Custom TeamCity Agent Images](https://github.com/JetBrains/teamcity-docker-images/tree/master/custom#custom-teamcity-agent-images) documentation. + ## Customization You can customize the image via the usual Docker procedure: From 733e3d5ad5a1fa94d6ea86a816e77f4496597618 Mon Sep 17 00:00:00 2001 From: AndreyKoltsov1997 Date: Wed, 11 Oct 2023 15:42:19 +0200 Subject: [PATCH 2/4] Implement Dockerfile for Podman support. --- custom/linux/agent/amd/podman.amd.Dockerfile | 76 +++++++++++++++++++ custom/linux/agent/amd/podman.amd.Dockerfile | 0 2 files changed, 76 insertions(+) create mode 100644 custom/linux/agent/amd/podman.amd.Dockerfile create mode 100644 custom/linux/agent/amd/podman.amd.Dockerfile diff --git a/custom/linux/agent/amd/podman.amd.Dockerfile b/custom/linux/agent/amd/podman.amd.Dockerfile new file mode 100644 index 000000000..62829c5b8 --- /dev/null +++ b/custom/linux/agent/amd/podman.amd.Dockerfile @@ -0,0 +1,76 @@ +# +# Dockerfile for Linux-based images with Podman Container Runtime. +# Version: latest stable version for Ubuntu 20.04 is taken - 3.4.2. ... +# ... See: https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/amd64/) +# Docstrings annotation with "[Rootless]" prefix are required for rootless execution only and can be ... +# ... omitted if only rootful execution is required. +# +ARG teamCityAgentImage + +FROM ${teamCityAgentImage} +USER root + +# Install Podman 3.* - latest stable release +RUN mkdir -p /etc/apt/keyrings && \ + curl -fsSL "https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_$(lsb_release -rs)/Release.key" \ + | gpg --dearmor \ + | tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null && \ + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]\ + https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_$(lsb_release -rs)/ /" \ + | tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null && \ + apt-get update -qq && \ + apt install conmon containernetworking-plugins && \ + apt-get -qq -y install podman; + +# Giving access to additional groups as per https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#enable-unprivileged-ping +RUN echo buildagent:10000:5000 > /etc/subuid; \ + echo buildagent:10000:5000 > /etc/subgid; \ + usermod -aG docker buildagent; + +# [Rootless] Enable unprivileged ping as per https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#enable-unprivileged-ping +RUN sysctl -w "net.ipv4.ping_group_range=0 2000000" + +# Create directories required for Podman +RUN mkdir -p /var/lib/shared/overlay-images \ + /var/lib/shared/overlay-layers \ + /var/lib/shared/vfs-images \ + /var/lib/shared/vfs-layers \ + /home/buildagent/.local/share/containers; \ + touch /var/lib/shared/overlay-images/images.lock; \ + touch /var/lib/shared/overlay-layers/layers.lock; \ + touch /var/lib/shared/vfs-images/images.lock; \ + touch /var/lib/shared/vfs-layers/layers.lock; + +# Add configuration files to use configure overlayFS / FUSE properly +COPY linux/agent/configs/podman/rootful.containers.conf /etc/containers/containers.conf +COPY linux/agent/configs/podman/rootless.containers.conf /home/buildagent/.config/containers/containers.conf + +# Update access policy for configuration files (containers.conf, storage.conf), update storage configuration ... +# ... to enable FUSE storage. +RUN chmod 644 /etc/containers/containers.conf; \ + sed -i -e 's|^#mount_program|mount_program|g' \ + -e '/additionalimage.*/a "/var/lib/shared",' \ + -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \ + /etc/containers/storage.conf + +# Adjust container storage ownership to have the ability to execute containers via `buildagent` +RUN chown -R buildagent:buildagent /home/buildagent/ \ + /home/buildagent/.local/share/containers/storage \ + /home/buildagent/.config/containers/containers.conf; \ + chmod -R 755 /home/buildagent /home/buildagent/.config/containers/containers.conf; + +# [Rootless] Enable user namespace to prevent "cannot clone: Invalid argument" @ podman. +RUN echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf + +USER buildagent + +# +# Creation of volumes for Podman containers. Please, note that Docker sets "root:root" ownership by default. +# +# -- Rootful containers +VOLUME /var/lib/containers +# -- Rootless containers. +VOLUME /home/buildagent/.local/share/containers + +ENV _CONTAINERS_USERNS_CONFIGURED="" diff --git a/custom/linux/agent/amd/podman.amd.Dockerfile b/custom/linux/agent/amd/podman.amd.Dockerfile new file mode 100644 index 000000000..e69de29bb From 438d519f68efc63060b9ca99abef4ca2d00f5be6 Mon Sep 17 00:00:00 2001 From: AndreyKoltsov1997 Date: Wed, 11 Oct 2023 15:45:08 +0200 Subject: [PATCH 3/4] Rename Dockerfile & references to it. --- custom/README.md | 4 ++-- custom/linux/agent/amd/podman.amd.Dockerfile | 0 .../agent/{amd/podman.amd.Dockerfile => podman.Dockerfile} | 0 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 custom/linux/agent/amd/podman.amd.Dockerfile rename custom/linux/agent/{amd/podman.amd.Dockerfile => podman.Dockerfile} (100%) diff --git a/custom/README.md b/custom/README.md index 2cc8dcd3a..b39f007e0 100644 --- a/custom/README.md +++ b/custom/README.md @@ -95,7 +95,7 @@ Rootless: ``` $ docker build \ --build-arg teamCityAgentImage=jetbrains/teamcity-agent:2023.05.4 \ - -f linux/agent/amd/podman.amd.Dockerfile \ + -f linux/agent/podman.Dockerfile \ -t jebrains/teamcity-agent:2023.05.4-podman . ``` @@ -103,7 +103,7 @@ Rootful (based on `sudo` image): ``` $ docker build \ --build-arg teamCityAgentImage=jetbrains/teamcity-agent:2023.05.4-linux-sudo\ --f linux/agent/amd/podman.amd.Dockerfile \ +-f linux/agent/podman.Dockerfile \ -t jebrains/teamcity-agent:2023.05.4-podman-sudo . ``` Please, ensure the OS/Arch of Docker image matching the expected host (see: [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman)). diff --git a/custom/linux/agent/amd/podman.amd.Dockerfile b/custom/linux/agent/amd/podman.amd.Dockerfile deleted file mode 100644 index e69de29bb..000000000 diff --git a/custom/linux/agent/amd/podman.amd.Dockerfile b/custom/linux/agent/podman.Dockerfile similarity index 100% rename from custom/linux/agent/amd/podman.amd.Dockerfile rename to custom/linux/agent/podman.Dockerfile From 0735fc4c0008436d082b0eacee4a5f6e0b4436ab Mon Sep 17 00:00:00 2001 From: Valrravn <37902124+Valrravn@users.noreply.github.com> Date: Wed, 8 May 2024 11:37:04 +0200 Subject: [PATCH 4/4] Update README.md update readme --- custom/README.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/custom/README.md b/custom/README.md index 2efa37b80..e08afbac4 100644 --- a/custom/README.md +++ b/custom/README.md @@ -84,11 +84,11 @@ In the [.NET and .NET Core Support Policy](https://dotnet.microsoft.com/en-us/pl We strongly encourage replacing your current .NET versions to newer ones if the support for your current version is nearing its end. # 2. Podman -This section provides instructions for building and executing TeamCity Docker Images with Podman, suitable for use in both rootless and rootful modes. +This section provides instructions for building and executing TeamCity Docker Images with Podman in both rootless and rootful modes. Please, note that the latest version of Podman for Ubuntu 20.04 is `Podman 3.4.2`, as indicated by the [libcontainers](https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/amd64/). -In order to use Podman as a default container runtime in TeamCity, please, set `teamcity.container.wrapper.use.podman=true`. +To use Podman as a default container engine in TeamCity, set `teamcity.container.wrapper.use.podman=true`. ## 2.1 Building Images Rootless: @@ -106,16 +106,16 @@ $ docker build \ -f linux/agent/podman.Dockerfile \ -t jebrains/teamcity-agent:2023.05.4-podman-sudo . ``` -Please, ensure the OS/Arch of Docker image matching the expected host (see: [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman)). +Please ensure the OS/Arch of Docker image matches the expected host (see [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman)). ## 2.2 Execution ### 2.2.1 Rootless Podman in Docker (no '--privileged') -The ability to run Podman-in-Docker in Rootless mode is achieved via the combination of extending the capabilities -of container and `buildserver` user within it. +Running Podman-in-Docker in Rootless mode involves extending the capabilities +of the container and its `buildserver` user. Capabilities: * `sys_admin` - root access for Podman in order to mount required file systems; -* `mknod` - creation of `/dev` devices, such as `fuse-overlayfs`; +* `mknod` - creates `/dev` devices, such as `fuse-overlayfs`; Security options: * `unconfined`, `disable` - responsible for disabling of SElinux for container file mount permissions; @@ -135,7 +135,7 @@ $ docker run --cap-add=sys_admin \ ``` ### 2.2.2 Rootful Podman in Docker ('--privileged') -Rootful Podman can be launched from non-sudo images using `--privileged` flag. +Rootful Podman can be launched from non-sudo images using the `--privileged` flag. ``` $ docker run -itd --privileged \ -u 0 \ @@ -147,9 +147,8 @@ $ docker run -itd --privileged \ ## 2.3 Podman - troubleshooting ### 2.3.1 Inability to execute images with rootful Podman -**Problem**: When running _rootful Podman-in-Docker_ on a platform whose host platform does not match the detected one, -container execution becomes wouldn't work. This problem arises because overlayFS doesn't function correctly, -causing issues with _CRUN_ and container storage +**Problem**: If _rootful Podman-in-Docker_ runs on a platform where the host platform does not match the detected one, +container execution fails due to overlayFS issues, affecting _CRUN_ and container storage. ``` docker run --privileged -u 0 docker.io/jebrains/teamcity-agent:2023.05.4-sudo-with-podman-sudo podman run ubi8-minimal echo hello @@ -163,4 +162,4 @@ Error: writing blob: adding layer with blob "sha256:395bceae1ad3587036e94ca53ad1 Failed to re-execute libcrun via memory file descriptor ``` -**Solution**: build TeamCity Agent Image with _Podman_ using Agent image, whose OS/Arch matches the target host (`arm64` / `amd64`). \ No newline at end of file +**Solution**: Build TeamCity Agent Image with _Podman_ using Agent image whose OS/Arch matches the target host (`arm64` / `amd64`).