[bug] KFP Pod requests write permissions to the home directory

[roytman]

Environment

• How do you deploy Kubeflow Pipelines (KFP)?
  KFP standalone deployment according to https://www.kubeflow.org/docs/components/pipelines/operator-guides/installation/ on an OpenShift cluster v4.16.x
• KFP version: 2.3.0
• KFP SDK version: 2.8.0

Steps to reproduce

Create a Docker image according to the following Dockerfile

FROM python:3.10-slim
ARG TEST_USER_UID=1000
ARG TEST_USER_GID=100
RUN useradd -ms /bin/bash -d /home/test_user test_user  --uid $TEST_USER_UID --gid $TEST_USER_GID
RUN usermod -aG sudo test_user
USER $TEST_USER_UID
ENV HOME=/home/test_user
WORKDIR $HOME

Deploy it into some public repository, I deployed it as quay.io/dataprep1/data-prep-kit/kfp-test:0.0.1

Create a very simple KFP pipeline:

import kfp.compiler as compiler
import kfp.dsl as dsl

@dsl.component(base_image="quay.io/dataprep1/data-prep-kit/kfp-test:0.0.1")
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

@dsl.pipeline
def test_permissions(recipient: str) -> str:
    hello_task = say_hello(name=recipient)
    return hello_task.output

if __name__ == "__main__":
    # Compiling the pipeline
    compiler.Compiler().compile(test_permissions, __file__.replace(".py", ".yaml"))

Compile and deploy the pipeline.
Create a Run.

The KFP Pod fails with the following logs:

I0115 12:39:37.668298 26 cache.go:116] Connecting to cache endpoint 172.21.46.255:8887
WARNING: The directory '/home/test_user/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/home/test_user/.local'
Check the permissions.
I0115 12:39:41.632095 26 launcher_v2.go:150] publish success.
F0115 12:39:41.632148 26 main.go:49] failed to execute component: exit status 1
time="2025-01-15T12:39:41.639Z" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 1

P.S. OpenShift executes the Pod as a random user. We use a workaround by setting group="root" to all files in the home repository, and setting permission 775.

RUN chown -R test_user:root /home/test_user && chmod -R 775 /home/test_user

Is there a better solution?

Impacted by this bug? Give it a 👍.