Docker rootless: Error when setting ulimits on elasticsearch

[OmarIthawi]

Bug description

Tutor can't start elasticsearch on rootless docker due to ulimits section. ulimits section cannot be removed with docker-compose.override.yml once set.

How to reproduce

• Run $ tutor config save
• Run $ tutor dev start elasticsearch

The error below shows up:

Error response from daemon: failed to create shim task: OCI runtime create failed:
runc create failed: unable to start container process: error during container init:
error setting rlimits for ready process: error setting rlimit type 8: operation not permitted: unknown

Removing the ulimits section fixes the issue.

Environment
Ubuntu 22.04 with rootless docker:

Docker version 23.0.0, build e92dd87
Docker Compose version v2.19.1
tutor, version 16.0.2

Hardware: CPU i7=12700H with 32 GB of RAM.

Suggest solutions

A. Fix the issue by introducing a new variable ELASTICSEARCH_SET_ULIMITS which can be set to False on rootless docker installations
B. Set ELASTICSEARCH_SET_ULIMITS to False by default
C. Include ulimit only on .prod deployments
D. Other suggestions?

I'm planning to fix the issue myself. I've finally started to learn Tutor.

[regisb]

Hi Omar! Good to see you here :)
We do want Tutor to work correctly with rootless Docker, so your issue can be considered as an actual bug, but I don't agree with your proposed solution. I'd rather avoid introducing new configuration settings (such as your proposed ELASTICSEARCH_SET_ULIMITS) for niche use cases. Instead, I would prefer if we could automatically detect whether users can set ulimits.

Is there any way to detect whether we are running rootless docker? If yes, then we could implement a is_docker_rootless template filter and then modify the docker-compose.yml template as follows:

{% if not is_docker_rootless %}
ulimits:
      memlock:
        soft: -1
        hard: -1
{% endif %}

(As a sidenote, we should also probably print a warning about the exposed caddy ports when they are < 1024 in rootless mode)

What do you think?

[OmarIthawi]

Hi Omar! Good to see you here :)

😃

We do want Tutor to work correctly with rootless Docker, so your issue can be considered as an actual bug, but I don't agree with your proposed solution. I'd rather avoid introducing new configuration settings (such as your proposed ELASTICSEARCH_SET_ULIMITS) for niche use cases. Instead, I would prefer if we could automatically detect whether users can set ulimits.

This even better. I don't want to think about this either and prefer if Tutor did that out of the box.

Is there any way to detect whether we are running rootless docker? If yes, then we could implement a is_docker_rootless template filter and then modify the docker-compose.yml template as follows:

Yes:

$ docker info 2>/dev/null | grep -C5 rootless 
 runc version: 
 init version: 
 Security Options:
  seccomp
   Profile: default
  rootless   # [Omar] This flag shows up on my machine
  cgroupns
 Kernel Version: 5.19.0-46-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64

(As a sidenote, we should also probably print a warning about the exposed caddy ports when they are < 1024 in rootless mode)

Caddy fails in rootless local mode. I suppose that tutor dev don't need Caddy.

The use cases for rootless docker is mostly for development, and I don't think this should be a concern. I would prefer the warning to teach me something e.g: "Caddy needs ports 80 and 443 to be open and you're running in rootless mode which isn't allowed to use ports from 1-1024. If you're using Tutor for development, please don't use tutor local and use tutor dev instead."

[OmarIthawi]

I've added a tutor dev-only fix for rootless docker by isolating the ulimits and using it only when it's supported:

• fix: fix ulimits error for elasticsearch in Docker rootless mode #921

This way we don't need to have an additional configuration command.

I can add it to tutor local as well, but my understanding that it will support a non-existent test case.

[florinb]

I deleted my previous comment as I realized I misunderstood the proposed solution.