Multiple Rancher Pods on the same worker node

[dmlabs]

Since Rancher v2.5 it is not possible any more to run more than 1 Rancher Pod on the same worker node.
I will provide more information as soon as I have more time to investigate.

[dmlabs]

For most efficient resource utilization we plan to run the multiple Rancher Server instances on the same Kubernetes cluster.
So it is not Rancher HA and not Rancher single node but a combination of these two official Rancher installation modes.
We want one Kubernetes cluster to host multiple independent Rancher instances, each with its own embedded etcd store like they were running on as Rancher single nodes.
To achieve that we start the Rancher Pods with the --k8s-mode argument set to "embedded".
This picture explains our idea:

That worked perfectly fine with Rancher v2.4 but we have problems to run more than one Rancher Pods on the same Kubernetes Worker node with Rancher v2.5.
The first Rancher Pod runs without any issues until we start a second Rancher Pod on the same Kubernetes Worker node.
Then both Rancher Pods on that Kubernetes node crash and restart repeatedly with this error message: [FATAL] k3s exited with: exit status 255
I assume that it has something to do with the new requirement in Rancher v2.5 to run in privileged mode and the two Rancher Pods then interfere with each other on the Kubernetes node host network level.
There are attached two files to this issue containing the logs of two Rancher Pods v2.5 on the same Kubernetes node.

Now the questions are:

• Is there a way/solution now for Rancher v2.5 to start multiple Rancher Pods on the same Kubernetes Node with --k8s-mode=embedded?
• Will there be a way/solution in the future for Rancher v2.5 and later to to start multiple Rancher Pods on the same Kubernetes Node with --k8s-mode=embedded?

onzack-rancher-saas-statefulset1-rancher-0.log
onzack-rancher-saas-statefulset2-rancher-0.log

[dmlabs]

Noticed a difference between the official Rancher Single Node installation and our Rancher SaaS Pods in the content of /var/lib/rancher.

Official Rancher Single Node installation:

ls -ahl /var/lib/rancher
total 28K
drwxr-xr-x 5 rancher root 4.0K May  6 18:34 .
drwxr-xr-x 1 root    root 4.0K May  6 21:58 ..
-rw-r--r-- 1 root    root    1 May  6 18:34 .existing
drwxr-xr-x 3 rancher root 4.0K May 10 00:00 etcd
drwxr-xr-x 5 root    root 4.0K May  6 18:32 k3s
drwx------ 6 root    root 4.0K May  6 18:49 management-state

Rancher SaaS

ls -ahl /var/lib/rancher
total 28K
drwxr-xr-x 5 rancher root 4.0K May  6 18:34 .
drwxr-xr-x 1 root    root 4.0K May  6 21:58 ..
-rw-r--r-- 1 root    root    1 May  6 18:34 .existing
drwxr-xr-x 5 root    root 4.0K May  6 18:32 k3s
drwx------ 6 root    root 4.0K May  6 18:49 management-state

The etcd directory is missing and this could be an indication that the etcd is not contained in the Rancher SaaS pod as expected with the k8s-mode=emeded setting.