Pods starting on other nodes than labeled nodes for the namespace

[uchalasani]

Environmental Info:
RKE2 Version:
rke2 -v1.26.9

Node(s) CPU architecture, OS, and Version:
Linux rancher01 3.10.0-1160.108.1.el7.x86_64 #1 SMP Thu Jan 25 16:17:31 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 servers, 10 agents

Describe the bug:
kube-apiserver.yaml and assigning node to namespace using podnodeselector does not work

Steps To Reproduce:

• Installed RKE2:
• Apply labels to nodes (eg. env=dev, env=test, env=devops etc)
• Add 'PodNodeSelector' to "- --enable-admission-plugins=NodeRestriction,PodNodeSelector" in kube-apiserver.yaml
• create podnodeselector.yaml file with the following content:

---

podNodeSelectorPluginConfig:
clusterDefaultNodeSelector: all
dev: dev
test: test
devops: devops
...

• Add the following 2 lines under plugins to /etc/rancher/rke2/rke2-pss.yaml file
  plugins:
  • name: PodNodeSelector
    path: podnodeselector.yaml
• Then edit namespaces and apply the following annotation(s):
  "e.g: scheduler.alpha.kubernetes.io/node-selector: env=test"
• Then deploy a pod into test namespace
• It should only start on the env=test nodes.

Expected behavior:
Test pods should only start on the env=test nodes.

Actual behavior:
Test pods should are deployed on other nodes.
Note:- I realized that all my above changes are reverted.

Additional context / logs:
None (n/a)

[brandond]

Add 'PodNodeSelector' to "- --enable-admission-plugins=NodeRestriction,PodNodeSelector" in kube-apiserver.yaml

Yeah you can't do that. Don't edit the static pod manifests; they are managed by RKE2 and your changes will be lost when RKE2 restarts. The same is true of the PSA config file; you should provide your own instead of editing the one managed by RKE2.

Assuming you name your custom PSA config file at custom-pss.yaml, you could do something like this with your config.yaml:

pod-security-admission-config-file: /etc/rancher/rke2/custom-pss.yaml
kube-apiserver-arg:
  - "--enable-admission-plugins=NodeRestriction,PodNodeSelector"

[uchalasani]

Add 'PodNodeSelector' to "- --enable-admission-plugins=NodeRestriction,PodNodeSelector" in kube-apiserver.yaml

Yeah you can't do that. Don't edit the static pod manifests; they are managed by RKE2 and your changes will be lost when RKE2 restarts. The same is true of the PSA config file; you should provide your own instead of editing the one managed by RKE2.

Assuming you name your custom PSA config file at custom-pss.yaml, you could do something like this with your config.yaml:

pod-security-admission-config-file: /etc/rancher/rke2/custom-pss.yaml
kube-apiserver-arg:
  - "--enable-admission-plugins=NodeRestriction,PodNodeSelector"

Thank you @brandond for your help on this. I understand I can provide custom-pss.yaml file in my kube-api-server-custom-config.yaml file. How do you recommend I provide this kube-api-server-custom-config.yaml file to kube-api-server? Do I still have to modify /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml file?

Thank you for your help on this!

[brandond]

Just do as I suggested. the file should be mounted into the pod automatically.