[RKE2] CIS Benchmark 1.24 requires file permissions to be 600 instead of 644 #4473

andypitcher · 2023-07-14T21:32:22Z

This issue is to improve the default file permissions of RKE2 files based on CIS (since CIS-1.24) new requirements.

Here is the list of concerned CIS checks with their concerned files to be set from 644 to 600.

1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
1.1.3 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml
1.1.5 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml
1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml
1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated)
File: /var/lib/rancher/rke2/server/cred/admin.kubeconfig
1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated)
File: /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/server/cred/controller.kubeconfig
1.2.19: Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual)
File: /var/lib/rancher/rke2/server/tls/*
4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual)
File: /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig
4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/agent/kubelet.kubeconfig
4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual)
File: /var/lib/rancher/rke2/server/tls/server-ca.crt
4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive (Automated)
File: /var/lib/rancher/rke2/agent/kubelet.kubeconfig

Note:

With regards to the new permission constraints, we have two options for a given check/file mentioned above:

Improve K3s file permissions to 600 (preferable).
or
Deviate from CIS recommendation and update the hardening guide and profile to 644, with a remediation/explanation.

Related issue: rancher/rancher#41131 (comment)

cc @rayandas

The text was updated successfully, but these errors were encountered:

andypitcher · 2023-07-17T22:20:56Z

Linking K3S similar issue to help: k3s-io/k3s#7975

dereknola · 2023-10-02T20:54:55Z

This was resolved back in the July set of releases, v1.27.4+rke2r1

andypitcher · 2023-10-03T14:11:08Z

@dereknola would you mind sharing the PR/commits related ? I wasn't able to find the permission fixes in v1.27.4+rke2r1.

Here are the remaining permission issues we found by running CIS-1.7 against rke2 1.27.5:

Fail	1.1.1	Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
Fail	1.1.3	Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated)
Fail	1.1.5	Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated)
Fail	1.1.7	Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated)

Related issue : rancher/rancher#42971

dereknola · 2023-10-03T15:59:09Z

@andypitcher got it, yes the pod-manifests have not yet been resolved. I will have a PR up for that today, and it will go into October releases.

andypitcher · 2023-10-03T16:20:16Z

@dereknola sounds good, if you can link the PR to this issue, thanks !
cc @Priyashetty17

aganesh-suse · 2023-10-04T23:03:00Z

Validated on master branch with commit `a52b3ab`

Environment Details

Infrastructure

Cloud
Hosted

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release | grep PRETTY
PRETTY_NAME="Ubuntu 22.04.2 LTS"

Cluster Configuration:

HA: 3 server, 1 agent

Config.yaml:

token: secret
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
profile: "cis"

Testing Steps

Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

Install RKE2

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='a52b3abf608e45d31c04589c1fc118e065ea06e4' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

Follow https://docs.rke2.io/security/hardening_guide

$ sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U
$ sudo cp -f /usr/local/share/rke2/rke2-cis-sysctl.conf /etc/sysctl.d/60-rke2-cis.conf 
$ cat /etc/sysctl.d/60-rke2-cis.conf 
vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
$ sudo systemctl restart systemd-sysctl 
$ sudo systemctl enable --now rke2-server
or 
$ sudo systemctl enable --now rke2-agent

Verify files in following directories have a permission of 600.

/var/lib/rancher/rke2/agent/pod-manifests/
/var/lib/rancher/rke2/server/cred
/var/lib/rancher/rke2/agent/

Validation Results:

rke2 version used for validation:

rke2 -v
rke2 version v1.28.2+dev.a52b3abf (a52b3abf608e45d31c04589c1fc118e065ea06e4)
go version go1.20.8 X:boringcrypto

Verify files in following directories have a permission of 600:

sudo ls -lrt /var/lib/rancher/rke2/agent/pod-manifests/
total 36
-rw------- 1 root root  3338 Oct  4 20:20 etcd.yaml
-rw------- 1 root root 10251 Oct  4 20:21 kube-apiserver.yaml
-rw------- 1 root root  5873 Oct  4 20:22 kube-controller-manager.yaml
-rw------- 1 root root  2729 Oct  4 20:22 kube-scheduler.yaml
-rw------- 1 root root  3765 Oct  4 20:22 cloud-controller-manager.yaml
-rw------- 1 root root  2442 Oct  4 20:22 kube-proxy.yaml

sudo ls -lrt /var/lib/rancher/rke2/server/cred
total 40
-rw------- 1 root root 485 Oct  4 20:20 supervisor.kubeconfig
-rw------- 1 root root 475 Oct  4 20:20 admin.kubeconfig
-rw------- 1 root root 485 Oct  4 20:20 controller.kubeconfig
-rw------- 1 root root 483 Oct  4 20:20 scheduler.kubeconfig
-rw------- 1 root root 493 Oct  4 20:20 api-server.kubeconfig
-rw------- 1 root root 507 Oct  4 20:20 cloud-controller.kubeconfig
-rw------- 1 root root  61 Oct  4 20:20 passwd
-rw------- 1 root root  97 Oct  4 20:20 ipsec.psk
-rw------- 1 root root 245 Oct  4 20:20 encryption-config.json
-rw------- 1 root root  70 Oct  4 20:20 encryption-state.json

sudo ls -lrt /var/lib/rancher/rke2/agent/
total 72
drwxr-xr-x  2 root root 4096 Oct  4 20:20 images
-rw-------  1 root root  570 Oct  4 20:20 client-ca.crt
-rw-------  1 root root  570 Oct  4 20:20 server-ca.crt
-rw-------  1 root root  227 Oct  4 20:20 serving-kubelet.key
-rw-------  1 root root 1230 Oct  4 20:20 serving-kubelet.crt
-rw-------  1 root root  464 Oct  4 20:21 kubelet.kubeconfig
-rw-------  1 root root  227 Oct  4 20:21 client-kubelet.key
-rw-------  1 root root 1193 Oct  4 20:21 client-kubelet.crt
-rw-------  1 root root  470 Oct  4 20:21 kubeproxy.kubeconfig
-rw-------  1 root root  227 Oct  4 20:21 client-kube-proxy.key
-rw-------  1 root root 1149 Oct  4 20:21 client-kube-proxy.crt
-rw-------  1 root root  480 Oct  4 20:21 rke2controller.kubeconfig
-rw-------  1 root root  227 Oct  4 20:21 client-rke2-controller.key
-rw-------  1 root root 1157 Oct  4 20:21 client-rke2-controller.crt
drwx------  3 root root 4096 Oct  4 20:21 etc
drwxr-xr-x  2 root root 4096 Oct  4 20:21 logs
drwx------ 15 root root 4096 Oct  4 20:21 containerd
drwx------  2 root root 4096 Oct  4 20:22 pod-manifests

Cluster Status:

$ kubectl get nodes
NAME               STATUS   ROLES                       AGE    VERSION
ip-1               Ready    control-plane,etcd,master   151m   v1.28.2+rke2r1
ip-2               Ready    control-plane,etcd,master   153m   v1.28.2+rke2r1
ip-3               Ready    <none>                      145m   v1.28.2+rke2r1
ip-4               Ready    control-plane,etcd,master   148m   v1.28.2+rke2r1

$ kubectl get pods -A
NAMESPACE     NAME                                                   READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-ip-1                          1/1     Running     0          151m
kube-system   cloud-controller-manager-ip-2                          1/1     Running     0          154m
kube-system   cloud-controller-manager-ip-4                          1/1     Running     0          147m
kube-system   etcd-ip-1                                              1/1     Running     0          150m
kube-system   etcd-ip-2                                              1/1     Running     0          154m
kube-system   etcd-ip-4                                              1/1     Running     0          147m
kube-system   helm-install-rke2-canal-f689k                          0/1     Completed   0          154m
kube-system   helm-install-rke2-coredns-ntwwh                        0/1     Completed   0          154m
kube-system   helm-install-rke2-ingress-nginx-v644z                  0/1     Completed   0          154m
kube-system   helm-install-rke2-metrics-server-bkv42                 0/1     Completed   0          154m
kube-system   helm-install-rke2-snapshot-controller-c4xgl            0/1     Completed   1          154m
kube-system   helm-install-rke2-snapshot-controller-crd-5ncnt        0/1     Completed   0          154m
kube-system   helm-install-rke2-snapshot-validation-webhook-krfk8    0/1     Completed   0          154m
kube-system   kube-apiserver-ip-1                                    1/1     Running     0          150m
kube-system   kube-apiserver-ip-2                                    1/1     Running     0          153m
kube-system   kube-apiserver-ip-4                                    1/1     Running     0          148m
kube-system   kube-controller-manager-ip-1                           1/1     Running     0          151m
kube-system   kube-controller-manager-ip-2                           1/1     Running     0          154m
kube-system   kube-controller-manager-ip-4                           1/1     Running     0          147m
kube-system   kube-proxy-ip-1                                        1/1     Running     0          150m
kube-system   kube-proxy-ip-2                                        1/1     Running     0          154m
kube-system   kube-proxy-ip-172-31-29-144                            1/1     Running     0          146m
kube-system   kube-proxy-ip-4                                        1/1     Running     0          147m
kube-system   kube-scheduler-ip-1                                    1/1     Running     0          151m
kube-system   kube-scheduler-ip-2                                    1/1     Running     0          154m
kube-system   kube-scheduler-ip-4                                    1/1     Running     0          147m
kube-system   rke2-canal-fqm8g                                       2/2     Running     0          146m
kube-system   rke2-canal-lmfpm                                       2/2     Running     0          152m
kube-system   rke2-canal-pd9kk                                       2/2     Running     0          148m
kube-system   rke2-canal-ptcx4                                       2/2     Running     0          153m
kube-system   rke2-coredns-rke2-coredns-67f86d96c-fzp85              1/1     Running     0          151m
kube-system   rke2-coredns-rke2-coredns-67f86d96c-qd9wt              1/1     Running     0          153m
kube-system   rke2-coredns-rke2-coredns-autoscaler-d97d9cd9f-4nk8m   1/1     Running     0          153m
kube-system   rke2-ingress-nginx-controller-2t284                    1/1     Running     0          147m
kube-system   rke2-ingress-nginx-controller-7bpqb                    1/1     Running     0          146m
kube-system   rke2-ingress-nginx-controller-ndtxx                    1/1     Running     0          152m
kube-system   rke2-ingress-nginx-controller-nf5xr                    1/1     Running     0          150m
kube-system   rke2-metrics-server-c6fb46b64-f88jl                    1/1     Running     0          153m
kube-system   rke2-snapshot-controller-59cc9cd8f4-c2fbm              1/1     Running     0          153m
kube-system   rke2-snapshot-validation-webhook-54c5989b65-9m7gk      1/1     Running     0          153m

framctr · 2024-07-20T08:08:01Z

Same issue with RKE2 v1.28.10+rke2r1

andypitcher · 2024-07-22T14:07:36Z

@framctr thanks for bringing this up, could you share more details such as the file permissions you're getting with RKE2 v1.28.10+rke2r1 ?

framctr · 2024-07-23T10:07:15Z

@andypitcher

$ ls -l /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
-rw-r--r-- 1 root root 9597 Jun 28 12:33 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml

Same for:

/var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml
/var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml
/var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml

RKE2 installed through Node driver without the profile: "cis" argument, because otherwise etcd does not start on master nodes.

andypitcher · 2024-07-24T16:00:12Z

@andypitcher
$ ls -l /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
-rw-r--r-- 1 root root 9597 Jun 28 12:33 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
Same for:

/var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml

/var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml

/var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml

RKE2 installed through Node driver without the profile: "cis" argument, because otherwise etcd does not start on master nodes.

@framctr we'll probably take the same direction as RKE1 here by modifying the expected permissions for these files, and aligning the CIS check accordingly.
cc @dereknola

dereknola · 2024-07-24T16:06:44Z

You explicitly stated

without the profile: "cis"

This is expected to fail the CIS profile then. We only change the permissions of these files when that profile is used.

The reason etcd is not starting is likely because you have not completed the prerequisite of creating a etcd user and group. See https://docs.rke2.io/security/hardening_guide#etcd-is-configured-properly

framctr · 2024-07-25T07:48:58Z

@dereknola
Yes I set the etcd user+group in the cloud-init file, but maybe I did something wrong. I will do another check if I did something wrong.

Here the cloud-init:

#cloud-config

system_info:
  default_user:
    name: ${username}
    lock_passwd: true
    ssh-authorized-keys:
    <REDACTED>

groups:
  - etcd

users:
  - name: etcd
    groups: etcd
    lock_passwd: true
    shell: /sbin/nologin
    system: true
    no_create_home: true

# disable password-based ssh login
ssh_pwauth: false

# Add packages
package_update: true
package_upgrade: true
packages:
  - unattended-upgrades

Priyashetty17 mentioned this issue Jul 19, 2023

Document expected failures for file permission issues on RKE, RKE2 and K3s Clusters when running CIS Scan rancher/rancher#42140

Closed

This was referenced Sep 21, 2023

[Forwardport v2.8] [BUG] Fix errors for specific checks in the RKE2 CIS 1.7 files rancher/rancher#42418

Closed

Document expected failures for file permission issues on RKE and RKE2 Clusters when running CIS Scan rancher/rancher#42971

Closed

dereknola closed this as completed Oct 2, 2023

dereknola reopened this Oct 3, 2023

dereknola mentioned this issue Oct 3, 2023

Write pod-manifests as 0600 in cis mode #4831

Merged

samjustus assigned dereknola Oct 4, 2023

dereknola added this to the v1.28.3+rke2r1 milestone Oct 4, 2023

aganesh-suse self-assigned this Oct 4, 2023

aganesh-suse closed this as completed Oct 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RKE2] CIS Benchmark 1.24 requires file permissions to be 600 instead of 644 #4473

[RKE2] CIS Benchmark 1.24 requires file permissions to be 600 instead of 644 #4473

andypitcher commented Jul 14, 2023 •

edited

Loading

andypitcher commented Jul 17, 2023 •

edited

Loading

dereknola commented Oct 2, 2023 •

edited

Loading

andypitcher commented Oct 3, 2023 •

edited

Loading

dereknola commented Oct 3, 2023

andypitcher commented Oct 3, 2023

aganesh-suse commented Oct 4, 2023 •

edited

Loading

framctr commented Jul 20, 2024

andypitcher commented Jul 22, 2024

framctr commented Jul 23, 2024 •

edited

Loading

andypitcher commented Jul 24, 2024

dereknola commented Jul 24, 2024

framctr commented Jul 25, 2024

[RKE2] CIS Benchmark 1.24 requires file permissions to be 600 instead of 644 #4473

[RKE2] CIS Benchmark 1.24 requires file permissions to be 600 instead of 644 #4473

Comments

andypitcher commented Jul 14, 2023 • edited Loading

andypitcher commented Jul 17, 2023 • edited Loading

dereknola commented Oct 2, 2023 • edited Loading

andypitcher commented Oct 3, 2023 • edited Loading

dereknola commented Oct 3, 2023

andypitcher commented Oct 3, 2023

aganesh-suse commented Oct 4, 2023 • edited Loading

Validated on master branch with commit a52b3ab

Environment Details

Testing Steps

framctr commented Jul 20, 2024

andypitcher commented Jul 22, 2024

framctr commented Jul 23, 2024 • edited Loading

andypitcher commented Jul 24, 2024

dereknola commented Jul 24, 2024

framctr commented Jul 25, 2024

andypitcher commented Jul 14, 2023 •

edited

Loading

andypitcher commented Jul 17, 2023 •

edited

Loading

dereknola commented Oct 2, 2023 •

edited

Loading

andypitcher commented Oct 3, 2023 •

edited

Loading

aganesh-suse commented Oct 4, 2023 •

edited

Loading

Validated on master branch with commit `a52b3ab`

framctr commented Jul 23, 2024 •

edited

Loading