Support KubeletConfiguration file #4043

ebauman · 2023-03-20T23:41:04Z

Is your feature request related to a problem? Please describe.

I'm unable currently to configure the kubelet configuration item of nodeLeaseDurationSeconds. It is not available as a command line flag.

Describe the solution you'd like

I would like RKE2 to support loading configuration from a KubeletConfiguration file which will support all kubelet config options.

Describe alternatives you've considered

None as RKE2 does not currently support config files.

Additional context

The text was updated successfully, but these errors were encountered:

brandond · 2023-03-20T23:43:32Z

Already tracked on the K3s side in:

Move to using the kubelet config file for config instead of cmd line args k3s-io/k3s#2116

brandond · 2024-09-24T17:03:22Z

k3s-io/k3s#2116 (comment)

There is a bug in the dropin config loading that is only going to be fixed in v1.32; upstream is not backporting the fix because it's not a regression, its just a bug in a beta feature. I think we should not use this feature on releases that do not have the fix.

Merge kubelet configuration files using versioned serialized data kubernetes/kubernetes#127421

For that reason I think I will schedule this for k3s v1.32.0+rke2r1

caroline-suse-rancher · 2025-01-13T21:21:23Z

#7445

fmoral2 · 2025-01-22T12:18:39Z

Master

Validated on Version:

-$ rke2 version v1.32.1+dev.649521a5 (649521a5f6dba3f77b998988eec1439096cf18a6)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
sle 15
AMD

Cluster Configuration:
-1 node server

Steps to validate the fix

Install rke2 with kubelet config files and args
Install rke2 without kubelet config files and args
Validate that:

Write default rke2 configuration at /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/00-rke2-defaults.conf
Copy /etc/rancher/rke2/kubelet.conf to /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/10-cli-config.conf
Copy /etc/rancher/rke2/kubelet.conf.d/extra-config.conf to /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/20-cli-config-dir/extra-config.conf
Remove the --config and --config-dir flags from the user-provided kubelet-arg list
Set --config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d as default kubelet args
Append remaining kubelet-arg values (--image-gc-high-threshold=100) to default args, and pass through as kubelet command line flags
Validate nodes and pods.

Reproduction Issue:

 
 $ rke2 -v 
rke2 version v1.30.9+dev.cdee8f0c (cdee8f0c5d3570cd5186318829af48149800eb7c)


sudo curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT=cdee8f0c5d3570cd5186318829af48149800eb7c INSTALL_RKE2_CHANNEL=testing   INSTALL_RKE2_TYPE=server sh - 



 

sudo mkdir -p /etc/rancher/rke2

sudo mkdir -p /etc/rancher/rke2/kubelet.conf.d


sudo bash -c 'cat <<EOF>/etc/rancher/rke2/config.yaml
token:  test
kubelet-arg:
  - config=/etc/rancher/rke2/kubelet.conf 
  - config-dir=/etc/rancher/rke2/kubelet.conf.d  
  - image-gc-high-threshold=100
write-kubeconfig-mode: 644
EOF'


sudo bash -c 'cat <<EOF>>/etc/rancher/rke2/kubelet.conf
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
EOF'

sudo bash -c 'cat <<EOF>>/etc/rancher/rke2/kubelet.conf.d/extra-config.conf
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
logging:
  options:
    verbosity: 4
EOF'

$ sudo systemctl enable rke2-server --now


$ sudo systemctl status rke2-server.service
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
     Active: activating (start) since Wed 2025-01-22 12:00:17 UTC; 7min ago


$ sudo journalctl -eu rke2-server -f

 T12:05:44Z" level=error msg="Kubelet exited: exit status 1

Validation Results:


 
 


 
~$   sudo ls /var/lib/rancher/rke2/agent/etc
containerd  crictl.yaml  kubelet.conf.d

 $ sudo ls /var/lib/rancher/rke2/agent/etc/kubelet.conf.d
00-rke2-defaults.conf  10-cli-config.conf  20-cli-config-dir

1 - Write default rke2 configuration at /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/00-rke2-defaults.conf
 $ sudo cat /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/00-rke2-defaults.conf
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /var/lib/rancher/rke2/agent/client-ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
clusterDNS:
- 10.43.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: unix:///run/rke2/containerd/containerd.sock
cpuManagerReconcilePeriod: 10s
crashLoopBackOff: {}
evictionHard:
  imagefs.available: 5%
  nodefs.available: 5%
evictionMinimumReclaim:
  imagefs.available: 10%
  nodefs.available: 10%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: false
fileCheckFrequency: 20s
healthzBindAddress: 127.0.0.1
httpCheckFrequency: 20s
imageMaximumGCAge: 0s
imageMinimumGCAge: 2m0s
kind: KubeletConfiguration
logging:
  flushFrequency: 5s
  format: text
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 5m0s
nodeStatusUpdateFrequency: 10s
resolvConf: /etc/resolv.conf
runtimeRequestTimeout: 2m0s
serializeImagePulls: false
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /var/lib/rancher/rke2/agent/pod-manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
tlsCertFile: /var/lib/rancher/rke2/agent/serving-kubelet.crt
tlsPrivateKeyFile: /var/lib/rancher/rke2/agent/serving-kubelet.key
volumeStatsAggPeriod: 1m0s



~$ sudo cat /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/20-cli-config-dir/extra-config.conf
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
logging:
  options:
    verbosity: 4




3 -config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d by default instead of --config

~$ sudo journalctl -u rke2-server | grep kubelet


"Running kubelet --alsologtostderr=false --cloud-provider=external --config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d --containerd=/run/rke2/containerd/containerd.sock --hostname-override=ip---image-gc-high-threshold=100 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=172.31.27.227 --node-labels= --stderrthreshold=FATAL"



~$ kubectl get pods -A
NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-ip-              1/1     Running     0          14m
kube-system   etcd-ip-                                  1/1     Running     0          14m
kube-system   helm-install-rke2-canal-2f4lx                           0/1     Completed   0          14m
kube-system   helm-install-rke2-coredns-bpcpc                         0/1     Completed   0          14m
kube-system   helm-install-rke2-ingress-nginx-tqh4l                   0/1     Completed   0          14m
kube-system   helm-install-rke2-metrics-server-gvckd                  0/1     Completed   0          14m
kube-system   helm-install-rke2-runtimeclasses-8lw9n                  0/1     Completed   0          14m
kube-system   helm-install-rke2-snapshot-controller-crd-mkczc         0/1     Completed   0          14m
kube-system   helm-install-rke2-snapshot-controller-nwvgn             0/1     Completed   1          14m
kube-system   kube-apiserver-ip-                        1/1     Running     0          14m
kube-system   kube-controller-manager-ip-               1/1     Running     0          14m
kube-system   kube-proxy-ip-                            1/1     Running     0          14m
kube-system   kube-scheduler-ip-                        1/1     Running     0          14m
kube-system   rke2-canal-hv6qc                                        2/2     Running     0          14m
kube-system   rke2-coredns-rke2-coredns-7895c6f5d8-2x29g              1/1     Running     0          14m
kube-system   rke2-coredns-rke2-coredns-autoscaler-5868d76f68-7bdxv   1/1     Running     0          14m
kube-system   rke2-ingress-nginx-controller-ln6zm                     1/1     Running     0          13m
kube-system   rke2-metrics-server-85479b695c-fp264                    1/1     Running     0          14m
kube-system   rke2-snapshot-controller-696989ffdd-6t2fj               1/1     Running     0          14m





-----------------------------------------
INSTALL WITH NO ARGS OR FILES
 
sudo curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT=649521a5f6dba3f77b998988eec1439096cf18a6  INSTALL_RKE2_CHANNEL=testing  INSTALL_RKE2_TYPE=server sh - 



sudo mkdir -p /etc/rancher/rke2

sudo bash -c 'cat <<EOF>/etc/rancher/rke2/config.yaml
token:  test
write-kubeconfig-mode: 644
EOF'




$ sudo ls /var/lib/rancher/rke2/agent/etc/kubelet.conf.d
00-rke2-defaults.conf

Running kubelet --alsologtostderr=false --cloud-provider=external --config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d --containerd=/run/k3s/containerd/containerd.sock --hostname-override=ip- --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=172.31.27.227 --node-labels= --stderrthreshold=FATAL"

$ k get nodes,pods -A
NAME                    STATUS   ROLES                       AGE    VERSION
node/ip-   Ready    control-plane,etcd,master   100s   v1.32.1+rke2r1

NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE
kube-system   pod/cloud-controller-manager-ip-               1/1     Running     0          95s
kube-system   pod/etcd-ip-                                   1/1     Running     0          94s
kube-system   pod/helm-install-rke2-canal-7xq6w                           0/1     Completed   0          94s
kube-system   pod/helm-install-rke2-coredns-nnmpr                         0/1     Completed   0          94s
kube-system   pod/helm-install-rke2-ingress-nginx-47sh5                   0/1     Completed   0          94s
kube-system   pod/helm-install-rke2-metrics-server-xjwzt                  0/1     Completed   0          94s
kube-system   pod/helm-install-rke2-runtimeclasses-z8kgw                  0/1     Completed   0          94s
kube-system   pod/helm-install-rke2-snapshot-controller-crd-fblj6         0/1     Completed   0          94s
kube-system   pod/helm-install-rke2-snapshot-controller-f8c6f             0/1     Completed   0          94s
kube-system   pod/kube-apiserver-ip-                         1/1     Running     0          95s
kube-system   pod/kube-controller-manager-ip-                1/1     Running     0          95s
kube-system   pod/kube-proxy-ip-                             1/1     Running     0          95s
kube-system   pod/kube-scheduler-ip-                         1/1     Running     0          95s
kube-system   pod/rke2-canal-w2x47                                        2/2     Running     0          74s
kube-system   pod/rke2-coredns-rke2-coredns-7895c6f5d8-9v26h              1/1     Running     0          75s
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-5868d76f68-5vt7j   1/1     Running     0          75s
kube-system   pod/rke2-ingress-nginx-controller-wh8dj                     0/1     Running     0          29s
kube-system   pod/rke2-metrics-server-85479b695c-mc86x                    1/1     Running     0          40s
kube-system   pod/rke2-snapshot-controller-696989ffdd-zztr6               1/1     Running     0          39s



---------------------------------------------
INSTALL WITH 1 kubelet config arg and file
 

sudo curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT=649521a5f6dba3f77b998988eec1439096cf18a6  INSTALL_RKE2_CHANNEL=testing  INSTALL_RKE2_TYPE=server sh - 


 

sudo mkdir -p /etc/rancher/rke2

sudo mkdir -p /etc/rancher/rke2/kubelet.conf.d

sudo bash -c 'cat <<EOF>/etc/rancher/rke2/config.yaml
token:  test
kubelet-arg:
  - config=/etc/rancher/rke2/kubelet.conf 
  - image-gc-high-threshold=100
write-kubeconfig-mode: 644
EOF'



~$ sudo ls /var/lib/rancher/rke2/agent/etc/kubelet.conf.d
00-rke2-defaults.conf  10-cli-config.conf


~$ sudo cat /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/00-rke2-defaults.conf
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /var/lib/rancher/rke2/agent/client-ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
clusterDNS:
- 10.43.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: unix:///run/k3s/containerd/containerd.sock
cpuManagerReconcilePeriod: 10s
crashLoopBackOff: {}
evictionHard:
  imagefs.available: 5%
  nodefs.available: 5%
evictionMinimumReclaim:
  imagefs.available: 10%
  nodefs.available: 10%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: false
fileCheckFrequency: 20s
healthzBindAddress: 127.0.0.1
httpCheckFrequency: 20s
imageMaximumGCAge: 0s
imageMinimumGCAge: 2m0s
kind: KubeletConfiguration
logging:
  flushFrequency: 5s
  format: text
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 5m0s
nodeStatusUpdateFrequency: 10s
resolvConf: /etc/resolv.conf
runtimeRequestTimeout: 2m0s
serializeImagePulls: false
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /var/lib/rancher/rke2/agent/pod-manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
tlsCertFile: /var/lib/rancher/rke2/agent/serving-kubelet.crt
tlsPrivateKeyFile: /var/lib/rancher/rke2/agent/serving-kubelet.key
volumeStatsAggPeriod: 1m0s



$ $ sudo journalctl -u rke2-server | grep kubelet 


"Running kubelet --alsologtostderr=false --cloud-provider=external --config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d --containerd=/run/k3s/containerd/containerd.sock --hostname-override=ip-172-31-27-227 --image-gc-high-threshold=100 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=172.31.27.227 --node-labels= --stderrthreshold=FATAL"


~$ k get nodes,pods -A
NAME                    STATUS   ROLES                       AGE    VERSION
node/ip- Ready    control-plane,etcd,master   117s   v1.32.1+rke2r1

NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE
kube-system   pod/cloud-controller-manager-ip-             1/1     Running     0          97s
kube-system   pod/etcd-ip-                                 1/1     Running     0          97s
kube-system   pod/helm-install-rke2-canal-68jlj                           0/1     Completed   0          108s
kube-system   pod/helm-install-rke2-coredns-d6h6b                         0/1     Completed   0          108s
kube-system   pod/helm-install-rke2-ingress-nginx-glc88                   0/1     Completed   0          108s
kube-system   pod/helm-install-rke2-metrics-server-xc855                  0/1     Completed   0          108s
kube-system   pod/helm-install-rke2-runtimeclasses-wv7mq                  0/1     Completed   0          108s
kube-system   pod/helm-install-rke2-snapshot-controller-crd-6jdkl         0/1     Completed   0          108s
kube-system   pod/helm-install-rke2-snapshot-controller-sqrkh             0/1     Completed   0          108s
kube-system   pod/kube-apiserver-ip-                       1/1     Running     0          97s
kube-system   pod/kube-controller-manager-ip-              1/1     Running     0          97s
kube-system   pod/kube-proxy-ip-                           1/1     Running     0          97s
kube-system   pod/kube-scheduler-ip-                       1/1     Running     0          97s
kube-system   pod/rke2-canal-s694j                                        2/2     Running     0          89s
kube-system   pod/rke2-coredns-rke2-coredns-7895c6f5d8-p4tx5              1/1     Running     0          91s
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-5868d76f68-ws9mr   1/1     Running     0          91s
kube-system   pod/rke2-ingress-nginx-controller-krz4k                     1/1     Running     0          42s
kube-system   pod/rke2-metrics-server-85479b695c-lk9sl                    1/1     Running     0          63s
kube-system   pod/rke2-snapshot-controller-696989ffdd-bgvpb               1/1     Running     0          53s

brandond mentioned this issue Aug 25, 2023

Starting rke2-server brings up kubelet with flags instead of config file #342

Closed

caroline-suse-rancher added this to the Backlog milestone Nov 27, 2023

brandond modified the milestones: Backlog, v1.32.0+rke2r1 Sep 24, 2024

brandond self-assigned this Sep 24, 2024

caroline-suse-rancher modified the milestones: v1.32.0+rke2r1, v1.32.1+rke2r1 Jan 13, 2025

brandond mentioned this issue Jan 13, 2025

Bump K3s version for release-1.32 #7445

Merged

fmoral2 self-assigned this Jan 14, 2025

fmoral2 closed this as completed Jan 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support KubeletConfiguration file #4043

Support KubeletConfiguration file #4043

ebauman commented Mar 20, 2023

brandond commented Mar 20, 2023

brandond commented Sep 24, 2024 •

edited

Loading

caroline-suse-rancher commented Jan 13, 2025

fmoral2 commented Jan 22, 2025

Validation Results:

Support KubeletConfiguration file #4043

Support KubeletConfiguration file #4043

Comments

ebauman commented Mar 20, 2023

brandond commented Mar 20, 2023

brandond commented Sep 24, 2024 • edited Loading

caroline-suse-rancher commented Jan 13, 2025

fmoral2 commented Jan 22, 2025

Validated on Version:

Environment Details

Steps to validate the fix

Reproduction Issue:

Validation Results:

brandond commented Sep 24, 2024 •

edited

Loading