Nodes are not getting upgraded to the latest version using System-upgrade-controller Automatic upgrade

[MANUDEV-H-MENON]

I have a cluster which has a master and worker node. I would like to upgrade node version to latest.
System-upgrade-controller approach as mentioned in the RKE2 doc was followed.
https://docs.rke2.io/upgrade/automated_upgrade
Steps followed were,
Install the system-upgrade-controller
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.2/system-upgrade-controller.yaml

And Configuring plan for server and agent

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
  labels:
    rke2-upgrade: server
spec:
  concurrency: 1
  nodeSelector:
    matchExpressions:
       - {key: rke2-upgrade, operator: Exists}
       - {key: rke2-upgrade, operator: NotIn, values: ["disabled", "false"]}
       # When using k8s version 1.19 or older, swap control-plane with master
       - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
  serviceAccountName: system-upgrade
  cordon: true
#  drain:
#    force: true
  upgrade:
    image: rancher/rke2-upgrade
  version: (specific version 28.9)
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
  labels:
    rke2-upgrade: agent
spec:
  concurrency: 2
  nodeSelector:
    matchExpressions:
      - {key: rke2-upgrade, operator: Exists}
      - {key: rke2-upgrade, operator: NotIn, values: ["disabled", "false"]}
      # When using k8s version 1.19 or older, swap control-plane with master
      - {key: node-role.kubernetes.io/control-plane, operator: NotIn, values: ["true"]}
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/rke2-upgrade
  serviceAccountName: system-upgrade
  cordon: true
  drain:
    force: true
  upgrade:
    image: rancher/rke2-upgrade
  version: (specific version 28.9)

---

Deployment and pods were created but the jobs are not executed. On taking the logs of the pod, following is the observation:
kubectl logs pod/system-upgrade-controller-56d9f7c4f5-7r29q -n system-upgrade

W0527 05:01:14.502769 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2024-05-27T05:01:15Z" level=info msg="Applying CRD plans.upgrade.cattle.io"
E0527 05:01:16.492868 1 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:21.514377 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:21.516154 1 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:26.518966 1 memcache.go:104] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
time="2024-05-27T05:01:26Z" level=info msg="Starting /v1, Kind=Node controller"
time="2024-05-27T05:01:26Z" level=info msg="Starting /v1, Kind=Secret controller"
E0527 05:01:26.565429 1 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:31.551195 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:31.554305 1 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:36.567858 1 memcache.go:104] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
time="2024-05-27T05:01:36Z" level=info msg="Starting batch/v1, Kind=Job controller"
E0527 05:01:36.644965 1 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:41.624581 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:46.629124 1 memcache.go:104] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0527 05:01:46.631218 1 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
time="2024-05-27T05:01:46Z" level=info msg="Starting upgrade.cattle.io/v1, Kind=Plan controller"

[brandond]

Why do your plans say version: (specific version 28.9)? What is the actual value you're using there? If you have a version set instead of a channel, it won't upgrade to latest in a channel, it'll just upgrade to that version. Which unless you've redacted this for some odd reason, doesn't even appear to be a valid version.

[MANUDEV-H-MENON]

v1.28.9+rke2r1 This is the latest version i had specified.

[brandond]

Did you label your nodes with the rke2-upgrade label required by the node selector you're using in your plan spec?

[MANUDEV-H-MENON]

no. i haven't labelled with rke2-upgrade label. But should it be a concern since nodes are not namespaced resources ?

[brandond]

What does namespacing have to do with anything? You have a label selector that only matches nodes with that label. If you don't add that label to the nodes, with a value other than disabled or false, they won't get upgraded

[brandond]

This is covered in the page you linked above:

3. The server-plan targets server nodes by specifying a label selector that selects nodes with the node-role.kubernetes.io/control-plane label (node-role.kubernetes.io/master for 1.19 or older). The agent-plan targets agent nodes by specifying a label selector that select nodes without that label. Optionally, additional labels can be included, like in the example above, which requires label "rke2-upgrade" to exist and not have the value "disabled" or "false".

You might give the rest of the page a complete read, in case you didn't make it that far.

[MANUDEV-H-MENON]

Node versions were upgraded but one windows worker worker node with kubetes version v1.29.5 was not upgraded to latest rke2 version provided in the agent stating image pull issue. Other agents were upgraded. Thanks for the help on above issue. labeling part was missed out.