Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSI snapshot controller upgrade fails after upgrade to v1.29.6+rke2r1 #6413

Closed
shindebshekhar opened this issue Jul 25, 2024 · 3 comments
Closed
Assignees

Comments

@shindebshekhar
Copy link

Environmental Info:
RKE2 Version: v1.29.6+rke2r1

:~ # rke2 -v
rke2 version v1.29.6+rke2r1 (21ad86f)
go version go1.21.11 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

Linux cp001 5.3.18-150300.59.161-default #1 SMP Thu May 9 06:59:05 UTC 2024 (26d1938) x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 Master 3 Worker nodes

Describe the bug:

We are trying to upgrade from v1.28.8+rke2r1 to v1.29.6+rke2r1
During master upgrade we get see vsphere-csi-controller pod goes into ImagePullBackOff

:~> kubectl get po -A | grep -i vsphere-csi-controller
kube-system                vsphere-csi-controller-578cb9f685-9f22f            6/6     Running            0               142m
kube-system                vsphere-csi-controller-578cb9f685-wh79g          6/6     Running            0               142m
**kube-system                vsphere-csi-controller-75495fd5bb-tvnb2           6/7     ImagePullBackOff   0     7m50s**

On further investigating we found that one of the container is failing to pull image docker.io/rancher/mirrored-sig-storage-csi-snapshotter:v6.2.2"

In release notes of version v1.29.6+rke2r1 the Image version mentioned is docker.io/rancher/mirrored-sig-storage-csi-snapshotter:v6.2.1

Do we have to upload mirrored-sig-storage-csi-snapshotter:v6.2.2 Image and test?

@brandond
Copy link
Member

brandond commented Jul 25, 2024

https://github.com/rancher/rke2/blob/v1.29.6%2Brke2r1/scripts/build-images#L75 shows rancher/mirrored-sig-storage-csi-snapshotter:v6.2.1

https://github.com/rancher/rke2/blob/v1.29.6%2Brke2r1/charts/chart_versions.yaml#L32-L34 shows that we are using 3.1.2-rancher400 of the vsphere CSI chart.

https://github.com/rancher/rke2-charts/blob/main/charts/rancher-vsphere-csi/rancher-vsphere-csi/3.1.2-rancher400/values.yaml#L174-L176 shows that Kubernetes >= 1.26 <= 1.29 will use v6.2.2 of the snapshotter image - so yes, we've made a mistake in our image list.

I suspect this was missed as part of the bump from #5801

@brandond
Copy link
Member

brandond commented Jul 25, 2024

I suspect this has already been fixed by #6342 - please test with v1.29.7+rke2r1, once it is available.

@shindebshekhar
Copy link
Author

Thank you @brandond for your quick response. I have have uploaded new Image rancher/mirrored-sig-storage-csi-snapshotter:v6.2.2 and tested it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants