Documentation improvement in cluster restore.

[shan100github]

Document Reference Section
https://docs.rke2.io/backup_restore/#cluster-reset
https://docs.rke2.io/backup_restore/#restoring-a-snapshot-to-existing-nodes

Issue:
I had a situation to perform cluster-reset/restore snapshots to existing nodes.
I would like to include the following points as a part of the general FAQ's or under https://docs.rke2.io/backup_restore/#other-notes-on-restoring-a-snapshot

Contents to be included:

• Perform rke2-killall.sh, and start restoring snapshots to existing nodes.
• After rke2-killall.sh, to perform cluster-reset or restoring-a-snapshot-to-existing-nodes, if the existing cluster uses any external storage like (NFS, EBS, Longhorn, and so on), make sure to rm -rf /var/lib/kubelet/pods/* to avoid RKE2 Cluster reset failure with error in kubelet_volumes.go:225 #3315
• For longhorn users rm /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock or rm /var/lib/kubelet/plugins_registry/io.rancher.longhorn-reg.sock depends on longhorn version.
• For cilium users rm /var/run/cilium/cilium.sock

Errors noticed during connection reset log

E0915 22:50:29.228689  936749 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"165028ee-9313-4201-80d4-f2a1dac6dd02\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/165028ee-9313-4201-80d4-f2a1dac6dd02/volumes/kubernetes.io~empty-dir/tmp-dir: directory not empty" numErrs=36
E0915 22:51:31.222991  936749 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"2b593e51-dc77-4d47-955f-2ca862d52cfc\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/2b593e51-dc77-4d47-955f-2ca862d52cfc/volumes/kubernetes.io~configmap/longhorn-default-setting: directory not empty" numErrs=34
E0915 22:52:23.222304  936749 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"3f7819ff-9d84-4ade-977e-f6fd751d582\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/3f7819ff-9d84-4ade-977e-f6fd2751d582/volumes/kubernetes.io~configmap/values: directory not empty" numErrs=32
E0915 22:53:35.223691  936749 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"41daaeee-bdc0-4f46-971c-494acddf474e\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/41daaeee-bdc0-4f46-971c-494acddf474e/volume-subpaths/config/notary-server/0: not a directory" numErrs=29
E0915 23:06:00.553405  937594 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"41daaeee-bdc0-4f46-971c-494acddf474e\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/41daaeee-bdc0-4f46-971c-494acddf474e/volume-subpaths/signer-certificate/notary-server/2: not a directory" numErrs=28
E0915 23:08:18.553258  937594 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"4b6aa7e5-531f-4e50-b9b0-95d75d886245\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/4b6aa7e5-531f-4e50-b9b0-95d75d886245/volumes/kubernetes.io~configmap/registry-config: directory not empty" numErrs=26
E0915 23:12:48.240116  938190 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"6525b999-6ed3-40dd-87e6-674063559a95\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/6525b999-6ed3-40dd-87e6-674063559a95/volumes/kubernetes.io~configmap/istiod-ca-cert: directory not empty" numErrs=22
E0915 23:13:30.240629  938190 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"73f2c9d5-d777-4587-aa56-a3f6ed6fd417\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/73f2c9d5-d777-4587-aa56-a3f6ed6fd417/volumes/kubernetes.io~configmap/config: directory not empty" numErrs=20
E0915 23:14:18.240335  938190 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e1252ab-8905-445c-936c-822cc9230803\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/8e1252ab-8905-445c-936c-822cc9230803/volume-subpaths/config/notary-signer/0: not a directory" numErrs=18
E0915 23:14:50.239579  938190 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"9660396a-0cad-44aa-b861-9d44a98ca2a2\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/9660396a-0cad-44aa-b861-9d44a98ca2a2/volumes/kubernetes.io~configmap/values: directory not empty" numErrs=15
E0915 23:15:26.239559  938190 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"abc82a81-50df-42fd-9c89-e18da3a9db9c\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/abc82a81-50df-42fd-9c89-e18da3a9db9c/volumes/kubernetes.io~configmap/config-volume: directory not empty" numErrs=12
E0915 23:16:12.240639  938190 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"c905a625-9ee9-46d6-b0bb-012f8e1895c4\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/c905a625-9ee9-46d6-b0bb-012f8e1895c4/volumes/kubernetes.io~configmap/cilium-config-path: directory not empty" numErrs=10

[kkmaslowski]

@shan100github I'm using vsphere-csi. I had to manually remove /var/lib/kubelet/plugins_registry/csi.vsphere.vmware.com-reg.sock. Please add it to manual.
My issue: #3541

[dereknola]

Looking at the list, most of these issues have been fixed by issues such as #3544. Transfering this issue to docs with focus on the following:

• Note that the Cilium Socket may need to be deleted.