[Release 1.29] whereabouts helm chart issue

[tardich]

Environmental Info:
RKE2 Version:
rke2 version v1.29.2+rke2r1 (08699df)
go version go1.21.7 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
Linux dietpi7 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

Cluster Configuration:
3 master nodes in HA mode (provided by kube-vip)

Describe the bug:
Installing rke2 v1.29.2+rke2r1 with multus and whereabouts (configured with HelmChartConfig such as):

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-multus
  namespace: kube-system
spec:
  valuesContent: |-
    rke2-whereabouts:
      enabled: true
      nodeSelector:
        kubernetes.io/arch: arm64
    labels:
      nodeSelector:
        kubernetes.io/arch: arm64

After installation, I was able to see that there are missing CRD's, from whereabouts pods logs:

Failed to watch *v1alpha1.IPPool: failed to list *v1alpha1.IPPool: the server could not find the requested resource (get ippools.whereabouts.cni.cncf.io)

From past whereabouts experience, I know that it is also supposed to have another CRD (overlappingrangeipreservations.whereabouts.cni.cncf.io) but not installed either:

kubectl get crd | grep whereabouts

which shows nothing

On the node logs, I can see that there seem to have some errors in the whereabouts helm chart:

msg="Failed to process config: failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/Chart.yaml: Addon.k3s.cattle.io \"Chart\" is invalid: metadata.name: Invalid value: \"Chart\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/Chart.yaml: Addon.k3s.cattle.io \"Chart\" is invalid: metadata.name: Invalid value: \"Chart\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/cluster_role.yaml: Addon.k3s.cattle.io \"cluster_role\" is invalid: metadata.name: Invalid value: \"cluster_role\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/cluster_role_binding.yaml: Addon.k3s.cattle.io \"cluster_role_binding\" is invalid: metadata.name: Invalid value: \"cluster_role_binding\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/daemonset.yaml: yaml: line 6: could not find expected ':', failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/serviceaccount.yaml: invalid character '{' looking for beginning of object key string, failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/values.yaml: Object 'Kind' is missing in '{\"affinity\":{},\"fullnameOverride\":\"\",\"image\":{\"pullPolicy\":\"IfNotPresent\",\"repository\":\"rancher/hardened-whereabouts\",\"tag\":\"v0.6.2-build20231009\"},\"imagePullSecrets\":[],\"nameOverride\":\"\",\"namespaceOverride\":\"kube-system\",\"nodeSelector\":{\"kubernetes.io/arch\":\"amd64\"},\"podAnnotations\":{},\"podSecurityContext\":{},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"100Mi\"}},\"securityContext\":{\"privileged\":true},\"serviceAccount\":{\"annotations\":{},\"create\":true},\"successfulJobsHistoryLimit\":0,\"tolerations\":[{\"effect\":\"NoSchedule\",\"operator\":\"Exists\"}],\"updateStrategy\":\"RollingUpdate\"}', failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/clusterRole.yaml: Addon.k3s.cattle.io \"clusterRole\" is invalid: metadata.name: Invalid value: \"clusterRole\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/clusterRoleBinding.yaml: Addon.k3s.cattle.io \"clusterRoleBinding\" is invalid: metadata.name: Invalid value: \"clusterRoleBinding\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/configMap.yaml: Addon.k3s.cattle.io \"configMap\" is invalid: metadata.name: Invalid value: \"configMap\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/customResourceDefinition.yaml: Addon.k3s.cattle.io \"customResourceDefinition\" is invalid: metadata.name: Invalid value: \"customResourceDefinition\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/daemonSet.yaml: Addon.k3s.cattle.io \"daemonSet\" is invalid: metadata.name: Invalid value: \"daemonSet\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/serviceAccount.yaml: Addon.k3s.cattle.io \"serviceAccount\" is invalid: metadata.name: Invalid value: \"serviceAccount\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?([\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*](file:///)'), failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/values.yaml: Object 'Kind' is missing in '{\"cniplugins\":{\"image\":{\"repository\":\"rancher/hardened-cni-plugins\",\"tag\":\"v1.2.0-build20231009\"},\"skipcnis\":\"flannel\"},\"config\":{\"cni_conf\":{\"multusConfFile\":\"auto\"}},\"global\":{\"systemDefaultRegistry\":\"\"},\"image\":{\"pullPolicy\":\"IfNotPresent\",\"repository\":\"rancher/hardened-multus-cni\",\"tag\":\"v4.0.2-build20231009\"},\"labels\":{\"nodeSelector\":{\"kubernetes.io/arch\":\"amd64\",\"kubernetes.io/os\":\"linux\"}},\"manifests\":{\"clusterRole\":true,\"clusterRoleBinding\":true,\"configMap\":false,\"customResourceDefinition\":true,\"daemonSet\":true,\"serviceAccount\":true},\"pod\":{\"resources\":{\"enabled\":false,\"multus\":{\"limits\":{\"cpu\":\"2000m\",\"memory\":\"1024Mi\"},\"requests\":{\"cpu\":\"250m\",\"memory\":\"128Mi\"}}}},\"rke2-whereabouts\":{\"enabled\":false},\"serviceAccount\":{\"name\":\"multus\"}}'"

Steps To Reproduce:

• Installed RKE2:
  All defined above

Expected behavior:
Expecting to have whereabouts CRD's installed

Actual behavior:
Whereabouts CRD's are absent

Additional context / logs:

[tardich]

I also referenced this issue in #5538

[brandond]

failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/Chart.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/Chart.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/cluster_role.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/cluster_role_binding.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/daemonset.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/templates/serviceaccount.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/charts/rke2-whereabouts/values.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/clusterRole.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/clusterRoleBinding.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/configMap.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/customResourceDefinition.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/daemonSet.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/templates/serviceAccount.yaml:
failed to process /var/lib/rancher/rke2/server/manifests/rke2-multus/values.yaml:

It looks like you just copied the whole rke2-multus chart into /var/lib/rancher/rke2/server/manifests/ - don't do that! Charts are not valid Kubernetes manifests, they are helm charts!

[tardich]

Oh! Didn't notice. I was probably trying to figure out the chart, and obviously was in the wrong directory when doing so. So I removed it, removed the dependency to have something clear, then added the dependency back. The issue regarding the CRD's haven't cleared out. They are still unavailable after a clean reinstall of whereabouts.

So the issue is still valid and shouldn't be closed.

[brandond]

They appear to be properly installed for me after enabling the subchart:

brandond@dev01:~$ kubectl get helmchartconfig -n kube-system rke2-multus -o yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"helm.cattle.io/v1","kind":"HelmChartConfig","metadata":{"annotations":{},"name":"rke2-multus","namespace":"kube-system"},"spec":{"valuesContent":"rke2-whereabouts:\n  enabled: true\n  nodeSelector:\n    kubernetes.io/arch: arm64\nlabels:\n  nodeSelector:\n    kubernetes.io/arch: arm64"}}
  creationTimestamp: "2024-03-08T18:32:34Z"
  generation: 1
  name: rke2-multus
  namespace: kube-system
  resourceVersion: "275"
  uid: fd652e3d-047b-4d60-98cc-eacb406c6636
spec:
  valuesContent: |-
    rke2-whereabouts:
      enabled: true
      nodeSelector:
        kubernetes.io/arch: arm64
    labels:
      nodeSelector:
        kubernetes.io/arch: arm64

brandond@dev01:~/$ kubectl api-resources --api-group=whereabouts.cni.cncf.io
NAME                             SHORTNAMES   APIVERSION                         NAMESPACED   KIND
ippools                                       whereabouts.cni.cncf.io/v1alpha1   true         IPPool
overlappingrangeipreservations                whereabouts.cni.cncf.io/v1alpha1   true         OverlappingRangeIPReservation

What do you see in the helm job pod for the rke2-multus chart?

[tardich]

if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
        echo "KUBERNETES_SERVICE_HOST is using IPv6"
        CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
        CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ true != \t\r\u\e ]]
+ [[ '' == \1 ]]
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/rke2-multus.tgz.base64
+ CHART_PATH=/tmp/rke2-multus.tgz
+ [[ ! -f /chart/rke2-multus.tgz.base64 ]]
+ base64 -d /chart/rke2-multus.tgz.base64
+ CHART=/tmp/rke2-multus.tgz
+ set +e
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ [[ /tmp/rke2-multus.tgz == stable/* ]]
+ [[ -n '' ]]
+ helm_update install --set-string global.clusterCIDR=10.46.0.0/16 --set-string global.clusterCIDRv4=10.46.0.0/16 --set-string global.clusterDNS=10.47.0.10 --set-string global.clusterDomain=cluster.01.int.servinfo.stba --set-string global.rke2DataDir=/var/lib/rancher/rke2 --set-string global.serviceCIDR=10.47.0.0/16
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ helm_v3 ls --all -f '^rke2-multus$' --namespace kube-system --output json
++ jq -r '"\(.[0].app_version),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
+ LINE=4.0.2,deployed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-10_HelmChartConfig.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ 4.0.2 =~ ^(|null)$ ]]
+ [[ deployed =~ ^(pending-install|pending-upgrade|pending-rollback)$ ]]
Already installed rke2-multus
+ [[ deployed == \d\e\p\l\o\y\e\d ]]
+ echo 'Already installed rke2-multus'
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis rke2-multus --namespace kube-system
2024/03/08 18:50:27 Release 'rke2-multus' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2024/03/08 18:50:27 Get release 'rke2-multus' latest version.
2024/03/08 18:50:27 Check release 'rke2-multus' for deprecated or removed APIs...
2024/03/08 18:50:27 Finished checking release 'rke2-multus' for deprecated or removed APIs.
2024/03/08 18:50:27 Release 'rke2-multus' has no deprecated or removed APIs.
2024/03/08 18:50:27 Map of release 'rke2-multus' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading rke2-multus'
Upgrading rke2-multus
+ shift 1
+ helm_v3 upgrade --set-string global.clusterCIDR=10.46.0.0/16 --set-string global.clusterCIDRv4=10.46.0.0/16 --set-string global.clusterDNS=10.47.0.10 --set-string global.clusterDomain=cluster.01.int.servinfo.stba --set-string global.rke2DataDir=/var/lib/rancher/rke2 --set-string global.serviceCIDR=10.47.0.0/16 rke2-multus /tmp/rke2-multus.tgz --values /config/values-10_HelmChartConfig.yaml
Release "rke2-multus" has been upgraded. Happy Helming!
NAME: rke2-multus
LAST DEPLOYED: Fri Mar  8 18:50:27 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 22
TEST SUITE: None
NOTES:
======
1. The following components have been deployed as part of this helm chart:
Cluster Role: multus
Cluster Role Binding: rke2-multus
Custom Resource Definition: network-attachment-definitions.k8s.cni.cncf.io
Daemon Set: rke2-multus
Service Account: multus

You can now deploy any other CNI and create its Network Attachment Defintion.
---------

2. To uninstall helm chart use the command:
helm delete rke2-multus

You may have to manually delete CRD -
kubectl delete crd network-attachment-definitions.k8s.cni.cncf.io
---------
+ exit

Looking at api-resources, I don<t see anything related to whereabouts. But I do see whereabouts pods:

rke2-multus-rke2-whereabouts-9dp2f                      1/1     Running     0                2m6s
rke2-multus-rke2-whereabouts-gzks2                      1/1     Running     0                2m6s
rke2-multus-rke2-whereabouts-x7xbz                      1/1     Running     0                2m6s

[tardich]

I just manually installed the CRD's from the chart (that I downloaded elsewhere) and now they stick to the cluster and whereabouts is now working properly. But I'd like these for sure to install from the chart.....

[brandond]

How long has this cluster been deployed with multus? Helm has some odd behavior around CRDs, if you've been running it for a bit, its possible they didn't get installed by helm when we added them to the chart?

[tardich]

It has been deployed with multus from its beginning, so more or less two years ago now. Whereabouts, on the opposite, has just been recently added as a dependency to multus on my cluster. Previously, it was installed as a manual process.

Something else I just noticed. whereabouts has an ip-reconciler job available to do some cleanup in the ip addresses allocation after a crash, for example: https://github.com/k8snetworkplumbingwg/whereabouts/blob/master/doc/extended-configuration.md

It would be interesting to have it available as an option in the whereabouts dependency install

[brandond]

Whereabouts, on the opposite, has just been recently added as a dependency to multus on my cluster. Previously, it was installed as a manual process.

That's probably where the issue came from. I don't think this is an issue if you start without whereabouts and just enable the subchart in our multus chart.

[brandond]

cc @manuelbuil @mgfritch for thoughts

[mat1010]

I have the same issue. It seems like that CRDs are not installable through the subchart and therefore wont / can't be automatically deployed on chart installation. Does this make sense? It might be nice, from a user experience perspective, to have them installable through the rke2 config.