Ubuntu Canonical CSI errors

[saurabhwani5]

Describe the bug

On Ubuntu Canonical env unable to run CSI , operator pod is coming up and when CR is applied it is giving error Synchronization of node/driver interface failed.
Daemonset ibm-spectrum-scale-csi is not getting created.

How to Reproduce?

1. Create a ubuntu canonical cluster : charmed kubernetes
2. Install CSI 2.8.0/2.9.0 operator.yaml. In this I have used 2.8.0 and check operator pod.

root@ubuntu-canonical-master-2:~# kubectl get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.8.0     False

3. Apply CR.yaml and check the CSI pods

root@ubuntu-canonical-master-2:~/tools# kubectl get pods -o wide
NAME                                                  READY   STATUS             RESTARTS         AGE   IP                NODE            NOMINATED NODE   READINESS GATES
ibm-spectrum-scale-csi-attacher-5c5f879f9b-92vlc      1/1     Running            18 (5m35s ago)   40m   192.168.135.137   juju-b543bc-8   <none>           <none>
ibm-spectrum-scale-csi-attacher-5c5f879f9b-lmsk4      0/1     CrashLoopBackOff   17 (4m15s ago)   40m   192.168.44.204    juju-b543bc-7   <none>           <none>
ibm-spectrum-scale-csi-operator-5f645d654f-rn7kn      1/1     Running            0                41m   192.168.44.203    juju-b543bc-7   <none>           <none>
ibm-spectrum-scale-csi-provisioner-6956678877-jskj9   1/1     Running            18 (5m15s ago)   40m   192.168.135.138   juju-b543bc-8   <none>           <none>
ibm-spectrum-scale-csi-resizer-bfb8f6df-sjrls         0/1     CrashLoopBackOff   17 (4m15s ago)   40m   192.168.135.140   juju-b543bc-8   <none>           <none>
ibm-spectrum-scale-csi-snapshotter-6cf649bb46-j28q7   0/1     CrashLoopBackOff   17 (3m35s ago)   40m   192.168.135.139   juju-b543bc-8   <none>           <none>
root@ubuntu-canonical-master-2:~/tools#

4. Check the operator logs :
   2023-05-16T10:35:40.779Z ERROR csiscaleoperator_controller.Reconcile Synchronization of node/driver interface failed. {"error": "DaemonSet.apps \"ibm-spectrum-scale-csi\" is invalid: [spec.template.spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy, spec.template.spec.containers[1].securityContext.privileged: Forbidden: disallowed by cluster policy]"}

Expected behavior

All CSI pods should be in running state.

Data Collection and Debugging

CSI Snap : /u/DUMPS/scale-csi/D.975

Env :

Kubernetes Setup :

root@ubuntu-canonical-master-2:~# hostnamectl
 Static hostname: ubuntu-canonical-master-2.fyre.ibm.com
       Icon name: computer-vm
         Chassis: vm
      Machine ID: 5b1ff40addc649abb3cf3ef56d9721bc
         Boot ID: 9c26d87c2b5d4a1a98e25c0dd14b0965
  Virtualization: kvm
Operating System: Ubuntu 22.04.2 LTS
          Kernel: Linux 5.15.0-71-generic
    Architecture: x86-64
 Hardware Vendor: QEMU
  Hardware Model: Standard PC _i440FX + PIIX, 1996_
root@ubuntu-canonical-master-2:~#

root@ubuntu-canonical-master-2:~# kubectl get nodes -owide
NAME            STATUS   ROLES    AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
juju-b543bc-5   Ready    <none>   38d   v1.24.13   10.165.206.251   <none>        Ubuntu 20.04.6 LTS   5.15.0-71-generic   containerd://1.6.12
juju-b543bc-6   Ready    <none>   38d   v1.24.13   10.165.206.144   <none>        Ubuntu 20.04.6 LTS   5.15.0-71-generic   containerd://1.6.12
juju-b543bc-7   Ready    <none>   38d   v1.24.13   10.165.206.157   <none>        Ubuntu 20.04.6 LTS   5.15.0-71-generic   containerd://1.6.12
juju-b543bc-8   Ready    <none>   38d   v1.24.13   10.165.206.15    <none>        Ubuntu 20.04.6 LTS   5.15.0-71-generic   containerd://1.6.12
juju-b543bc-9   Ready    <none>   38d   v1.24.13   10.165.206.242   <none>        Ubuntu 20.04.6 LTS   5.15.0-71-generic   containerd://1.6.12
root@ubuntu-canonical-master-2:~#

root@ubuntu-canonical-master-2:~# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-15T02:14:53Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.13", GitCommit:"49433308be5b958856b6949df02b716e0a7cf0a3", GitTreeState:"clean", BuildDate:"2023-04-14T02:06:44Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

Storage Cluster :

[root@saurabh9-scalegui ~]# mmdiag
Current GPFS build: "5.1.7.0 ".
Built on Feb 20 2023 at 14:31:36
Running 36 days 18 hours 30 minutes 53 secs, pid 51367

[Jainbrt]

we are keeping this as sev3, as we do not support this env to our customer.