Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MountVolume.MountDevice failed for volume "pvc--..." : rpc error: code = Internal desc = VolumeDeviceNotFound #150

Open
Diegunio opened this issue Dec 20, 2024 · 0 comments
Open

MountVolume.MountDevice failed for volume "pvc--..." : rpc error: code = Internal desc = VolumeDeviceNotFound #150

Diegunio opened this issue Dec 20, 2024 · 0 comments

Comments

@Diegunio
Copy link

Diegunio commented Dec 20, 2024 •
edited
Loading

I've deployed helm chart and set up all required configuration for OceanStor Dorado V6

Then I succsessfully created backend with such config providing credentials for vstore user having admin right and rest access

storage: "oceanstor-san"
name: "dorado"
namespace: "huawei-csi"
urls:
  - "https://192.168.1.21:8088"
  - "https://192.168.1.20:8088"
pools:
  - "Storage"
parameters:
  protocol: "iscsi"
  portals:
    - "172.16.1.202"
    - "172.16.1.201"
maxClientThreads: "30"
vstoreName: "vStore"

Then I created matching StorageClass for this driver

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: huawei-csi
provisioner: csi.huawei.com
parameters:
  allocType: thin
  backend: dorado
  fsType: xfs
  volumeType: lun
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

Then I attempt to deploy sample application having storageclass used, unformately pod is stuck on ContainerCreating

MountVolume.MountDevice failed for volume "pvc--..." : rpc error: code = Internal desc = VolumeDeviceNotFound

I investigated the node that pod is about to be created on, service icsci is inactive by it's own and I need to restart it, iccsid service works well. I checked the session by iscsiadm -m session and it's connected.

New disk spawned

Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2516F31B-AB89-499B-G5DW-76A3AFDA124D

Device       Start       End   Sectors  Size Type
/dev/vda1     2048      4095      2048    1M BIOS boot
/dev/vda2     4096    264191    260096  127M EFI System
/dev/vda3   264192   1050623    786432  384M Linux filesystem
/dev/vda4  1050624 209715166 208664543 99.5G Linux filesystem


Disk /dev/sda: 11 GiB, 11811160064 bytes, 23068672 sectors
Disk model: XSG1            
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdb: 11 GiB, 11811160064 bytes, 23068672 sectors
Disk model: XSG1            
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Journal logs look like this

Dec 20 13:24:38 worker-4.prod.example.com ovs-vswitchd[1456]: ovs|00489|connmgr|INFO|br-ex<->unix#2466: 4 flow_mods in the last 0 s (3 adds, 1 deletes)
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.046639    3604 kubelet.go:2456] "SyncLoop ADD" source="api" pods=["postgresql/postgresql-0"]
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.046701    3604 topology_manager.go:215] "Topology Admit Handler" podUID="beb1c30a-316f-41bf-96c9-8de6c5a62cb5" podNamespace="postgresql" podName="postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.047659    3604 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="postgresql/postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.049410    3604 reflector.go:359] Caches populated for *v1.Secret from object-"postgresql"/"harborcfg-live-pull"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.049841    3604 reflector.go:359] Caches populated for *v1.Secret from object-"postgresql"/"postgresql"
Dec 20 13:24:39 worker-4.prod.example.com systemd[1]: Created slice libcontainer container kubepods-burstable-podbeb1c30a_316f_41bf_96c9_8de6c5a62cb5.slice.
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.058317    3604 kubelet.go:2463] "SyncLoop UPDATE" source="api" pods=["postgresql/postgresql-0"]
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.196657    3604 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1c30a-316f-41bf-96c9-8de6c5a62cb5-dshm\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.196714    3604 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"empty-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1c30a-316f-41bf-96c9-8de6c5a62cb5-empty-dir\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.297774    3604 reconciler_common.go:220] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1c30a-316f-41bf-96c9-8de6c5a62cb5-dshm\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.297861    3604 reconciler_common.go:220] "operationExecutor.MountVolume started for volume \"empty-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1c30a-316f-41bf-96c9-8de6c5a62cb5-empty-dir\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.298283    3604 operation_generator.go:721] "MountVolume.SetUp succeeded for volume \"empty-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1c30a-316f-41bf-96c9-8de6c5a62cb5-empty-dir\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:39 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:39.300354    3604 operation_generator.go:721] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1c30a-316f-41bf-96c9-8de6c5a62cb5-dshm\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:47 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:47.963571    3604 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:47 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:47.969787    3604 operation_generator.go:1576] "Controller attach succeeded for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") device path: \"\"" pod="postgresql/postgresql-0"
Dec 20 13:24:48 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:48.064130    3604 reconciler_common.go:220] "operationExecutor.MountVolume started for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:24:48 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:48.064247    3604 operation_generator.go:622] "MountVolume.WaitForAttach entering for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") DevicePath \"\"" pod="postgresql/postgresql-0"
Dec 20 13:24:48 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:24:48.067154    3604 operation_generator.go:632] "MountVolume.WaitForAttach succeeded for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") DevicePath \"csi-b3f54cd9b8f7b296157042103a87dd87e66c6e57f831b03ae836f2ac6ff12533\"" pod="postgresql/postgresql-0"
Dec 20 13:24:48 worker-4.prod.example.com kernel: scsi 5:0:0:1: Direct-Access     HUAWEI   XSG1             6000 PQ: 0 ANSI: 6
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: Attached scsi generic sg1 type 0
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: [sda] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB)
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: [sda] Write Protect is off
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: [sda] Mode Sense: 97 00 00 08
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 13:24:48 worker-4.prod.example.com kernel: scsi 4:0:0:1: Direct-Access     HUAWEI   XSG1             6000 PQ: 0 ANSI: 6
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: Attached scsi generic sg2 type 0
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: [sdb] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB)
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: [sdb] Write Protect is off
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: [sdb] Mode Sense: 97 00 00 08
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 5:0:0:1: [sda] Attached SCSI disk
Dec 20 13:24:48 worker-4.prod.example.com kernel: sd 4:0:0:1: [sdb] Attached SCSI disk
Dec 20 13:24:53 worker-4.prod.example.com ovs-vswitchd[1456]: ovs|00490|connmgr|INFO|br-ex<->unix#2476: 2 flow_mods in the last 0 s (1 adds, 1 deletes)
Dec 20 13:24:55 worker-4.prod.example.com ovs-vswitchd[1456]: ovs|00491|connmgr|INFO|br-int<->unix#3: 30 flow_mods in the 1 s starting 17 s ago (30 adds)
Dec 20 13:25:06 worker-4.prod.example.com kubenswrapper[3604]: E1220 13:25:06.112683    3604 csi_attacher.go:366] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = VolumeDeviceNotFound
Dec 20 13:25:06 worker-4.prod.example.com kubenswrapper[3604]: E1220 13:25:06.112887    3604 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9 podName: nodeName:}" failed. No retries permitted until 2024-12-20 13:25:06.612865573 +0000 UTC m=+5460.554038269 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9" (UniqueName: "kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9") pod "postgresql-0" (UID: "beb1c30a-316f-41bf-96c9-8de6c5a62cb5") : rpc error: code = Internal desc = VolumeDeviceNotFound
Dec 20 13:25:06 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:06.173482    3604 kubelet_getters.go:218] "Pod status updated" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-worker-4.prod.example.com" status="Running"
Dec 20 13:25:06 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:06.621368    3604 reconciler_common.go:220] "operationExecutor.MountVolume started for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:25:06 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:06.621471    3604 operation_generator.go:622] "MountVolume.WaitForAttach entering for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") DevicePath \"\"" pod="postgresql/postgresql-0"
Dec 20 13:25:06 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:06.624313    3604 operation_generator.go:632] "MountVolume.WaitForAttach succeeded for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") DevicePath \"csi-b3f54cd9b8f7b296157042103a87dd87e66c6e57f831b03ae836f2ac6ff12533\"" pod="postgresql/postgresql-0"
Dec 20 13:25:06 worker-4.prod.example.com kernel: scsi 4:0:0:1: Direct-Access     HUAWEI   XSG1             6000 PQ: 0 ANSI: 6
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 4:0:0:1: Attached scsi generic sg1 type 0
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB)
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Write Protect is off
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Mode Sense: 97 00 00 08
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 13:25:06 worker-4.prod.example.com kernel: scsi 5:0:0:1: Direct-Access     HUAWEI   XSG1             6000 PQ: 0 ANSI: 6
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 5:0:0:1: Attached scsi generic sg2 type 0
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB)
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Write Protect is off
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Mode Sense: 97 00 00 08
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Attached SCSI disk
Dec 20 13:25:06 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Attached SCSI disk
Dec 20 13:25:08 worker-4.prod.example.com crio[3484]: time="2024-12-20 13:25:08.665385553Z" level=warning msg="Failed to get the status of process with PID 62974: open /proc/62974/stat: no such file or directory"
Dec 20 13:25:08 worker-4.prod.example.com ovs-vswitchd[1456]: ovs|00492|connmgr|INFO|br-ex<->unix#2480: 2 flow_mods in the last 0 s (1 adds, 1 deletes)
Dec 20 13:25:23 worker-4.prod.example.com ovs-vswitchd[1456]: ovs|00493|connmgr|INFO|br-ex<->unix#2489: 2 flow_mods in the last 0 s (1 adds, 1 deletes)
Dec 20 13:25:24 worker-4.prod.example.com kubenswrapper[3604]: E1220 13:25:24.734771    3604 csi_attacher.go:366] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = VolumeDeviceNotFound
Dec 20 13:25:24 worker-4.prod.example.com kubenswrapper[3604]: E1220 13:25:24.734933    3604 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9 podName: nodeName:}" failed. No retries permitted until 2024-12-20 13:25:25.734914646 +0000 UTC m=+5479.676087342 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9" (UniqueName: "kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9") pod "postgresql-0" (UID: "beb1c30a-316f-41bf-96c9-8de6c5a62cb5") : rpc error: code = Internal desc = VolumeDeviceNotFound
Dec 20 13:25:25 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:25.758031    3604 reconciler_common.go:220] "operationExecutor.MountVolume started for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") " pod="postgresql/postgresql-0"
Dec 20 13:25:25 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:25.758146    3604 operation_generator.go:622] "MountVolume.WaitForAttach entering for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") DevicePath \"\"" pod="postgresql/postgresql-0"
Dec 20 13:25:25 worker-4.prod.example.com kubenswrapper[3604]: I1220 13:25:25.761441    3604 operation_generator.go:632] "MountVolume.WaitForAttach succeeded for volume \"pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\" (UniqueName: \"kubernetes.io/csi/csi.huawei.com^dorado.pvc-bba649ee-4279-49ee-bba6-c8dd5b07f1d9\") pod \"postgresql-0\" (UID: \"beb1c30a-316f-41bf-96c9-8de6c5a62cb5\") DevicePath \"csi-b3f54cd9b8f7b296157042103a87dd87e66c6e57f831b03ae836f2ac6ff12533\"" pod="postgresql/postgresql-0"
Dec 20 13:25:25 worker-4.prod.example.com kernel: scsi 4:0:0:1: Direct-Access     HUAWEI   XSG1             6000 PQ: 0 ANSI: 6
Dec 20 13:25:25 worker-4.prod.example.com kernel: scsi 5:0:0:1: Direct-Access     HUAWEI   XSG1             6000 PQ: 0 ANSI: 6
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 4:0:0:1: Attached scsi generic sg1 type 0
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB)
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 5:0:0:1: Attached scsi generic sg2 type 0
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Write Protect is off
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Mode Sense: 97 00 00 08
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB)
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Write Protect is off
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Mode Sense: 97 00 00 08
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 4:0:0:1: [sda] Attached SCSI disk
Dec 20 13:25:25 worker-4.prod.example.com kernel: sd 5:0:0:1: [sdb] Attached SCSI disk
Dec 20 13:25:38 worker-4.prod.example.com ovs-vswitchd[1456]: ovs|00494|connmgr|INFO|br-ex<->unix#2493: 2 flow_mods in the last 0 s (1 adds, 1 deletes)
Dec 20 13:25:43 worker-4.prod.example.com kubenswrapper[3604]: E1220 13:25:43.881796    3604 csi_attacher.go:366] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = VolumeDeviceNotFound

The only intresting log i've found in controller pod

E1220 11:57:27.501718 1 sidecar_controller.go:226] syncContent content-7345e937-030b-4f81-ac84-b3fc9317b921 failed, error: Run sync content failed, error: Operation cannot be fulfilled on storagebackendcontents.xuanwu.huawei.io "content-7345e937-9fk4-4f81-ag32-b3fc9317b921": the object has been modified; please apply your changes to the latest version and try again.

in node pod

I1220 11:54:12.304148 7282 main.go:164] Version: v2.3.0
I1220 11:54:12.304185 7282 main.go:165] Running node-driver-registrar in mode=registration
I1220 11:54:13.307681 7282 node_register.go:52] Starting Registration Server at: /registration/csi.huawei.com-reg.sock
I1220 11:54:13.307854 7282 node_register.go:61] Registration Server started at: /registration/csi.huawei.com-reg.sock
I1220 11:54:13.307963 7282 node_register.go:91] Skipping healthz server because HTTP endpoint is set to: ""
I1220 11:54:15.023007 7282 main.go:100] Received GetInfo call: &InfoRequest{}
I1220 11:54:15.023492 7282 main.go:107] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/csi.huawei.com/registration"
I1220 11:54:15.060131 7282 main.go:118] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

Additionaly pod from daemonset huawei-csi-node where application pod should be located keep restarting, it's the only one restarting, controller pod also restarts

My environment is OKD 4.17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Diegunio