something wrong when apply example pod in k8s v1.25.x

[tojoyccnu]

MOOSE-CSI-PLUGIN

Test cluster servers:

k8s87	192.168.2.87	centos7.9-2009
k8s89	192.168.2.89	centos7.9-2009
k8s90	192.168.2.90	centos7.9-2009

k8s version : v1.25.3 (latest version)
moosefs (standalone): maintained in k8s90, v 3.0.116

1. First step:

download files , edited the kustomization.yaml file , change MFS_SERVER option to my moosefs master ip 192.168.2.90 , make direcotory /kubernetes in moosefs for k8s

2. Next step:

install moosefs-csi plugin

pod/csi-moosefs-controll pod/csi-moosefs-node are running

(base) [root@k8s87 kubernetes]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default moosfs-example-mount 0/1 ContainerCreating 0 42m
default nginx-0 1/1 Running 0 7d6h
default nginx-1 1/1 Running 0 7d6h
kube-flannel kube-flannel-ds-2k8zn 1/1 Running 0 25d
kube-flannel kube-flannel-ds-n55l6 1/1 Running 0 25d
kube-flannel kube-flannel-ds-tccqw 1/1 Running 0 26d
kube-system coredns-c676cc86f-ghxlx 1/1 Running 0 20d
kube-system coredns-c676cc86f-pcdc6 1/1 Running 0 20d
kube-system csi-moosefs-controller-0 4/4 Running 0 45m
kube-system csi-moosefs-node-b4x7d 2/2 Running 0 45m
kube-system csi-moosefs-node-mhg8r 2/2 Running 0 45m
kube-system etcd-k8s87 1/1 Running 0 26d
......

3. Next Step:

apply examples (pv ,pvc pod)

kubectl apply -f moosefs-csi-example-pv.yaml
kubectl apply -f moosefs-csi-example-pvc.yaml

(base) [root@k8s87 examples]# kubectl get pv,pvc -n default
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/moosefs-csi-pv 1Gi RWO Retain Bound default/moosefs-csi-pvc moosefs-file-storage 48m
persistentvolume/pvc-a584409f-d77a-42d6-9870-5768e3844666 1Gi RWX Delete Bound nginx-www/nginx-web-pvc ufs-file-storage 3d6h

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/moosefs-csi-pvc Bound moosefs-csi-pv 1Gi RWO moosefs-file-storage 48m

(base) [root@k8s87 kubernetes]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default moosfs-example-mount 0/1 ContainerCreating 0 42m
......

apply example pods something wrong

kubectl apply -f moosefs-csi-example-pod.yaml

(base) [root@k8s87 kubernetes]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default moosfs-example-mount 0/1 ContainerCreating 0 42m
......

(base) [root@k8s87 examples]# kubectl describe pod moosfs-example-mount -n default
Name: moosfs-example-mount
Namespace: default
Priority: 0
Service Account: default
Node: k8s90/172.16.120.90
Start Time: Mon, 07 Nov 2022 15:30:15 +0800
Labels:
Annotations:
Status: Pending
IP:
IPs:
Containers:
busybox:
Container ID:
Image: busybox
Image ID:
Port:
Host Port:
Command:
sleep
1000000
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/data from moosefs-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rw8bs (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
moosefs-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: moosefs-csi-pvc
ReadOnly: false
kube-api-access-rw8bs:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

------

Normal Scheduled 52m default-scheduler Successfully assigned default/moosfs-example-mount to k8s90
Warning FailedMount 32m (x3 over 50m) kubelet Unable to attach or mount volumes: unmounted volumes=[moosefs-volume], unattached volumes=[kube-api-access-rw8bs moosefs-volume]: timed out waiting for the condition
Warning FailedAttachVolume 4m1s (x16 over 50m) attachdetach-controller AttachVolume.Attach failed for volume "moosefs-csi-pv" : timed out waiting for external-attacher of csi.kunde21.moosefs CSI driver to attach volume moosefs-csi-pv
Warning FailedMount 23s (x20 over 48m) kubelet Unable to attach or mount volumes: unmounted volumes=[moosefs-volume], unattached volumes=[moosefs-volume kube-api-access-rw8bs]: timed out waiting for the condition

mount client like this:

(base) [root@k8s90 kubernetes]# mfscli -SMS -s "|"
active mounts, parameters:,8,k8s89,/opt/mfs/kubernetes/kubernetes,0,1,3.0.115,/kubernetes,rw,yes,no,yes,0,0,-,-,-,-,-,-,000,00000000
active mounts, parameters:,1,k8s90,/mnt/testmount,2,1,3.0.116,/,rw,yes,no,yes,0,0,-,-,-,-,-,-,000,00000000

inactive mounts, parameters:,3,k8s89,/opt/mfs/kubernetes/kubernetes,0,77446
inactive mounts, parameters:,4,k8s89,/opt/mfs/kubernetes/kubernetes,0,77477
inactive mounts, parameters:,5,k8s89,/opt/mfs/kubernetes/kubernetes,0,80662
inactive mounts, parameters:,6,k8s89,/opt/mfs/kubernetes/kubernetes,0,80747
inactive mounts, parameters:,7,k8s89,/opt/mfs/kubernetes/kubernetes,0,82765

4. Questions

Question 1: moose-csi-driverinfo.yaml errors in k8s 1.25.x

something wrong when invoke "kustomize build | kubectl apply -f -".

error: resource mapping not found for name: "csi.kunde21.moosefs" namespace: "" from "STDIN": no matches for kind "CSIDriver" in version "storage.k8s.io/v1beta1"
resolution: edit moosefs-csi-driverinfo.yaml line 2, change to to storage.k8s.io/v1. It's easy

Question 2: moose-exmample-pod.yaml Attach PV(PVC) failed

May I miss steps or something else to success?

[tojoyccnu]

Problem 1:
[/var/log/message] has logs in blew

Nov 9 16:28:59 k8s90 kubelet: E1109 16:28:59.212598 9170 nodeinfomanager.go:561] Invalid attach limit value 0 cannot be added to CSINode object for

FIX : Set MaxVolumesPerNode on NodeGetInfo call based on Node Type after k8s 1.17+.
kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#19.

Now this problem is done.

Problem 2:
I still get an error;
AttachVolume.Attach failed for volume "moosefs-csi-pv" : timed out waiting for external-attacher of csi.ufs CSI driver to attach volume moosefs-csi-pv
Unable to attach or mount volumes: unmounted volumes=[moosefs-volume], unattached volumes=[kube-api-access-k4bkt moosefs-volume]: timed out waiting for the condition

 It seem to be the same infomation of  https://github.com/ctrox/csi-s3/issues/77