Occasional volume attach failure

[brianbuquoi]

The initial StatefulSet will create and start running successfully (on apollo [node 1]), then the subsequent ones will fail as follows (on starbuck [node 2]):

Warning  FailedAttachVolume  26s                attachdetach-controller  Multi-Attach error for volume "pvc-e4056d51-0609-4140-8f87-ceb4c660895e" Volume is already exclusively attached to one node and can't be attached to another

and if I cordon the node and put them on the same node I get the following:

Warning  FailedMount  1s (x5 over 9s)  kubelet, apollo    MountVolume.WaitForAttach failed for volume "pvc-e4056d51-0609-4140-8f87-ceb4c660895e" : volume attachment is being deleted

If I scale the StatefulSet to 0 and delete the PVC and then rescale to 1 and let it recreate the PVC and deploy to the same as the original node (apollo) I get the following:

Warning  FailedMount       19s                    kubelet, apollo    Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data config default-token-ws2vb]: timed out waiting for the condition

This is also starting to occur on regular deployments as well as of k8s 1.17.

[brianbuquoi]

At this point, on 1.17 nothing new will attach, but it isn't consistent. If I delete something over and over, occasionally it will attach.

  Warning  FailedMount       5m3s (x3 over 45m)  kubelet, apollo    Unable to attach or mount volumes: unmounted volumes=[app-config], unattached volumes=[default-token-vw6gx app-config]: timed out waiting for the condition

The LUN and target both successfully create and the PVC gets created successfully, but nothing will attach and the pod will stay in a ContainerCreating state.

The node logs show the following error:

I0214 17:14:07.207742       1 nodeserver.go:224] Unable to find target of ID: 6
E0214 17:14:07.207801       1 utils.go:100] GRPC error: rpc error: code = NotFound desc = Unable to find target of ID: 6