New node cant connect to proxmox API

[alejperez01]

Bug Report

Description

Hello! I'm having some problems adding new nodes to my kubernetes cluster. Im adding the labels correctly and proxmox-csi-plugin is running properly on all nodes except the new one....

The strange thing here is that the proxmox-csi-plugin pod on the node is crashing all the time because of the following error:
1 main.go:122] GRPC error: failed to get node ale-k8s-node-1: Get "https://10.96.0.1:443/api/v1/nodes/ale-k8s-node-1": dial tcp 10.96.0.1:443: i/o timeout
The node is named exactly like that on proxmox and the ip proxmox-csi-plugin is trying to lookup is the kubernetes default clusterip service.

Logs

Controller: Defaulted container "proxmox-csi-plugin-controller" out of: proxmox-csi-plugin-controller, csi-attacher, csi-provisioner, csi-resizer, liveness-probe I0111 21:56:46.401285 1 main.go:57] "Version" version="0.4.0" csiVersion="1.9.0" gitVersion="edge" gitCommit="27dd451" I0111 21:56:46.406330 1 merged_client_builder.go:163] Using in-cluster namespace I0111 21:56:46.406387 1 merged_client_builder.go:121] Using in-cluster configuration I0111 21:56:46.409337 1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false I0111 21:56:46.409347 1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false I0111 21:56:46.409351 1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false I0111 21:56:46.409355 1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false I0111 21:56:46.458863 1 main.go:139] "Listening for connection on address" address="/csi/csi.sock"

Node: I0111 22:37:23.670680 1 main.go:54] Driver version 0.4.0, GitVersion edge, GitCommit 27dd451 I0111 22:37:23.671466 1 main.go:55] Driver CSI Spec version: 1.9.0 I0111 22:37:23.671473 1 main.go:83] Building kube configs for running in cluster... I0111 22:37:23.671649 1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false I0111 22:37:23.671659 1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false I0111 22:37:23.671663 1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false I0111 22:37:23.671667 1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false I0111 22:37:23.875896 1 mount_linux.go:334] Detected umount with safe 'not mounted' behavior I0111 22:37:23.875968 1 main.go:140] Listening for connection on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"} I0111 22:37:23.914698 1 identity.go:40] "GetPluginInfo: called" I0111 22:37:24.580519 1 identity.go:40] "GetPluginInfo: called" I0111 22:37:25.286751 1 node.go:541] "NodeGetInfo: called" E0111 22:37:55.317077 1 main.go:122] GRPC error: failed to get node ale-k8s-node-1: Get "https://10.96.0.1:443/api/v1/nodes/ale-k8s-node-1": dial tcp 10.96.0.1:443: i/o timeout I0111 22:38:07.331740 1 identity.go:40] "GetPluginInfo: called" I0111 22:38:09.310942 1 node.go:541] "NodeGetInfo: called" E0111 22:38:39.335861 1 main.go:122] GRPC error: failed to get node ale-k8s-node-1: Get "https://10.96.0.1:443/api/v1/nodes/ale-k8s-node-1": dial tcp 10.96.0.1:443: i/o timeout

Environment

• Plugin version: 0.7.0
• Kubernetes version: 1.31
• CSI capasity: proxmox-data-ssd 318133226387 map[topology.kubernetes.io/region:ale-proxmox topology.kubernetes.io/zone:proxmox-2] proxmox-data-xfs 0 map[topology.kubernetes.io/region:ale-proxmox topology.kubernetes.io/zone:proxmox-2] proxmox-data-xfs 0 map[topology.kubernetes.io/region:ale-proxmox topology.kubernetes.io/zone:proxmox] proxmox-data-hdd 5387727061104 map[topology.kubernetes.io/region:ale-proxmox topology.kubernetes.io/zone:proxmox] proxmox-data-hdd 0 map[topology.kubernetes.io/region:ale-proxmox topology.kubernetes.io/zone:proxmox-2] proxmox-data-ssd 698206464980 map[topology.kubernetes.io/region:ale-proxmox topology.kubernetes.io/zone:proxmox]
• CSI resource on the node: [`apiVersion: storage.k8s.io/v1
  kind: CSINode
  metadata:
  annotations:
  storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-volume,kubernetes.io/vsphere-volume
  creationTimestamp: "2025-01-11T22:28:05Z"
  name: ale-k8s-node-1
  ownerReferences:
  • apiVersion: v1
    kind: Node
    name: ale-k8s-node-1
    uid: a3eca635-c7f7-4b1e-b353-409da3ee1b4a
    resourceVersion: "28478729"
    uid: 2c6d6f71-982e-40c8-ab8a-92f63d275dc7
    spec:
    drivers: null`]
• Node describe: [`Name: ale-k8s-node-1
  Roles:
  Labels: beta.kubernetes.io/arch=amd64
  beta.kubernetes.io/os=linux
  kubernetes.io/arch=amd64
  kubernetes.io/hostname=ale-k8s-node-1
  kubernetes.io/os=linux
  topology.kubernetes.io/region=ale-proxmox
  topology.kubernetes.io/zone=proxmox
  Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
  node.alpha.kubernetes.io/ttl: 0
  volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp: Sat, 11 Jan 2025 23:28:04 +0100
  Taints:
  Unschedulable: false
  Lease:
  HolderIdentity: ale-k8s-node-1
  AcquireTime:
  RenewTime: Sun, 12 Jan 2025 00:34:18 +0100
  Conditions:
  Type Status LastHeartbeatTime LastTransitionTime Reason Message

  ---

  NetworkUnavailable False Sat, 11 Jan 2025 23:37:12 +0100 Sat, 11 Jan 2025 23:37:12 +0100 WeaveIsUp Weave pod has set this
  MemoryPressure False Sun, 12 Jan 2025 00:33:12 +0100 Sat, 11 Jan 2025 23:28:04 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
  DiskPressure False Sun, 12 Jan 2025 00:33:12 +0100 Sat, 11 Jan 2025 23:28:04 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
  PIDPressure False Sun, 12 Jan 2025 00:33:12 +0100 Sat, 11 Jan 2025 23:28:04 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
  Ready True Sun, 12 Jan 2025 00:33:12 +0100 Sat, 11 Jan 2025 23:28:12 +0100 KubeletReady kubelet is posting ready status
  Addresses:
  InternalIP: 192.168.10.221
  Hostname: ale-k8s-node-1
  Capacity:
  cpu: 2
  ephemeral-storage: 29801344Ki
  hugepages-2Mi: 0
  memory: 6067400Ki
  pods: 110
  Allocatable:
  cpu: 2
  ephemeral-storage: 27464918585
  hugepages-2Mi: 0
  memory: 5965000Ki
  pods: 110
  System Info:
  Machine ID: 38b416b4c4ae46d8ad035523d16c9cfa
  System UUID: 90872a3e-443e-448f-96fb-ac989a3133ad
  Boot ID: 0217a29d-25f5-4ff8-adc1-4f96e34fd890
  Kernel Version: 6.1.0-29-amd64
  OS Image: Debian GNU/Linux 12 (bookworm)
  Operating System: linux
  Architecture: amd64
  Container Runtime Version: containerd://1.7.24
  Kubelet Version: v1.31.4
  Kube-Proxy Version: v1.31.4
  PodCIDR: 10.244.1.0/24
  PodCIDRs: 10.244.1.0/24
  Non-terminated Pods: (4 in total)
  Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age

  ---

  csi-proxmox proxmox-csi-plugin-node-6hxc9 20m (1%) 0 (0%) 32Mi (0%) 0 (0%) 59m
  kube-system kube-proxy-rwcs6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 66m
  kube-system weave-net-4fght 100m (5%) 0 (0%) 0 (0%) 0 (0%) 66m
  monitoring kube-prometheus-prometheus-node-exporter-fmbg6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 66m
  Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource Requests Limits

  ---

  cpu 120m (6%) 0 (0%)
  memory 32Mi (0%) 0 (0%)
  ephemeral-storage 0 (0%) 0 (0%)
  hugepages-2Mi 0 (0%) 0 (0%)
  Events:
  Type Reason Age From Message

  ---

  Normal Starting 57m kube-proxy
  Normal Starting 57m kubelet Starting kubelet.
  Warning InvalidDiskCapacity 57m kubelet invalid capacity 0 on image filesystem
  Normal NodeAllocatableEnforced 57m kubelet Updated Node Allocatable limit across pods
  Normal NodeHasSufficientMemory 57m kubelet Node ale-k8s-node-1 status is now: NodeHasSufficientMemory
  Normal NodeHasNoDiskPressure 57m kubelet Node ale-k8s-node-1 status is now: NodeHasNoDiskPressure
  Normal NodeHasSufficientPID 57m kubelet Node ale-k8s-node-1 status is now: NodeHasSufficientPID
  Warning Rebooted 57m kubelet Node ale-k8s-node-1 has been rebooted, boot id: 0217a29d-25f5-4ff8-adc1-4f96e34fd890`]
• OS version [cat /etc/os-release]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request

[sergelogvinov]

It seems that the pod is unable to reach the https://10.96.0.1:443 endpoint, which I believe corresponds to the Kubernetes API service kubernetes.default.svc.

I recommend checking the status of the network CNI or manually verifying whether the Kubernetes API can be accessed from this node.