Creating a PV or PVC

[GrayR0ot]

Hello,

I just started using this amazing tool for doing some tests.

I'm trying to create a simple volume/pvc to make an app test.
But when I create a PVC I'm getting theses events:

PVC:
waiting for pod player-service-database-0 to be scheduled
waiting for first consumer to be created before binding

Pod:
0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {CriticalAddonsOnly: true}. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

Shouldn't it create an Hetzner volume, attach it to a node, and then bind my PVC to this pv ?

Regards

[vitobotta]

Hi,

Are you using the hcloud-volumes storage class? Also, are your worker nodes in a ready state? Could you share more details, like the manifests you are using for the PVC and the deployment?

[GrayR0ot]

Damn, fast answer haha.

Sure, here are my manifests:

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: player-service-database-pvc
  namespace: minecraft-staging
  labels:
    app: player-service-database
spec:
  storageClassName: hcloud-volumes
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

StatefulSet:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: player-service-database
  namespace: minecraft-staging
  labels:
    app: player-service-database
spec:
  serviceName: player-service-database
  replicas: 1
  selector:
    matchLabels:
      app: player-service-database
  template:
    metadata:
      labels:
        app: player-service-database
    spec:
      nodeSelector:
        node-type: dev-worker
      volumes:
        - name: player-service-database-pvc
          persistentVolumeClaim:
            claimName: player-service-database-pvc
      containers:
        - name: database-sftp-server
          image: atmoz/sftp
          resources:
            requests:
              cpu: "10m"
              memory: "50Mi"
            limits:
              cpu: "100m"
              memory: "100Mi"
          ports:
            - containerPort: 22
              name: sftp-port
          volumeMounts:
            - mountPath: /home/postgres/data
              name: player-service-database-pvc
          env:
            - name: SFTP_USERS
              valueFrom:
                secretKeyRef:
                  name: player-service-secret
                  key: SFTP_USERS
        - name: s3-backup
          image: peterrus/s3-cron-backup
          resources:
            requests:
              cpu: "10m"
              memory: "50Mi"
            limits:
              cpu: "20m"
              memory: "100Mi"
          volumeMounts:
            - mountPath: /data
              readOnly: true
              name: player-service-database-pvc
          env:
            - name: BACKUP_NAME
              value: player-service-database
            - name: CRON_SCHEDULE
              value: "0 */3 * * *"
            - name: AWS_ACCESS_KEY_ID
              value: "XXX"
            - name: AWS_SECRET_ACCESS_KEY
              value: "XXX"
            - name: S3_BUCKET_URL
              value: "s3://staging/player-service-database"
            - name: S3_ENDPOINT
              value: "https://XXX"
        - name: postgres
          image: postgres:17.3
          imagePullPolicy: IfNotPresent
          env:
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: player-service-secret
                  key: DB_PASSWORD
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: player-service-secret
                  key: DB_USERNAME
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  name: player-service-secret
                  key: DB_USERNAME
          resources:
            requests:
              cpu: "0.25"
              memory: "1Gi"
            limits:
              cpu: "0.5"
              memory: "1Gi"
          ports:
            - containerPort: 5432
              name: postgres-port
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: player-service-database-pvc

❯ kubectl get nodes
NAME STATUS ROLES AGE VERSION
hetzner-k3s-master1 Ready control-plane,etcd,master 14m v1.32.0+k3s1
hetzner-k3s-master2 Ready control-plane,etcd,master 13m v1.32.0+k3s1
hetzner-k3s-master3 Ready control-plane,etcd,master 13m v1.32.0+k3s1
hetzner-k3s-pool-small-worker1 Ready 12m v1.32.0+k3s1
hetzner-k3s-pool-small-worker2 Ready 12m v1.32.0+k3s1

[vitobotta]

I provided a quick response because I'm working a lot on hetzner-k3s at the moment for work :)

Uh, I can't spot any obvious issues from the manifests. Could you describe the volume and see what info it gives you? Also, it might be worth checking the logs for the CSI pods, both the controller and the daemonset, which you can find in kube-system. If there's a problem like attaching the volume or something similar, those logs should provide some useful information.

[GrayR0ot]

❯ kubectl describe pvc player-service-database-pvc -n minecraft-staging

Name:          player-service-database-pvc
Namespace:     minecraft-staging
StorageClass:  hcloud-volumes
Status:        Pending
Volume:
Labels:        app=player-service-database
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       player-service-database-0
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  WaitForFirstConsumer  24m                   persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled   4m40s (x81 over 24m)  persistentvolume-controller  waiting for pod player-service-database-0 to be scheduled

CSI node logs:

I0219 13:43:47.756925       1 node_register.go:96] "Skipping HTTP server"
I0219 13:43:47.837599       1 main.go:96] "Received GetInfo call" request="&InfoRequest{}"
I0219 13:43:47.897204       1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}"
I0219 13:43:47.924310       1 main.go:96] "Received GetInfo call" request="&InfoRequest{}"
I0219 13:43:48.064312       1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}"
I0219 13:43:49.588074       1 main.go:150] "Version" version="v2.13.0"
I0219 13:43:49.588174       1 main.go:151] "Running node-driver-registrar" mode=""
I0219 13:43:49.860193       1 main.go:150] "Version" version="v2.13.0"
I0219 13:43:49.860336       1 main.go:151] "Running node-driver-registrar" mode=""
time=2025-02-19T13:43:51.017Z level=INFO source=/home/runner/work/csi-driver/csi-driver/cmd/node/main.go:49 msg="Fetched data from metadata service" id=60336815 location=fsn1
I0219 13:43:51.033946       1 main.go:135] "Calling CSI driver to discover driver name"
I0219 13:43:51.035805       1 main.go:143] "CSI driver name" driver="csi.hetzner.cloud"
I0219 13:43:51.035828       1 main.go:172] "ServeMux listening" address="0.0.0.0:9808"
time=2025-02-19T13:43:51.076Z level=INFO source=/home/runner/work/csi-driver/csi-driver/cmd/node/main.go:49 msg="Fetched data from metadata service" id=60336814 location=fsn1
I0219 13:43:51.196310       1 main.go:135] "Calling CSI driver to discover driver name"
I0219 13:43:51.200011       1 main.go:143] "CSI driver name" driver="csi.hetzner.cloud"
I0219 13:43:51.200069       1 main.go:172] "ServeMux listening" address="0.0.0.0:9808"
I0219 13:43:51.548234       1 node_register.go:56] "Starting Registration Server" socketPath="/registration/csi.hetzner.cloud-reg.sock"
I0219 13:43:51.548608       1 node_register.go:66] "Registration Server started" socketPath="/registration/csi.hetzner.cloud-reg.sock"
I0219 13:43:51.548829       1 node_register.go:96] "Skipping HTTP server"
I0219 13:43:51.855498       1 node_register.go:56] "Starting Registration Server" socketPath="/registration/csi.hetzner.cloud-reg.sock"
I0219 13:43:51.855731       1 node_register.go:66] "Registration Server started" socketPath="/registration/csi.hetzner.cloud-reg.sock"
I0219 13:43:51.855879       1 node_register.go:96] "Skipping HTTP server"
I0219 13:43:52.367723       1 main.go:96] "Received GetInfo call" request="&InfoRequest{}"
I0219 13:43:52.401248       1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}"
I0219 13:43:52.587886       1 main.go:96] "Received GetInfo call" request="&InfoRequest{}"
I0219 13:43:52.621631       1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}

CSI Controller:

I0219 13:43:50.436325       1 main.go:110] "Version" version="v4.8.0"
I0219 13:43:51.915944       1 main.go:108] "Version" version="v1.12.0"
W0219 13:43:53.375377       1 feature_gate.go:354] Setting GA feature gate Topology=true. It will be removed in a future release.
I0219 13:43:53.375554       1 csi-provisioner.go:166] "Version" version="v5.2.0"
I0219 13:43:53.375565       1 csi-provisioner.go:189] Building kube configs for running in cluster...
I0219 13:43:53.852179       1 common.go:143] "Probing CSI driver for readiness"
I0219 13:43:53.857708       1 controller.go:130] "Register Pod informer for resizer" controller="csi.hetzner.cloud"
I0219 13:43:53.857752       1 controller.go:272] "Starting external resizer" controller="csi.hetzner.cloud"
I0219 13:43:54.277645       1 common.go:143] "Probing CSI driver for readiness"
I0219 13:43:54.280839       1 controller.go:129] "Starting CSI attacher" driver="csi.hetzner.cloud"
I0219 13:43:54.377879       1 common.go:143] "Probing CSI driver for readiness"
I0219 13:43:54.379926       1 csi-provisioner.go:313] CSI driver supports PUBLISH_UNPUBLISH_VOLUME, watching VolumeAttachments
I0219 13:43:54.445147       1 main.go:135] "Calling CSI driver to discover driver name"
I0219 13:43:54.446682       1 main.go:143] "CSI driver name" driver="csi.hetzner.cloud"
I0219 13:43:54.446723       1 main.go:172] "ServeMux listening" address="0.0.0.0:9808"
I0219 13:43:54.480580       1 controller.go:838] "Starting provisioner controller" component="csi.hetzner.cloud_hcloud-csi-controller-586f87b5b5-45l65_7409d9e3-0635-4a26-8dd3-68202d125b41"
I0219 13:43:54.480647       1 volume_store.go:98] "Starting save volume queue"
I0219 13:43:54.581176       1 controller.go:887] "Started provisioner controller" component="csi.hetzner.cloud_hcloud-csi-controller-586f87b5b5-45l65_7409d9e3-0635-4a26-8dd3-68202d125b41"

[GrayR0ot]

IDK if the 60336815, and 60336814 are Hetzner volume IDs ? But there is nothing in hetzner cloud portal

[vitobotta]

Weird. If it was a problem with volume attachment you would see something in there. Are you sure there isn't something that is making the workload crash or something?

[GrayR0ot]

No It's just a fresh cluster created with the following config:

hetzner_token: XXX
cluster_name: hetzner-k3s
kubeconfig_path: "XXX"
k3s_version: v1.32.0+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    # Make sure private & public keys are in 0700/0600 chmod
    public_key_path: "~/.ssh/id_rsa.pub"
    private_key_path: "~/.ssh/id_rsa"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0

create_load_balancer_for_the_kubernetes_api: false

masters_pool:
  instance_type: cpx21
  instance_count: 3
  locations:
    - fsn1
    - hel1
    - nbg1

worker_node_pools:
  - name: small
    instance_type: cpx21
    instance_count: 2
    location: fsn1

[GrayR0ot]

Just for clarification, I shoudn't create an Hetzner volume or PV manually right ?

[vitobotta]

If you use a PVC, the volume is set up automatically for you. Can you see the volume in the Hetzner Console? If it’s a new setup, try deleting it and then give it another shot after making sure the volume isn’t listed in the Hetzner Console.

I was thinking it might be a temporary glitch with the Hetzner API, but it should show something in the CSI logs.

I am gonna do a quick test.

[GrayR0ot]

Damn, I just found I think, I copied my deployment from my previous cluster, but I had a nodeSelector

[GrayR0ot]

That wasn't displayed in my kube events, but it was under my eyes

[vitobotta]

😂

I guess we can close this then?

[GrayR0ot]

Just a simple question, can we prevent the deletion of hetzner volumes when we release the PVC ? If I delete a pvc for example^. I know there is a retainPolicy for PV, but for PVC nothing exists right?

in the cluster creation there is a protect_against_deletion, but it's not documented

[vitobotta]

As far as I know, this depends on the reclaimPolicy setting of the storage class you're using with the PVC. You might want to try creating a clone of the hcloud-volumes storage class and set the reclaimPolicy to Retain to see if that helps.

By the way, I'm changing this issue to a discussion since it doesn't seem to be about a bug or problem with the tool.