cdi fails to clone filesystem #3119

Smithx10 · 2024-03-03T00:20:53Z

Attempting to import and use Fedora from Example:

I import Fedora

root@rke2-s0:~# kubectl logs -f importer-prime-67234c22-9b18-40be-aaa2-c8ad1d5f6744 -n vm-images
I0303 00:14:56.214817       1 importer.go:103] Starting importer
I0303 00:14:56.215021       1 importer.go:176] begin import process
I0303 00:14:57.200265       1 data-processor.go:356] Calculating available size
I0303 00:14:57.200331       1 data-processor.go:368] Checking out file system volume size.
I0303 00:14:57.200341       1 data-processor.go:376] Request image size not empty.
I0303 00:14:57.200370       1 data-processor.go:381] Target size 5681173672.
I0303 00:14:57.255964       1 data-processor.go:255] New phase: TransferDataFile
I0303 00:14:57.256315       1 util.go:194] Writing data...
I0303 00:14:58.256825       1 prometheus.go:75] 0.02
I0303 00:14:59.266337       1 prometheus.go:75] 0.33
...
I0303 00:16:58.143989       1 prometheus.go:75] 99.98
I0303 00:16:59.145063       1 prometheus.go:75] 99.99
I0303 00:17:00.145488       1 prometheus.go:75] 99.99
I0303 00:17:01.145856       1 prometheus.go:75] 99.99
I0303 00:17:02.147496       1 prometheus.go:75] 99.99
I0303 00:17:03.165908       1 prometheus.go:75] 100.00
I0303 00:17:04.153830       1 data-processor.go:255] New phase: Resize
I0303 00:17:04.167083       1 prometheus.go:75] 100.00
E0303 00:17:04.180605       1 prlimit.go:155] failed to kill the process; os: process already finished
W0303 00:17:04.181622       1 data-processor.go:338] Available space less than requested size, resizing image to available space 5368709120.
I0303 00:17:04.181722       1 data-processor.go:341] No need to resize image. Requested size: 5681173672, Image size: 5368709120.
I0303 00:17:04.181962       1 data-processor.go:261] Validating image
E0303 00:17:04.188340       1 prlimit.go:155] failed to kill the process; os: process already finished
I0303 00:17:04.188480       1 data-processor.go:255] New phase: Complete
I0303 00:17:04.188618       1 importer.go:223] Import Complete

I than Define a VM see definition below... and it attempts to clone the PV

Source pod log failing

root@rke2-s0:~# kubectl logs -f f3d3cae7-fa02-4e01-aca4-41512aa2710c-source-pod -n vm-images
VOLUME_MODE=filesystem
MOUNT_POINT=/var/run/cdi/clone/source
/var/run/cdi/clone/source /
UPLOAD_BYTES=5368729600
I0303 00:18:59.909560      10 clone-source.go:220] content-type is "filesystem-clone"
I0303 00:18:59.909936      10 clone-source.go:221] mount is "/var/run/cdi/clone/source"
I0303 00:18:59.909990      10 clone-source.go:222] upload-bytes is 5368729600
I0303 00:18:59.910054      10 clone-source.go:239] Starting cloner target
I0303 00:18:59.910232      10 clone-source.go:177] Executing [/usr/bin/tar cv -S disk.img]
I0303 00:19:00.594647      10 clone-source.go:251] Set header to filesystem-clone
I0303 00:19:00.912371      10 prometheus.go:75] 2.18
I0303 00:19:01.912428      10 prometheus.go:75] 7.60
I0303 00:19:02.912628      10 prometheus.go:75] 12.71
I0303 00:19:03.912777      10 prometheus.go:75] 18.34
I0303 00:19:04.912982      10 prometheus.go:75] 23.11
I0303 00:19:05.913059      10 prometheus.go:75] 26.60
I0303 00:19:06.913253      10 prometheus.go:75] 29.98
I0303 00:19:07.913393      10 prometheus.go:75] 34.27
I0303 00:19:08.913606      10 prometheus.go:75] 40.25
I0303 00:19:09.913749      10 prometheus.go:75] 44.74
I0303 00:19:10.913817      10 prometheus.go:75] 47.34
I0303 00:19:11.913923      10 prometheus.go:75] 50.38
I0303 00:19:12.914073      10 prometheus.go:75] 55.24
I0303 00:19:13.914228      10 prometheus.go:75] 61.63
I0303 00:19:14.914310      10 prometheus.go:75] 67.05
I0303 00:19:15.914998      10 prometheus.go:75] 72.78
I0303 00:19:16.915087      10 prometheus.go:75] 78.53
I0303 00:19:17.965801      10 prometheus.go:75] 83.82
I0303 00:19:18.965878      10 prometheus.go:75] 90.04
I0303 00:19:19.966007      10 prometheus.go:75] 95.90
I0303 00:19:20.574333      10 clone-source.go:127] Wrote 5368719360 bytes
F0303 00:19:20.577763      10 clone-source.go:260] Unexpected status code 500

#cdi upload logs

root@rke2-s0:~# kubectl logs -f cdi-upload-tmp-pvc-1d761a41-fc76-47ea-9cad-f1c29b75bac2 -n vm-images
I0303 00:18:54.308174       1 uploadserver.go:74] Running server on 0.0.0.0:8443
I0303 00:19:00.611111       1 uploadserver.go:389] Content type header is "filesystem-clone"
I0303 00:19:00.611315       1 util.go:207] begin untar to /data...
I0303 00:19:00.611347       1 util.go:213] running untar cmd: [/usr/bin/tar --preserve-permissions --no-same-owner -xvC /data]
E0303 00:19:20.576544       1 util.go:222] exit status 2
E0303 00:19:20.576914       1 uploadserver.go:402] Saving stream failed: error unarchiving to /data: exit status 2
I0303 00:19:22.030210       1 uploadserver.go:389] Content type header is "filesystem-clone"
I0303 00:19:22.030376       1 util.go:207] begin untar to /data...
I0303 00:19:22.030391       1 util.go:213] running untar cmd: [/usr/bin/tar --preserve-permissions --no-same-owner -xvC /data]
E0303 00:19:42.279237       1 util.go:222] exit status 2
E0303 00:19:42.279336       1 uploadserver.go:402] Saving stream failed: error unarchiving to /data: exit status 2
I0303 00:19:57.195179       1 uploadserver.go:389] Content type header is "filesystem-clone"
I0303 00:19:57.195323       1 util.go:207] begin untar to /data...
I0303 00:19:57.195371       1 util.go:213] running untar cmd: [/usr/bin/tar --preserve-permissions --no-same-owner -xvC /data]

mounts where the Pod is attempting to write "disk.img"

root@rke2-a0:~# mount | grep pvc
/dev/zd16 on /var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~csi/pvc-f3d3cae7-fa02-4e01-aca4-41512aa2710c/mount type ext4 (rw,relatime,stripe=4)
/dev/zd0 on /var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~csi/pvc-8da0c402-4527-4cc0-8fc5-c4a58d57725b/mount type ext4 (rw,relatime,stripe=4)
root@rke2-a0:~# ls /var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~csi/pvc-f3d3cae7-fa02-4e01-aca4-41512aa2710c/mount
lost+found
root@rke2-a0:~# ls /var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~csi/pvc-8da0c402-4527-4cc0-8fc5-c4a58d57725b/mount
disk.img  lost+found
root@rke2-a0:~#



# The Source container
root@rke2-a0:~# crictl inspect  286881c923e1f | jq -r '.info.config.mounts'
[
  {
    "container_path": "/var/run/cdi/clone/source",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~csi/pvc-8da0c402-4527-4cc0-8fc5-c4a58d57725b/mount",
    "readonly": true
  },
  {
    "container_path": "/var/run/secrets/kubernetes.io/serviceaccount",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~projected/kube-api-access-dwvzc",
    "readonly": true
  },
  {
    "container_path": "/etc/hosts",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/etc-hosts"
  },
  {
    "container_path": "/dev/termination-log",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/containers/cdi-clone-source/07dee6b3"
  }
]

# Upload Container
root@rke2-a0:~# crictl inspect  992eb5929e7fa | jq -r '.info.config.mounts'
[
  {
    "container_path": "/data",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~csi/pvc-f3d3cae7-fa02-4e01-aca4-41512aa2710c/mount"
  },
  {
    "container_path": "/var/run/secrets/kubernetes.io/serviceaccount",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~projected/kube-api-access-4gtch",
    "readonly": true
  },
  {
    "container_path": "/etc/hosts",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/etc-hosts"
  },
  {
    "container_path": "/dev/termination-log",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/containers/cdi-upload-server/472c045e"
  }
]

I have the following Defined:
root@rke2-s0:~# kubectl version
Client Version: v1.29.0+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0+rke2r1

VM Definition

root@rke2-s0:~# cat vm1_pvc.yml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: bruce
  name: bruce
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: bruce
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolumedisk1
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 4Gi
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: bruce
        name: datavolumedisk1
      - cloudInitNoCloud:
          userData: |
            #cloud-config
            hostname: bruce
            ssh_pwauth: True
            disable_root: false
            ssh_authorized_keys:
            - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/
        name: cloudinitdisk
  dataVolumeTemplates:
  - metadata:
      name: bruce
    spec:
      pvc:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 7Gi
      source:
        pvc:
          namespace: vm-images
          name: fedora

DataVolume Source

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "fedora"
  namespace: "vm-images"
spec:
  storage:
    resources:
      requests:
        storage: 5Gi
    storageClassName: "images"
  source:
    http:
      url: "https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz"

Storage Profile

root@rke2-s0:~# kubectl get storageprofile images -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  creationTimestamp: "2024-03-02T05:53:07Z"
  generation: 3
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    cdi.kubevirt.io: ""
  name: images
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi
    uid: 02f7dc11-6ca6-4036-ab3a-08d5b71a5feb
  resourceVersion: "559411"
  uid: 9c584532-a0bd-42cf-bfe5-47f6c9c7c9ff
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
status:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
  cloneStrategy: copy
  dataImportCronSourceFormat: pvc
  provisioner: zfs.csi.openebs.io
  storageClass: images

Storage Classes

root@rke2-s0:~# kubectl get storageclass -o yaml
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4SRP0/DMBTEvwq6OU2VtvlTSwyIkRGJ/Tl5bU0Sv8jPBEHV745MO2SBjvbd/XzWnUGTe+OgTjwMNEqgI+d9o7mT9VwgQ+98B4PXq/Q8kCoyjBypo0gwZ5D3Eik68ZqOYt+5jcoxD07ylmIcONFcwiD7U5dPz2F1nHsY9FtdKHORPbw43z0+dZ34uwhPI8PAjXTk9tb3fkInalOs/7C80i+NPOKSYSDLw7//OpGeYHCoinpPVWlLtjWXddOUm93BbnlfbXi7qwvu6qqiJkGXDTVdTBRo5Mjh96VJZLhZrkXWC2eQ2aW5OMDg+6B5qy6XiT3bNBouPwEAAP//DxMpHtQBAAA
      objectset.rio.cattle.io/id: ""
      objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
      objectset.rio.cattle.io/owner-name: imageclass
      objectset.rio.cattle.io/owner-namespace: kube-system
    creationTimestamp: "2024-03-02T05:51:51Z"
    labels:
      objectset.rio.cattle.io/hash: f6179a65b5eb7e5788524fb3e962e3471ed766a8
    name: images
    resourceVersion: "339"
    uid: f737dde3-589d-45f8-bc48-df295cfd570f
  parameters:
    poolname: system/images
  provisioner: zfs.csi.openebs.io
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"volumes"},"parameters":{"poolname":"system/volumes"},"provisioner":"zfs.csi.openebs.io"}
      storageclass.kubernetes.io/is-default-class: "true"
    creationTimestamp: "2024-03-02T23:45:52Z"
    name: volumes
    resourceVersion: "559539"
    uid: 169e19da-ec92-45c1-bdc1-32632332c3b0
  parameters:
    poolname: system/volumes
  provisioner: zfs.csi.openebs.io
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
kind: List
metadata:
  resourceVersion: ""

#Kubernetes version

root@rke2-s0:~# kubectl version
Client Version: v1.29.0+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0+rke2r1

#CDI Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    operator.cdi.kubevirt.io/lastAppliedConfiguration: '{"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"cdi-deployment","namespace":"cdi","creationTimestamp":null,"labels":{"app":"containerized-data-importer","app.kubernetes.io/component":"storage","app.kubernetes.io/managed-by":"cdi-operator","cdi.kubevirt.io":"","prometheus.cdi.kubevirt.io":"true"}},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"containerized-data-importer"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"containerized-data-importer","app.kubernetes.io/component":"storage","app.kubernetes.io/managed-by":"cdi-operator","cdi.kubevirt.io":"","prometheus.cdi.kubevirt.io":"true"}},"spec":{"volumes":[{"name":"cdi-api-signing-key","secret":{"secretName":"cdi-api-signing-key","items":[{"key":"id_rsa.pub","path":"id_rsa.pub"},{"key":"id_rsa","path":"id_rsa"}],"defaultMode":420}},{"name":"uploadserver-ca-cert","secret":{"secretName":"cdi-uploadserver-signer","items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"}],"defaultMode":420}},{"name":"uploadserver-client-ca-cert","secret":{"secretName":"cdi-uploadserver-client-signer","items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"}],"defaultMode":420}},{"name":"uploadserver-ca-bundle","configMap":{"name":"cdi-uploadserver-signer-bundle","items":[{"key":"ca-bundle.crt","path":"ca-bundle.crt"}],"defaultMode":420}},{"name":"uploadserver-client-ca-bundle","configMap":{"name":"cdi-uploadserver-client-signer-bundle","items":[{"key":"ca-bundle.crt","path":"ca-bundle.crt"}],"defaultMode":420}}],"containers":[{"name":"cdi-controller","image":"quay.io/kubevirt/cdi-controller:v1.58.1","args":["-v=1"],"ports":[{"name":"metrics","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"IMPORTER_IMAGE","value":"quay.io/kubevirt/cdi-importer:v1.58.1"},{"name":"CLONER_IMAGE","value":"quay.io/kubevirt/cdi-cloner:v1.58.1"},{"name":"UPLOADSERVER_IMAGE","value":"quay.io/kubevirt/cdi-uploadserver:v1.58.1"},{"name":"UPLOADPROXY_SERVICE","value":"cdi-uploadproxy"},{"name":"PULL_POLICY","value":"IfNotPresent"},{"name":"INSTALLER_PART_OF_LABEL","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.labels[''app.kubernetes.io/part-of'']"}}},{"name":"INSTALLER_VERSION_LABEL","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.labels[''app.kubernetes.io/version'']"}}}],"resources":{"requests":{"cpu":"100m","memory":"150Mi"}},"volumeMounts":[{"name":"cdi-api-signing-key","mountPath":"/var/run/cdi/token/keys"},{"name":"uploadserver-ca-cert","mountPath":"/var/run/certs/cdi-uploadserver-signer"},{"name":"uploadserver-client-ca-cert","mountPath":"/var/run/certs/cdi-uploadserver-client-signer"},{"name":"uploadserver-ca-bundle","mountPath":"/var/run/ca-bundle/cdi-uploadserver-signer-bundle"},{"name":"uploadserver-client-ca-bundle","mountPath":"/var/run/ca-bundle/cdi-uploadserver-client-signer-bundle"}],"readinessProbe":{"exec":{"command":["cat","/tmp/ready"]},"initialDelaySeconds":2,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["ALL"]},"runAsNonRoot":true,"allowPrivilegeEscalation":false,"seccompProfile":{"type":"RuntimeDefault"}}}],"nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"cdi-sa","securityContext":{"runAsNonRoot":true},"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}],"priorityClassName":"kubevirt-cluster-critical"}},"strategy":{}},"status":{}}'
  creationTimestamp: "2024-03-02T05:52:53Z"
  generation: 1
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-operator
    cdi.kubevirt.io: ""
    operator.cdi.kubevirt.io/createVersion: v1.58.1
    prometheus.cdi.kubevirt.io: "true"
  name: cdi-deployment
  namespace: cdi
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi
    uid: 02f7dc11-6ca6-4036-ab3a-08d5b71a5feb
  resourceVersion: "1878"
  uid: 1b3aa410-8b11-4571-b02a-6631dc74e3f3
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: containerized-data-importer
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: containerized-data-importer
        app.kubernetes.io/component: storage
        app.kubernetes.io/managed-by: cdi-operator
        cdi.kubevirt.io: ""
        operator.cdi.kubevirt.io/createVersion: v1.58.1
        prometheus.cdi.kubevirt.io: "true"
    spec:
      containers:
      - args:
        - -v=1
        env:
        - name: IMPORTER_IMAGE
          value: quay.io/kubevirt/cdi-importer:v1.58.1
        - name: CLONER_IMAGE
          value: quay.io/kubevirt/cdi-cloner:v1.58.1
        - name: UPLOADSERVER_IMAGE
          value: quay.io/kubevirt/cdi-uploadserver:v1.58.1
        - name: UPLOADPROXY_SERVICE
          value: cdi-uploadproxy
        - name: PULL_POLICY
          value: IfNotPresent
        - name: INSTALLER_PART_OF_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/part-of']
        - name: INSTALLER_VERSION_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/version']
        image: quay.io/kubevirt/cdi-controller:v1.58.1
        imagePullPolicy: IfNotPresent
        name: cdi-controller
        ports:
        - containerPort: 8080
          name: metrics
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/ready
          failureThreshold: 3
          initialDelaySeconds: 2
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        res      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: containerized-data-importer
        app.kubernetes.io/component: storage
        app.kubernetes.io/managed-by: cdi-operator
        cdi.kubevirt.io: ""
        operator.cdi.kubevirt.io/createVersion: v1.58.1
        prometheus.cdi.kubevirt.io: "true"
    spec:
      containers:
      - args:
        - -v=1
        env:
        - name: IMPORTER_IMAGE
          value: quay.io/kubevirt/cdi-importer:v1.58.1
        - name: CLONER_IMAGE
          value: quay.io/kubevirt/cdi-cloner:v1.58.1
        - name: UPLOADSERVER_IMAGE
          value: quay.io/kubevirt/cdi-uploadserver:v1.58.1
        - name: UPLOADPROXY_SERVICE
          value: cdi-uploadproxy
        - name: PULL_POLICY
          value: IfNotPresent
        - name: INSTALLER_PART_OF_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/part-of']
        - name: INSTALLER_VERSION_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/version']
        image: quay.io/kubevirt/cdi-controller:v1.58.1
        imagePullPolicy: IfNotPresent
        name: cdi-controller
        ports:
        - containerPort: 8080
          name: metrics
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/ready
          failureThreshold: 3
          initialDelaySeconds: 2
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:ources:
          secretName: cdi-uploadserver-signer
      - name: uploadserver-client-ca-cert
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          secretName: cdi-uploadserver-client-signer
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: ca-bundle.crt
          name: cdi-uploadserver-signer-bundle
        name: uploadserver-ca-bundle
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: ca-bundle.crt
          name: cdi-uploadserver-client-signer-bundle
        name: uploadserver-client-ca-bundle
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2024-03-02T05:53:08Z"
    lastUpdateTime: "2024-03-02T05:53:08Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-03-02T05:52:53Z"
    lastUpdateTime: "2024-03-02T05:53:08Z"
    message: ReplicaSet "cdi-deployment-74b786dcc6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

The text was updated successfully, but these errors were encountered:

akalenyu · 2024-03-03T09:30:30Z

Hey, thanks for reporting this!

I think if you increase the CDI log verbosity with https://github.com/kubevirt/containerized-data-importer/blob/release-v1.58/doc/debug.md#log-verbosity
We should get the actual stdout/stderr of the untar command

May be a similar issue to #2982

aglitke · 2024-05-06T12:44:33Z

@Smithx10 If you'd still like to pursue this issue could you please retry with more verbosity?

cccsss01 · 2024-06-13T04:20:58Z

I'm running into OOMKilled w/ 32Gi RAM

awels · 2024-06-13T12:23:26Z

I'm running into OOMKilled w/ 32Gi RAM

So the import/clone/upload pods have a request/limit set on them, and you get OOM killed if you go over that limit. See https://github.com/kubevirt/containerized-data-importer/blob/main/doc/quota.md on how to modify that limit. In most cases it should not be an issue, but we are seeing that slow storage has the ability to exhaust that memory because the OS is caching the writes in memory that counts against that limit.

cccsss01 · 2024-06-13T18:01:56Z

@awels
Applying the example CDIConfig fails.
error: error parsing cdiconfig.yaml: error converting YAML to JSON: yaml: line 9: did not find expected ',' or '}'

I've tried the example, and
spec:
podResourceRequirements:
limits:
cpu: "4"
memory: "1Gi"
requests:
cpu: "1"
memory: "250Mi"
This fails with unknown field "spec.limits", unknown field "spec.requests"

PR submitted for what the CDI components that worked for me, however still testing the overall process

cccsss01 · 2024-06-13T18:38:45Z

I did get farther, however I received
Error io: read/write on closed pipe piping to gzip on the source pod
Error Post "https://cdi-upload-tmp-pvc-58b0c7a5-f12b-4b0f-9ef5-471bd95e5fde.default.svc/v1beta1/upload": write tcp 10.0.0.99:51356->10.111.44.190:443: write: connection reset by peer POSTing to https://cdi-upload-tmp-pvc-58b0c7a5-f12b-4b0f-9ef5-471bd95e5fde.default.svc/v1beta1/upload

cccsss01 · 2024-06-14T03:31:50Z

for vm1_pvc.yml I get failed to create virtual machine pod: pods "virt-launcher-vm1-r6rvd" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu for: compute; limits.memory for: compute

akalenyu · 2024-07-01T12:50:58Z

for vm1_pvc.yml I get failed to create virtual machine pod: pods "virt-launcher-vm1-r6rvd" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu for: compute; limits.memory for: compute

Is there quota/limitrange on the namespace? In that case, you would have to specify requests/limits on the VM resource

akalenyu · 2024-07-15T12:29:35Z

@cccsss01 hey, did you get a chance to try this since? specifically interested if there were some limits on the namespace

cccsss01 · 2024-07-15T13:19:25Z

There was no quota limitrange on the namespace, however, specifying the requests/limits worked.

kubevirt-bot · 2024-10-13T14:05:11Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kubevirt-bot · 2024-11-12T14:42:53Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

kubevirt-bot · 2024-12-12T15:26:40Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

kubevirt-bot · 2024-12-12T15:26:45Z

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Smithx10 added the kind/bug label Mar 3, 2024

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 13, 2024

kubevirt-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 12, 2024

kubevirt-bot closed this as completed Dec 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cdi fails to clone filesystem #3119

cdi fails to clone filesystem #3119

Smithx10 commented Mar 3, 2024 •

edited

Loading

akalenyu commented Mar 3, 2024 •

edited

Loading

aglitke commented May 6, 2024

cccsss01 commented Jun 13, 2024

awels commented Jun 13, 2024

cccsss01 commented Jun 13, 2024 •

edited

Loading

cccsss01 commented Jun 13, 2024 •

edited

Loading

cccsss01 commented Jun 14, 2024

akalenyu commented Jul 1, 2024

akalenyu commented Jul 15, 2024

cccsss01 commented Jul 15, 2024

kubevirt-bot commented Oct 13, 2024

kubevirt-bot commented Nov 12, 2024

kubevirt-bot commented Dec 12, 2024

kubevirt-bot commented Dec 12, 2024

cdi fails to clone filesystem #3119

cdi fails to clone filesystem #3119

Comments

Smithx10 commented Mar 3, 2024 • edited Loading

Source pod log failing

mounts where the Pod is attempting to write "disk.img"

VM Definition

DataVolume Source

Storage Profile

Storage Classes

akalenyu commented Mar 3, 2024 • edited Loading

aglitke commented May 6, 2024

cccsss01 commented Jun 13, 2024

awels commented Jun 13, 2024

cccsss01 commented Jun 13, 2024 • edited Loading

cccsss01 commented Jun 13, 2024 • edited Loading

cccsss01 commented Jun 14, 2024

akalenyu commented Jul 1, 2024

akalenyu commented Jul 15, 2024

cccsss01 commented Jul 15, 2024

kubevirt-bot commented Oct 13, 2024

kubevirt-bot commented Nov 12, 2024

kubevirt-bot commented Dec 12, 2024

kubevirt-bot commented Dec 12, 2024

Smithx10 commented Mar 3, 2024 •

edited

Loading

akalenyu commented Mar 3, 2024 •

edited

Loading

cccsss01 commented Jun 13, 2024 •

edited

Loading

cccsss01 commented Jun 13, 2024 •

edited

Loading