Exporting ZFS pools in preparation for a move to a new cluster

[dhess]

Hi, thanks for this project! We've been using it for over a year, and it's been incredibly stable and performant.

We'd like to do the following:

1. Add 2 new drives per node to our existing bare metal Kubernetes cluster, which is running LocalPV-ZFS. Let's call this cluster-a.
2. Create one ZFS pool per new drive.
3. Create a StorageClass in the cluster that encompasses the new pools.
4. Create some fstype: zfs PVCs using the StorageClass from step 3.
5. Create new files on the PVCs created in step 4.
6. [... undo steps 3 and 4 in a non-destructive way to prepare for step 7]
7. zfs export the ZFS pools created in step 2.
8. Pull the new drives from cluster-a.
9. Add the new drives to a different bare metal Kubernetes cluster, also running LocalPV-ZFS, and import them into LocalPV-ZFS. Let's call this cluster-b.

In a nutshell, we want to use LocalPV-ZFS on cluster-a to create some new ZFS filesystems, then export the underlying pools so that we can move the drives from cluster-a to cluster-b and access the existing filesystems on cluster-b.

Steps 1-5 and 8-9 are straightforward. We plan to use this helpful document to import the existing drives into cluster-b: https://github.com/openebs/zfs-localpv/blob/master/docs/import-existing-volume.md

However, it's not clear how to do step 6. I assume that, for starters, we'll want to create the StorageClass in step 3 with reclaimPolicy: Retain, so that once the PVCs in step 4 are deleted from cluster-a, the underlying ZFS volumes aren't deleted from their pools. But what else do we need to do to make LocalPV-ZFS happy when we eventually export the ZFS pools and remove the drives from cluster-a? I assume some additional custom resources associated with the underlying ZFS resources will need to be deleted, but it's not clear how or what.

[Abhinandan-Purkait]

I think you would need to scale down the controller and the csi-node to be able to delete the resources, otherwise they will trigger the operations. I believe the intention here is that the underlying zfs volume is to be kept while the k8s resources are removed, so why not scale down the controller and csi-node, do the force delete of the resources. Export the zpool, and then redo the creation after scaling the controller and csi-node backup? Does that make sense?

[dhess]

In case it isn't clear, we have existing LocalPV-ZFS volumes that we want to keep in cluster-a. It's only the new drives that we want to temporarily add to cluster-a and load up with data before moving them to cluster-b.

Anyway, do I understand correctly that your suggestion would require taking all the LocalPV-ZFS volumes offline temporarily by scaling down the LocalPV-ZFS controller and csi-nodes during the export process for the new drives?

[sinhaashish]

Can you please elaborate more about the use case. What i get is that you have acluster-a with drive let say p . Now you want to add drive q in the cluster , get it populated with data , and then remove drive q from the cluster and reuse it in cluster-b ?

And your question is how to do this process effectively.
Attaching and using drive q in cluster-b is already documented as you pointed out earlier.

[Abhinandan-Purkait]

In case it isn't clear, we have existing LocalPV-ZFS volumes that we want to keep in cluster-a. It's only the new drives that we want to temporarily add to cluster-a and load up with data before moving them to cluster-b.

Anyway, do I understand correctly that your suggestion would require taking all the LocalPV-ZFS volumes offline temporarily by scaling down the LocalPV-ZFS controller and csi-nodes during the export process for the new drives?

Yes, the intention is not to delete any underlying ZFS volume while doing steps 3 and 4 but just delete the K8s resources(ZV,PVC,SC) so that once we recreate the SC, PVC, and ZV to import the newer one, it should be just a use existing ZFS volume operation rather than a fresh create.

[dhess]

Can you please elaborate more about the use case. What i get is that you have acluster-a with drive let say p . Now you want to add drive q in the cluster , get it populated with data , and then remove drive q from the cluster and reuse it in cluster-b ?

And your question is how to do this process effectively. Attaching and using drive q in cluster-b is already documented as you pointed out earlier.

The use case is to make a mirror of the contents of drive(s) p onto drive(s) q, which is much faster to bootstrap while the drives are all present in cluster-a. Then we want to export the ZFS filesystem(s) on q, remove the drives from cluster-a, and bring them back up in cluster-b.

[dhess]

In case it isn't clear, we have existing LocalPV-ZFS volumes that we want to keep in cluster-a. It's only the new drives that we want to temporarily add to cluster-a and load up with data before moving them to cluster-b.
Anyway, do I understand correctly that your suggestion would require taking all the LocalPV-ZFS volumes offline temporarily by scaling down the LocalPV-ZFS controller and csi-nodes during the export process for the new drives?

Yes, the intention is not to delete any underlying ZFS volume while doing steps 3 and 4 but just delete the K8s resources(ZV,PVC,SC) so that once we recreate the SC, PVC, and ZV to import the newer one, it should be just a use existing ZFS volume operation rather than a fresh create.

OK, thanks for clarifying. That would be pretty inconvenient to take all the LocalPV-ZFS volumes offline during the export. We may need to look for a different solution.

[Abhinandan-Purkait]

In case it isn't clear, we have existing LocalPV-ZFS volumes that we want to keep in cluster-a. It's only the new drives that we want to temporarily add to cluster-a and load up with data before moving them to cluster-b.
Anyway, do I understand correctly that your suggestion would require taking all the LocalPV-ZFS volumes offline temporarily by scaling down the LocalPV-ZFS controller and csi-nodes during the export process for the new drives?

Yes, the intention is not to delete any underlying ZFS volume while doing steps 3 and 4 but just delete the K8s resources(ZV,PVC,SC) so that once we recreate the SC, PVC, and ZV to import the newer one, it should be just a use existing ZFS volume operation rather than a fresh create.

OK, thanks for clarifying. That would be pretty inconvenient to take all the LocalPV-ZFS volumes offline during the export. We may need to look for a different solution.

I might not be grasping it fully, but how can we do a export without making the volumes offline at some point? cc @tiagolobocastro @avishnu any thoughts here?

[tiagolobocastro]

The remaining volumes should remain online because our components are simply a K8s control-plane for the kernel zfs volumes, so nothing will go offline here or am I missing something here?