How to resize PVC using Helm chart?

[stemid]

I tried simply editing config.jetstream.fileStore.pvc.size from 2Gi to 3Gi and running helm upgrade, but nothing is happening.

I also tried scaling the nats statefulset down to 0, waiting for all the containers to terminate, PVC are not being resized.

I know my storageClass supports resizing because I use it often with the postgres operator. And I have done a resize with nats once, from 1Gi to 2Gi, but it was a long time ago so I don't remember how I did it.

[caleblloyd]

It's more of a Kubernetes limitation for resizing PVCs in a Stateful Set than anything. Easiest way is to leave the Stateful Set alone and resize the underlying disks to be the size that you want them to be.

There are some hacks documented out there on Stack Overflow and the likes that involve force-deleting the Stateful Set and then re-applying it. I stay away from those though, force deleting Stateful Set that you don't actually want to delete sounds scary

[stemid]

Ok so there's no real "nats way", you just have to do whatever kubernetes and statefulsets allow.

I don't mind deleting my statefulset, I have a testing cluster. But I wish this method worked, it does not.

kubectl -n nats delete sts/nats --cascade=orphan
kubectl -n nats rollout restart sts/nats

Still same PVC size. The problem is that updating the sts does not update the PVC.

kubectl -n nats get sts/nats -o jsonpath='{ .spec.volumeClaimTemplates[*].spec.resources.requests.storage }'
3Gi
kubectl -n nats get pvc nats-js-pvc-nats-0 --template='{{ .spec.resources.requests.storage }}'
2Gi

So I just had to patch each PVC manually.

kubectl -n nats patch pvc nats-js-pvc-nats-0 -p '{ "spec": { "resources": { "requests": { "storage": "3Gi"} } } }'
kubectl -n nats patch pvc nats-js-pvc-nats-1 -p '{ "spec": { "resources": { "requests": { "storage": "3Gi"} } } }'
kubectl -n nats patch pvc nats-js-pvc-nats-2 -p '{ "spec": { "resources": { "requests": { "storage": "3Gi"} } } }'

And then after each pod restarted I could see the new size.

kubectl -n nats rollout restart sts/nats
kubectl -n nats get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
nats-js-pvc-nats-0   Bound    pvc-40f352af-b85e-463e-bfe4-686556ca9ee9   3Gi        RWO            rook-ceph-block   650d
nats-js-pvc-nats-1   Bound    pvc-f297eb1c-3d26-4a67-b9e1-c07b95292535   3Gi        RWO            rook-ceph-block   650d
nats-js-pvc-nats-2   Bound    pvc-94b1366e-5e2d-4bff-9d49-a4450637b3ec   3Gi        RWO            rook-ceph-block   650d

Solution

1. First upgrade your helm chart with your new size in values.yaml.
2. Ensure your StatefulSet is updated by checking kubectl -n nats get sts/nats -o jsonpath='{ .spec.volumeClaimTemplates[*].spec.resources.requests.storage }' and ensure it outputs the new size.
3. Patch each PVC with the new size with kubectl -n nats patch pvc nats-js-pvc-nats-0 -p '{ "spec": { "resources": { "requests": { "storage": "3Gi"} } } }'
4. Do a rolling restart of your StatefulSet kubectl -n nats rollout restart sts/nats

[komronu]

I got the error when I did what you wrote.

Error: UPGRADE FAILED: cannot patch "nats" with kind StatefulSet: StatefulSet.apps "nats" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

[komronu]

Hello! It's My Solution :)

1. Add to sc this parametr

allowVolumeExpansion: true

2. Update PV/PVC. For example change storage size from 10GB to 20GB
3. aws console -> ec2 -> volumes -> modify -> change disk size
4. Wait disk volume update seccessfuly on aws
5. uninstall my nats

helm uninstall nats 

6. Update values.yml. Change storage size to 20GB
7. Install nats again

helm install nats . -f values.yaml

All working