Kubernetes StatefulSets are used to deploy stateful applications inside your cluster. Each Pod in the StatefulSet can access local persistent volumes that stick to it even after it's rescheduled. This allows Pods to maintain individual state that's separate from their neighbors in the set.

Unfortunately these volumes come with a big limitation: Kubernetes doesn't provide a way to resize them from the StatefulSet object. The spec.resources.requests.storage property of the StatefulSet's

        volumeClaimTemplates
    

field is immutable, preventing you from applying any capacity increases you require. This article will show you how to workaround the problem.

Creating a StatefulSet

Copy this YAML and save it to

        ss.yaml
    

:

apiVersion: v1
    
    
            kind: Service
    
    
            metadata:
    
    
             name: nginx
    
    
             labels:
    
    
             app: nginx
    
    
            spec:
    
    
             selector:
    
    
             app: nginx
    
    
             ports:
    
    
             - name: nginx
    
    
             port: 80
    
    
             clusterIP: None
    
    
            ---
    
    
            apiVersion: apps/v1
    
    
            kind: StatefulSet
    
    
            metadata:
    
    
             name: nginx
    
    
            spec:
    
    
             selector:
    
    
             matchLabels:
    
    
             app: nginx
    
    
             replicas: 3
    
    
             serviceName: nginx
    
    
             template:
    
    
             metadata:
    
    
             labels:
    
    
             app: nginx
    
    
             spec:
    
    
             containers:
    
    
             - name: nginx
    
    
             image: nginx:latest
    
    
             ports:
    
    
             - name: web
    
    
             containerPort: 80
    
    
             volumeMounts:
    
    
             - name: data
    
    
             mountPath: /usr/share/nginx/html
    
    
             volumeClaimTemplates:
    
    
             - metadata:
    
    
             name: data
    
    
             spec:
    
    
             accessModes: ["ReadWriteOnce"]
    
    
             resources:
    
    
             requests:
    
    
             storage: 1Gi

Apply the YAML to your cluster with Kubectl:

$ kubectl apply -f ss.yaml
    
    
            service/nginx created
    
    
            statefulset.apps/nginx created

You'll need a storage class and provisioner in your cluster to run this example. It creates a StatefulSet that runs three replicas of an NGINX web server.

While this isn't representative of when StatefulSets should be used, it's adequate as a demo of the volume problems you can face. A volume claim with 1 Gi of storage is mounted to NGINX's data directory. Your web content could outgrow this relatively small allowance as your service scales. However trying to modify the volumeClaimTemplates.spec.resources.requests.storage field to 10Gi will report the following error when you run kubectl apply:

$ kubectl apply -f ss.yaml
    
    
            service/nginx unchanged
    
    
            The StatefulSet "nginx" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

This occurs because almost all the fields of a StatefulSet's manifest are immutable after creation.

Manually Resizing StatefulSet Volumes

You can bypass the restriction by manually resizing the persistent volume claim (PVC). You'll then need to recreate the StatefulSet to release and rebind the volume from your Pods. This will trigger the actual volume resize event.

First use Kubectl to find the PVCs associated with your StatefulSet:

$ kubectl get pvc
    
    
            NAME STATUS VOLUME CAPACITY ACCESS MODES
    
    
            data-nginx-0 Bound pvc-ccb2c835-e2d3-4632-b8ba-4c8c142795e4 1Gi RWO 
    
    
            data-nginx-1 Bound pvc-1b0b27fe-3874-4ed5-91be-d8e552e515f2 1Gi RWO 
    
    
            data-nginx-2 Bound pvc-4b7790c2-3ae6-4e04-afee-a2e1bae4323b 1Gi RWO

There are three PVCs because there are three replicas in the StatefulSet. Each Pod gets its own individual volume.

Now use kubectl edit to adjust the capacity of each volume:

$ kubectl edit pvc data-nginx-0

The PVC's YAML manifest will appear in your editor. Find the spec.resources.requests.storage field and change it to your new desired capacity:

# ...
    
    
            spec:
    
    
             resources:
    
    
             requests:
    
    
             storage: 10Gi
    
    
            # ...

Save and close the file. Kubectl should report that the change has been applied to your cluster.

persistentvolumeclaim/data-nginx-0 edited

Now repeat these steps for the StatefulSet's remaining PVCs. Listing your cluster's persistent volumes should then show the new size against each one:

$ kubectl get pv
    
    
            NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM 
    
    
            pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7 10Gi RWO Delete Bound default/data-nginx-2
    
    
            pvc-33af452d-feff-429d-80cd-a45232e700c1 10Gi RWO Delete Bound default/data-nginx-0
    
    
            pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5 10Gi RWO Delete Bound default/data-nginx-1

The claims will maintain the old size for now:

$ kubectl get pvc
    
    
            NAME STATUS VOLUME CAPACITY ACCESS MODES
    
    
            data-nginx-0 Bound pvc-33af452d-feff-429d-80cd-a45232e700c1 10Gi RWO 
    
    
            data-nginx-1 Bound pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5 10Gi RWO 
    
    
            data-nginx-2 Bound pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7 10Gi RWO

This is because the volume can't be resized while Pods are still using it.

Recreating the StatefulSet

Complete the resize by releasing the volume claim from the StatefulSet that's holding it. Delete the StatefulSet but use the orphan cascading mechanism so its Pods remain in your cluster. This will help minimize downtime.

$ kubectl delete statefulset --cascade=orphan nginx
    
    
            statefulset.apps "nginx" deleted

Next edit your original YAML file to include the new volume size in the spec.resources.requests.storage file. Then use kubectl apply to recreate the StatefulSet in your cluster:

$ kubectl apply -f ss.yaml
    
    
            service/nginx unchanged
    
    
            statefulset.apps/nginx created

The new StatefulSet will assume ownership of the previously orphaned Pods because they'll already meet its requirements. The volumes may get resized at this point but in most cases you'll have to manually initiate a rollout that restarts your Pods:

$ kubectl rollout restart statefulset nginx

The rollout proceeds sequentially, targeting one Pod at a time. This ensures your service remains accessible throughout.

Now your PVCs should show the new size:

$ kubectl get pvc
    
    
            NAME STATUS VOLUME CAPACITY ACCESS MODES
    
    
            data-nginx-0 Bound pvc-33af452d-feff-429d-80cd-a45232e700c1 10Gi RWO 
    
    
            data-nginx-1 Bound pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5 10Gi RWO 
    
    
            data-nginx-2 Bound pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7 10Gi RWO

Try connecting to one of your Pods to check the increased capacity is visible from within:

$ kubectl exec -it nginx-0 bash
    
    
            root@nginx-0:/# df -h /usr/share/nginx/html
    
    
            Filesystem Size Used Avail Use% Mounted on
    
    
            /dev/disk/by-id/scsi-0DO_Volume_pvc-33af452d-feff-429d-80cd-a45232e700c1 9.9G 4.5M 9.4G 1% /usr/share/nginx/html

The Pod's reporting the expected 10 Gi of storage.

Summary

Kubernetes StatefulSets let you run stateful applications in Kubernetes with persistent storage volumes that are scoped to individual Pods. However the flexibility this permits ends when you need to resize one of your volumes. This is a missing feature which currently requires several manual steps to be completed in sequence.

The Kubernetes maintainers are aware of the issue. There's an open feature request to develop a solution which should eventually let you initiate volume resizes by editing a StatefulSet's manifest. This will be much quicker and safer than the current situation.

One final caveat is that volume resizes are dependent on a storage driver that permits dynamic expansion. This feature only became generally available in Kubernetes v1.24 and not all drivers, Kubernetes distributions, and cloud platforms will support it. You can check whether yours does by running kubectl get sc and looking for true in the ALLOWVOLUMEXPANSION column of the storage driver you're using with your StatefulSets.