Persistent volume

StatefulSets are mostly immutable. You cannot change fields like: volumeClaimTemplates (size, storageClass) after creation Certain spec fields other than replicas, template, updateStrategy, minReadySeconds Helm tries to apply your values.yaml → Kubernetes rejects it because the StatefulSet spec changed in an immutable way.

AllowVolumeExpansion: → behaves as false

your current ebs-csi StorageClass is missing

Prerequisites:

ebs csi driver installed and configured

storageclass

storageclass provisioning

sc-ebs3,yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-expandable
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
  encrypted: "true"
  fsType: ext4

kubectl apply -f ebs-gp3-expandable.yaml

storageclass.storage.k8s.io/ebs-gp3-expandable created

this Storage class automatically provision new ebs for pod if we call it

use in pod helm values.yml

loki:
  persistence:
    enabled: true
    storageClassName: ebs-gp3-expandable   # switched to expandable SC
    size: 20Gi
    accessModes:
      - ReadWriteOnce

Redeploy Loki & Grafana

Because StatefulSets don’t allow changing PVC storageClass, you’ll need to recreate Loki & Grafana PVCs:

helm uninstall loki -n monitoring
kubectl delete pvc -n monitoring -l app.kubernetes.io/instance=loki
helm install loki grafana/loki-stack -f values.yml -n monitoring

if i do this i will loose all my custom dashboards in grafana can i keep grafana as it is i am ok if i loose loki data

Exactly ✅ you spotted the key point.

• Grafana PVC holds your dashboards, users, and settings. If you delete that PVC, you’ll lose all custom dashboards.

• Loki PVC only holds logs (which are ephemeral by nature) — safe to lose if you’re okay with resetting log history.

So yes, you can keep Grafana as-is and only redeploy Loki.

do not update ebs driver is grafana keep old one for our old backup/data

safe

delete only loki

kubectl scale sts loki --replicas=0 -n monitoring kubectl delete sts loki -n monitoring

redeploy

helm upgrade --install loki grafana/loki-stack -f values.yml -n monitoring

now we have pvc created for loki as well