Logs and events

best practices

1. use gitops - argocd for kubernetes deployments

2. add persistent volume for loki and grafana

By default, if you don’t enable persistence for Loki, it stores logs only in the container’s ephemeral storage. (That means any pod restart or node restart will lose logs.)

Add persistent storage

loki:
  persistence:
    enabled: true
    storageClassName: ebs-csi        # your EKS storage class
    size: 50Gi                        # adjust based on log volume
    accessModes:
      - ReadWriteOnce

1. add retention policy

policy

loki:
  config:
    table_manager:
      retention_deletes_enabled: true
      retention_period: 7d

we are collecting pods logs using promtail here

here is our helm setup

Readme.md

helm show values grafana/loki-stack > values.yamlhelm install loki 

helm repo add grafana https://grafana.github.io/helm-charts


helm install loki grafana/loki-stack -f values.yml --set promtail.enabled=true -n monitoring --create-namespace

helm list -n monitoring

kubectl get pods -n monitoring

helm upgrade loki grafana/loki-stack -f values.yaml --set promtail.enabled=true -n monitoring 

Access grafana using Cloudflared tunnel

add service <service-name>.<namespace>:port

# get grafana password
kubectl get secret loki-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode

Values.yml

test_pod:
  enabled: true
  image: bats/bats:1.8.2
  pullPolicy: IfNotPresent

loki:
  enabled: true
  isDefault: true
  image:
    tag: 2.9.3
  url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  datasource:
    jsonData: "{}"
    uid: ""


promtail:
  enabled: true
  config:
    logLevel: info
    serverPort: 3101
    clients:
      - url: http://{{ .Release.Name }}:3100/loki/api/v1/push

fluent-bit:
  enabled: false

grafana:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClassName: ebs-csi
    accessModes:
      - ReadWriteOnce
  sidecar:
    datasources:
      label: ""
      labelValue: ""
      enabled: true
      maxLines: 1000
  image:
    tag: 10.3.3
  plugins:
    - grafana-clickhouse-datasource 4.8.0
    - redis-datasource

prometheus:
  enabled: false
  isDefault: false
  url: http://{{ include "prometheus.fullname" .}}:{{ .Values.prometheus.server.service.servicePort }}{{ .Values.prometheus.server.prefixURL }}
  datasource:
    jsonData: "{}"

filebeat:
  enabled: false
  filebeatConfig:
    filebeat.yml: |
      # logging.level: debug
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      output.logstash:
        hosts: ["logstash-loki:5044"]

logstash:
  enabled: false
  image: grafana/logstash-output-loki
  imageTag: 1.0.1
  filters:
    main: |-
      filter {
        if [kubernetes] {
          mutate {
            add_field => {
              "container_name" => "%{[kubernetes][container][name]}"
              "namespace" => "%{[kubernetes][namespace]}"
              "pod" => "%{[kubernetes][pod][name]}"
            }
            replace => { "host" => "%{[kubernetes][node][name]}"}
          }
        }
        mutate {
          remove_field => ["tags"]
        }
      }
  outputs:
    main: |-
      output {
        loki {
          url => "http://loki:3100/loki/api/v1/push"
          #username => "test"
          #password => "test"
        }
        # stdout { codec => rubydebug }
      }

# proxy is currently only used by loki test pod
# Note: If http_proxy/https_proxy are set, then no_proxy should include the
# loki service name, so that tests are able to communicate with the loki
# service.
proxy:
  http_proxy: ""
  https_proxy: ""
  no_proxy: ""

updated values.yml

values.yml

pesistent volume

storageclass

sc-ebs3,yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-expandable
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
  encrypted: "true"
  fsType: ext4

kubectl apply -f ebs-gp3-expandable.yaml

storageclass.storage.k8s.io/ebs-gp3-expandable created

this Storage class automatically provision new ebs for pod if we call it

use in pod helm values.yml

loki:
  persistence:
    enabled: true
    storageClassName: ebs-gp3-expandable   # switched to expandable SC
    size: 20Gi
    accessModes:
      - ReadWriteOnce

Redeploy Loki & Grafana

Because StatefulSets don’t allow changing PVC storageClass, you’ll need to recreate Loki & Grafana PVCs:

helm uninstall loki -n monitoring
kubectl delete pvc -n monitoring -l app.kubernetes.io/instance=loki
helm install loki grafana/loki-stack -f values.yml -n monitoring

if i do this i will loose all my custom dashboards in grafana can i keep grafana as it is i am ok if i loose loki data

Exactly ✅ you spotted the key point.

• Grafana PVC holds your dashboards, users, and settings. If you delete that PVC, you’ll lose all custom dashboards.

• Loki PVC only holds logs (which are ephemeral by nature) — safe to lose if you’re okay with resetting log history.

So yes, you can keep Grafana as-is and only redeploy Loki.

do not update ebs driver is grafana keep old one for our old backup/data

safe

delete only loki

kubectl scale sts loki --replicas=0 -n monitoring kubectl delete sts loki -n monitoring

redeploy

helm upgrade --install loki grafana/loki-stack -f values.yml -n monitoring

now we have pvc created for loki as well

filter logs by json

filter logs for alert

rishabh exposed logs in json format

working below

{job="default/vgerp-master-rust"} | json  | level="INFO"

logs below 
-----------------------
{"timestamp":"2025-09-19T07:53:20.013147Z","level":"INFO","fields":{"message":"HTTP request completed","client_ip":"171.6.87.180","method":"GET","path":"/api/productiontracking/fetch/order/barcodeid","status":200,"duration_ms":"3"}}
---------------------
i am able to filter log level
using 
{job="default/vgerp-master-rust"} | json  | level="INFO"

{
  "timestamp":"2025-09-19T07:53:20.013147Z",
  "level":"INFO",
  "fields":{
    "message":"HTTP request completed",
    "client_ip":"171.6.87.180",
    "method":"GET",
    "path":"/api/productiontracking/fetch/order/barcodeid",
    "status":200,
    "duration_ms":"3"
  }
}

reference: https://signoz.io/guides/loki-json-logs-filter-by-detected-fields-from-grafana/

working query

{job="default/vgerp-master-rust"} 
| json 
| line_format "{{.fields.status}}" 
|~ "400"

above command just highlight show full logs where status is 400

Kubernetes Event by Prom tail

our grafana/loki-stack helm chart support service account

we need service account to trust promtail pod to give access to Kubernetes api via role and clusterrolebinding

kubectl get clusterrole,clusterrolebinding -A | grep promtail

clusterrole.rbac.authorization.k8s.io/loki-promtail 2025-01-29T09:59:00Z clusterrolebinding.rbac.authorization.k8s.io/loki-promtail

oki-Grafana (main) $ kubectl describe clusterrole loki-promtail Name: loki-promtail Labels: app.kubernetes.io/instance=loki app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=promtail app.kubernetes.io/version=2.9.3 helm.sh/chart=promtail-6.15.5 Annotations: meta.helm.sh/release-name: loki meta.helm.sh/release-namespace: monitoring PolicyRule: Resources Non-Resource URLs Resource Names Verbs

---

endpoints [] [] [get watch list] nodes/proxy [] [] [get watch list] nodes [] [] [get watch list] pods [] [] [get watch list] services [] [] [get watch list]

step 1

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: loki-promtail-events
rules:
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["events.k8s.io"]
  resources: ["events"]
  verbs: ["get", "list", "watch"]


---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: loki-promtail-events
subjects:
- kind: ServiceAccount
  name: loki-promtail
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: loki-promtail-events
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f promtail-events-rbac.yaml

(This gives the loki-promtail ServiceAccount permission to watch both legacy and events.k8s.io events.)

test_pod:
  enabled: true
  image: bats/bats:1.8.2
  pullPolicy: IfNotPresent

loki:
  enabled: true
  isDefault: true
  image:
    tag: 2.9.3
  url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  persistence:
    enabled: true
    storageClassName: ebs-gp3-expandable   # switched to expandable SC
    size: 20Gi
    accessModes:
      - ReadWriteOnce
  config:
    table_manager:
      retention_deletes_enabled: true
      retention_period: 7d
  datasource:
    jsonData: "{}"
    uid: ""


promtail:
  enabled: true
  serviceAccount:
    create: true
    name: loki-promtail
  rbac:
    create: true
  config:
    logLevel: info
    clients:
      - url: http://loki:3100/loki/api/v1/push
    positions:
      filename: /tmp/positions.yaml

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
          - json:
              expressions:
                client_ip: fields.client_ip
                method: fields.method
                path: fields.path
                status: fields.status
                duration_ms: fields.duration_ms
          # convert numeric fields to string
          - output:
              source: client_ip
          - output:
              source: method
          - output:
              source: path
          - output:
              source: status
          - output:
              source: duration_ms
          - labels:
              client_ip:
              method:
              path:
              status:
              duration_ms:

      - job_name: kubernetes-events
        kubernetes_sd_configs:
          - role: event
        pipeline_stages:
          # event objects are JSON; parse top-level fields but NOT the full message into labels
          - json:
              expressions:
                reason: reason
                type: type
                involvedKind: involvedObject.kind
                involvedName: involvedObject.name
                message: message
                firstTimestamp: firstTimestamp
                lastTimestamp: lastTimestamp
        relabel_configs:
          # label events for easy querying (avoid huge cardinality)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_event_reason]
            target_label: reason
          - source_labels: [__meta_kubernetes_event_type]
            target_label: event_type
          - source_labels: [__meta_kubernetes_event_involvedObject_kind]
            target_label: involved_kind
          - source_labels: [__meta_kubernetes_event_involvedObject_name]
            target_label: involved_name
          # set job label (optional)
          - target_label: job
            replacement: kubernetes-events


fluent-bit:
  enabled: false

grafana:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClassName: ebs-csi
    accessModes:
      - ReadWriteOnce
  sidecar:
    datasources:
      label: ""
      labelValue: ""
      enabled: true
      maxLines: 1000
  image:
    tag: 10.3.3
  plugins:
    - grafana-clickhouse-datasource 4.8.0
    - redis-datasource

prometheus:
  enabled: false
  isDefault: false
  url: http://{{ include "prometheus.fullname" .}}:{{ .Values.prometheus.server.service.servicePort }}{{ .Values.prometheus.server.prefixURL }}
  datasource:
    jsonData: "{}"

filebeat:
  enabled: false
  filebeatConfig:
    filebeat.yml: |
      # logging.level: debug
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      output.logstash:
        hosts: ["logstash-loki:5044"]

logstash:
  enabled: false
  image: grafana/logstash-output-loki
  imageTag: 1.0.1
  filters:
    main: |-
      filter {
        if [kubernetes] {
          mutate {
            add_field => {
              "container_name" => "%{[kubernetes][container][name]}"
              "namespace" => "%{[kubernetes][namespace]}"
              "pod" => "%{[kubernetes][pod][name]}"
            }
            replace => { "host" => "%{[kubernetes][node][name]}"}
          }
        }
        mutate {
          remove_field => ["tags"]
        }
      }
  outputs:
    main: |-
      output {
        loki {
          url => "http://loki:3100/loki/api/v1/push"
          #username => "test"
          #password => "test"
        }
        # stdout { codec => rubydebug }
      }

# proxy is currently only used by loki test pod
# Note: If http_proxy/https_proxy are set, then no_proxy should include the
# loki service name, so that tests are able to communicate with the loki
# service.
proxy:
  http_proxy: ""
  https_proxy: ""
  no_proxy: ""

# 1) Apply RBAC so promtail can watch events
kubectl apply -f promtail-events-rbac.yaml

# 2) (Optional) preview the rendered manifests
helm template loki grafana/loki-stack -f values.yaml -n monitoring | less

# 3) Upgrade/install the chart with your modified values.yaml
helm upgrade --install loki grafana/loki-stack -f values.yaml -n monitoring --create-namespace

# 4) Wait / check pods
kubectl get pods -n monitoring -w
# or
kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail

Steps to collect event by promtail

Perfect 👍 you already confirmed that the Helm chart created ServiceAccounts for:

• Grafana → loki-grafana

• Loki → loki

• Promtail → loki-promtail

So yes — the chart does handle ServiceAccounts automatically.

Now the missing piece is: RBAC rules. ServiceAccounts by themselves don’t have permissions. Promtail needs ClusterRole + ClusterRoleBinding so it can get, list, watch things like Pods, Namespaces, and Events.

---

🔍 How to check RBAC created by Helm

Run:

kubectl get clusterrole,clusterrolebinding -A | grep promtail

You should see something like:

clusterrole.rbac.authorization.k8s.io/loki-promtail
clusterrolebinding.rbac.authorization.k8s.io/loki-promtail

Then inspect:

kubectl describe clusterrole loki-promtail

Expected rules include:

rules:
- apiGroups: [""]
  resources:
  - pods
  - nodes
  - namespaces
  - events   # <- this is the key one for Kubernetes events
  verbs: ["get", "list", "watch"]

---

✅ If events is missing

If your loki-promtail ClusterRole doesn’t include events, you need to extend it. You can do this either by:

1. Editing values.yaml Some Helm charts expose RBAC rules as configurable (check with:

   helm show values grafana/loki-stack | less

   and search for rbac).

2. Creating a custom ClusterRole patch For example:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: loki-promtail-extra
   rules:
   - apiGroups: [""]
     resources: ["events"]
     verbs: ["get", "list", "watch"]
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: loki-promtail-extra
   subjects:
   - kind: ServiceAccount
     name: loki-promtail
     namespace: monitoring
   roleRef:
     kind: ClusterRole
     name: loki-promtail-extra
     apiGroup: rbac.authorization.k8s.io

   Apply it with kubectl apply -f.

---

👉 Do you want me to write you the final Promtail values.yaml snippet (with config + RBAC for events) so you can drop it into your Helm setup? That way you won’t need to manually patch later.