Cluster Operations: What We Learned

Introduction

Running a K3s cluster that serves AI models 24/7 has taught us a lot about operational realities. This post documents the incidents we’ve faced, the lessons learned, and the operational playbooks that now keep things running smoothly.

Incident Timeline

Incident #1: GPU OOM on Model Load

Date: 2026-06-15 Severity: P2 (partial outage) Duration: 45 minutes

What happened: A model update increased VRAM requirements. The LiteLLM pod crashed repeatedly with OOMKilled on GPU memory.

Root cause: Resource limits were set based on the old model’s requirements. The new Nemotron-4 model needed ~20 GB VRAM, but the container limit was set to 16 GB.

Resolution:

# Updated resource limits
resources:
  limits:
    nvidia.com/gpu: "1"
    memory: "24Gi"  # Increased from 16Gi

Lesson: GPU memory limits must match model requirements exactly. Add a monitoring alert for GPU memory at 90% capacity.

Incident #2: etcd Database Bloat

Date: 2026-06-28 Severity: P3 (degraded performance) Duration: 2 hours

What happened: K3s control plane became slow. API response times went from 50ms to 3000ms+.

Root cause: etcd database exceeded the default compaction threshold. With frequent CronJob deployments, the object count grew rapidly.

Resolution:

# Check etcd size
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/server.key \
  endpoint health

# Trigger compaction
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/server.key \
  defrag

# Check etcd status
kubectl get etcdmembers.etcd.database.coreos.com -n k3s

Lesson: Set up automated etcd backups and compaction. Monitor etcd database size with a Grafana alert at 500 MB.

Incident #3: Network Policy Lockout

Date: 2026-07-05 Severity: P1 (complete outage) Duration: 30 minutes

What happened: All agent pods could not reach LiteLLM gateway.

Root cause: A NetworkPolicy update accidentally blocked traffic from the forgejo-runners namespace to ia-services-payed.

Resolution:

# Temporary fix: remove the policy
kubectl delete networkpolicy litellm-policy -n ia-services-payed

# Verify connectivity
kubectl run --rm -it --restart=Never --namespace=forgejo-runners \
  temp-test --image=busybox -- \
  wget -qO- --timeout=5 http://litellm.ia-services-payed.svc.cluster.local:4000/health

Lesson: Always test NetworkPolicy changes in a staging namespace first. Use a deny-by-default approach and add allow rules one at a time.

Backup Strategy

etcd Backups

K3s provides built-in etcd backup to S3-compatible storage:

# Manual backup
kubectl exec -n k3s k3s-server-0 -- k3s etcd-snapshot save \
  /backup/k3s-etcd-$(date +%Y%m%d).db

# Automated with crontab
0 3 * * * /usr/local/bin/k3s-backup.sh

The backup script:

#!/bin/bash
set -euo pipefail

DATE=$(date +%Y%m%d)
BACKUP_DIR="/backup/k3s-etcd"
S3_BUCKET="s3://company-backups/k3s-etcd"

mkdir -p "$BACKUP_DIR"

# Create snapshot
kubectl exec -n k3s k3s-server-0 -- \
  k3s etcd-snapshot save "${BACKUP_DIR}/k3s-etcd-${DATE}.db"

# Upload to S3
aws s3 cp "${BACKUP_DIR}/k3s-etcd-${DATE}.db" "${S3_BUCKET}/"

# Clean old backups (keep 30 days)
aws s3 ls "${S3_BUCKET}/" --recursive \
  --expires $(date -d '30 days ago' +%Y-%m-%d) \
  | awk '{print $NF}' | xargs -r aws s3 rm

Model Backup

Model weights should be backed up separately:

#!/bin/bash
# backup-models.sh

MODEL_DIR="/data/model-registry"
BACKUP_DIR="/backup/models"

rsync -avz --delete "$MODEL_DIR/" "$BACKUP_DIR/models-$(date +%Y%m%d)/"

# Verify backup
echo "Backup size: $(du -sh "$BACKUP_DIR/models-$(date +%Y%m%d)/" | cut -f1)"

Disaster Recovery

Test recovery quarterly:

# 1. Create a test cluster on a different node
curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat /var/lib/rancher/k3s/server/node-token) sh -

# 2. Restore etcd from backup
kubectl exec -n k3s k3s-server-0 -- \
  k3s etcd-snapshot restore --dir=/backup/k3s-etcd/latest \
  /var/lib/rancher/k3s/server/db

# 3. Verify services
kubectl get pods -A
kubectl logs -n ia-services-payed -l app=litellm --tail=10

Scaling Strategy

Vertical Scaling (Already Running)

When a single node’s resources are exhausted:

Add GPU memory: Upgrade from L4 (24 GB) to A100 (80 GB)
Add RAM: Increase from 32 GB to 64 GB
Add storage: NVMe upgrade for model registry

Horizontal Scaling (Multiple Nodes)

When concurrent request volume exceeds capacity:

# Add a second GPU node
apiVersion: v1
kind: Node
metadata:
  name: gx10-db4d-02
  labels:
    nvidia.com/gpu.deploy-gpu-feature: "true"
    ai-workload: "true"
  taints:
  - key: nvidia.com/gpu
    effect: NoSchedule
    value: present

Update the GPU Operator to recognize the new node — it should auto-discover the GPU and install the device plugin.

Auto-Scaling with KEDA

Scale LiteLLM based on request queue:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: litellm-autoscaler
  namespace: ia-services-payed
spec:
  scaleTargetRef:
    name: litellm-ha
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-monitoring.svc.cluster.local:9090
      query: |
        sum(rate(litellm_proxy_request_count{status!="200"}[2m]))
      threshold: "10"
  cooldownPeriod: 300

Debugging Playbook

Symptom: Pod in `CrashLoopBackOff`

# Step 1: Check events
kubectl describe pod <pod-name> -n ia-services-payed

# Step 2: Check logs
kubectl logs <pod-name> -n ia-services-payed --tail=50

# Step 3: Check resource usage
kubectl top pod <pod-name> -n ia-services-payed

# Step 4: If GPU-related, check GPU health
nvidia-smi

# Step 5: Check DCGM metrics
kubectl port-forward -n gpu-operator \
  pod/gpu-operator-dcgm-exporter 9400:9400
curl http://localhost:9400/metrics | grep XID

Symptom: Model Serving Slow

# Step 1: Check GPU utilization
kubectl exec -n gpu-operator \
  pod/gpu-operator-dcgm-exporter -- \
  dcgm-exporter --help  # verify running

# Step 2: Check model server logs
kubectl logs -n ia-services-payed \
  -l app=vllm --tail=100 | grep -E "error|timeout|slow"

# Step 3: Check network latency between agents and gateway
kubectl run --rm -it --restart=Never --namespace=forgejo-runners \
  latency-test --image=busybox -- \
  time wget -qO- http://litellm.ia-services-payed.svc.cluster.local:4000/health

# Step 4: Check Prometheus metrics
# Query: litellm_proxy_latency_seconds

Symptom: GPU Not Detected

# Step 1: Check GPU operator pods
kubectl get pods -n gpu-operator

# Step 2: Check device plugin
kubectl logs -n gpu-operator \
  $(kubectl get pod -n gpu-operator -l app.kubernetes.io/component=nvidia-device-plugin \
    -o name)

# Step 3: Verify GPU on the node
kubectl describe node gx10-db4d | grep -A 10 NVIDIA

# Step 4: Restart device plugin if needed
kubectl rollout restart deployment/nvidia-device-plugin-daemonset \
  -n gpu-operator

Monitoring Dashboards

Critical Dashboards to Set Up

GPU Overview: All GPU metrics across all nodes
- GPU utilization, temperature, memory
- Power consumption
- XID errors
Model Serving: LiteLLM performance
- Request rate, latency, error rate
- Token usage by model
- Model health status
Cluster Health: K3s cluster status
- Node status, pod health
- etcd size and health
- Network policy status
Agent Operations: Hermes agent metrics
- Cron job success/failure rate
- Agent task completion time
- Task queue depth

Alert Rules

# prometheus-alerting-rules.yaml
groups:
- name: k3s-ai-alerts
  rules:
  - alert: GPUOverheating
    expr: DCGM_FI_DEV_TEMP_GPU > 85
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "GPU overheating on {{ $labels.device }}"

  - alert: GPUVramNearLimit
    expr: DCGM_FI_DEV_MEM_USED / DCGM_FI_DEV_MEM_TOTAL > 0.9
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU memory near limit on {{ $labels.device }}"

  - alert: ModelServerErrorRate
    expr: >
      sum(rate(litellm_proxy_request_error_count[5m]))
      / sum(rate(litellm_proxy_request_count[5m]))
      > 0.05
    for: 3m
    labels:
      severity: critical
    annotations:
      summary: "Model server error rate > 5%"

  - alert: HighModelLatency
    expr: litellm_proxy_latency_seconds_p99 > 30
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "P99 model latency > 30 seconds"

  - alert: EtcdDatabaseLarge
    expr: etcd_server_db_size_bytes > 5e8
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "etcd database size > 500MB"

  - alert: PodCrashLooping
    expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.pod }} is crash looping"

Operational Checklist

Daily

Check Grafana for GPU temperature and memory alerts
Verify CronJob completion status
Review error rates on LiteLLM gateway
Check etcd database size

Weekly

Review model usage and token consumption
Check disk space on model storage
Verify backup success
Review NetworkPolicy effectiveness
Check pod resource utilization

Monthly

Test disaster recovery procedure
Review and update alert thresholds
Audit NetworkPolicy rules
Check GPU driver updates
Review model versions and update if needed
Capacity planning review

Conclusion

The operational reality of AI workloads on K3s comes down to three principles:

Monitor everything — GPU metrics, request latency, token usage, etcd health. If you can’t measure it, you can’t fix it.
Automate backups — etcd snapshots, model registry, and configuration should all be backed up automatically with retention policies.
Document everything — incident post-mortems, debugging playbooks, and operational checklists prevent repeat failures.

The biggest operational lesson: start with comprehensive monitoring before you need it. When your GPU is at 92% memory and requests are timing out, you’ll wish you had those dashboards already set up.

Introduction

Incident Timeline

Incident #1: GPU OOM on Model Load

Incident #2: etcd Database Bloat

Incident #3: Network Policy Lockout

Backup Strategy

etcd Backups

Model Backup

Disaster Recovery

Scaling Strategy

Vertical Scaling (Already Running)

Horizontal Scaling (Multiple Nodes)

Auto-Scaling with KEDA

Debugging Playbook

Symptom: Pod in CrashLoopBackOff

Symptom: Model Serving Slow

Symptom: GPU Not Detected

Monitoring Dashboards

Critical Dashboards to Set Up

Alert Rules

Operational Checklist

Daily

Weekly

Monthly

Conclusion

Symptom: Pod in `CrashLoopBackOff`