Kubernetes Architecture Notes for AI Workloads

Introduction

Running AI workloads on Kubernetes introduces unique challenges that differ significantly from traditional application deployment. GPU scheduling, model loading patterns, and observability all require specialized approaches. This document captures our architectural decisions and the rationale behind them.

Core Architecture

Our K3s cluster for AI workloads follows a layered architecture:

                    ┌──────────────────────┐
                    │     Traefik L7       │
                    │   Ingress / TLS      │
                    └──────────┬───────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
       ┌────────────┐  ┌────────────┐  ┌────────────┐
       │  Hermes    │  │  LiteLLM   │  │  Cron Jobs │
       │  Agents    │  │  Gateway   │  │  (scheduled│
       │  (workloads│  │            │  │   tasks)   │
       └────────────┘  └────────────┘  └────────────┘
              │                │                │
              ▼                ▼                ▼
       ┌────────────────────────────────────────────┐
       │              K3s Cluster                   │
       │  ┌──────────┐  ┌──────────┐               │
       │  │ Control  │  │  Worker  │  (gamorcloud01)│
       │  │  Plane   │  │  Node    │               │
       │  └──────────┘  └──────────┘               │
       │  ┌──────────┐  ┌──────────┐               │
       │  │  GPU     │  │  GPU     │  (gx10-db4d)  │
       │  │  Worker  │  │  Worker  │               │
       │  └──────────┘  └──────────┘               │
       └────────────────────────────────────────────┘

Node Separation Strategy

We split the cluster into two specialized nodes:

Control Plane Node (gamorcloud01)

• Role: API server, controller manager, scheduler
• CPU: General-purpose (AMD EPYC / Intel Xeon)
• RAM: 32 GB
• Storage: 200 GB NVMe
• Workloads: Control plane components, lightweight agents, monitoring

GPU Node (gx10-db4d)

• Role: AI workloads, model serving, inference
• GPU: NVIDIA L4 (24 GB VRAM) or A100 (80 GB VRAM)
• RAM: 64 GB minimum
• Storage: 500 GB NVMe for model weights
• Workloads: LiteLLM, model servers, GPU-intensive jobs

Why Separate?

1. GPU scheduling: NVIDIA GPU Operator uses node selectors and tolerations
2. Resource contention: AI workloads spike GPU memory — don’t share with control plane
3. Cost efficiency: GPU nodes are expensive; keep control plane on cheaper hardware
4. Fault isolation: A GPU node failure shouldn’t take down the control plane

GPU Scheduling

NVIDIA GPU Operator

The NVIDIA GPU Operator automates GPU driver installation, device plugin, and DCGM:

# Install GPU Operator
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set toolkit.enabled=true \
  --set driver.enabled=false \
  --set devicePlugin.enabled=true \
  --set dcgm.enabled=true

Node Labels and Tolerations

GPU nodes receive special labels:

apiVersion: v1
kind: Node
metadata:
  name: gx10-db4d
  labels:
    nvidia.com/gpu.deploy-gpu-feature: "true"
    nvidia.com/gpu.deploy-container-engine-launcher: "true"
    nvidia.com/gpu.deploy-dcgm: "true"
    ai-workload: "true"
    kubernetes.io/os: linux
  taints:
  - key: nvidia.com/gpu
    effect: NoSchedule
    value: present

Pods that need GPUs add the matching toleration:

tolerations:
- key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"

Resource Requests

Every AI pod must request GPU resources:

resources:
  requests:
    nvidia.com/gpu: "1"
    memory: "16Gi"
    cpu: "4"
  limits:
    nvidia.com/gpu: "1"
    memory: "24Gi"
    cpu: "8"

Without the nvidia.com/gpu request, the pod won’t be scheduled on a GPU node.

Model Serving Architecture

Model Registry Pattern

Store model weights in a shared directory with proper node affinity:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: model-registry
spec:
  capacity:
    storage: 500Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: nvidia-ai-store
  local:
    path: /data/model-registry
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - gx10-db4d
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-registry-pvc
  namespace: ia-services-payed
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: nvidia-ai-store
  resources:
    requests:
      storage: 500Gi
  volumeName: model-registry

Model Server Comparison

Our Choice: vLLM + LiteLLM

We use vLLM for actual inference and LiteLLM as the API gateway:

# vLLM deployment (runs on GPU node)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-serving
  namespace: ia-services-payed
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.3
        args:
        - --model
        - "nvidia/Nemotron-4-35B"
        - --tensor-parallel-size
        - "1"
        - --max-model-len
        - "8192"
        resources:
          limits:
            nvidia.com/gpu: "1"
        volumeMounts:
        - name: model-cache
          mountPath: /data/models
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-registry-pvc

Observability Stack

GPU Monitoring with DCGM

DCGM exports GPU metrics to Prometheus:

# Prometheus scrape config (added to prometheus.yaml)
- job_name: 'dcgm-exporter'
  static_configs:
  - targets: ['gpu-operator-dcgm-exporter.gpu-operator:9400']

Key DCGM metrics:

Request-Level Observability

Each AI request should be traced:

# Add tracing headers to requests
import requests
import uuid

request_id = str(uuid.uuid4())
headers = {
    "X-Request-ID": request_id,
    "X-Model": "nemotron-35b",
    "X-Consumer": "hermes-agent",
}

response = requests.post(
    "http://litellm.ia-services-payed.svc.cluster.local:4000/v1/completions",
    headers=headers,
    json=payload,
)

Centralized Logging

All components log in JSON format for easy parsing:

# Container log config
env:
- name: LOG_LEVEL
  value: "info"
- name: LOG_FORMAT
  value: "json"
- name: LOG_OUTPUT
  value: "stdout"

Network Architecture

Ingress Configuration

Traefik handles external routing with automatic TLS:

# TraefikMiddleware for rate limiting
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: rate-limit
  namespace: ia-services-payed
spec:
  rateLimit:
    average: 100
    burst: 200
    period: 1s
---
# TraefikIngressRoute
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: litellm-ingress
  namespace: ia-services-payed
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`intranet.cris-mora-cv.es`) && PathPrefix(`/api/litellm`)
    kind: Rule
    services:
    - name: litellm
      port: 4000
    middlewares:
    - name: rate-limit
    - name: strip-prefix
  tls:
    secretName: intranet-tls

Service Mesh Considerations

For multi-cluster setups, consider:

• K3s built-in services: Sufficient for single-cluster
• Istio/Linkerd: Adds complexity, not worth it for most setups
• Traefik + cert-manager: Our choice — simple, effective, Kubernetes-native

High Availability

Model Server HA

For production, run multiple replicas behind a service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-ha
  namespace: ia-services-payed
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Auto-Scaling with KEDA

Scale model servers based on request queue depth:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: litellm-autoscaler
  namespace: ia-services-payed
spec:
  scaleTargetRef:
    name: litellm-ha
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-monitoring.svc.cluster.local:9090
      query: |
        sum(rate(litellm_proxy_request_count[2m]))
      threshold: "50"

Cost Considerations

GPU Cost Optimization

Cost Monitoring

Track GPU hours and token usage:

# GPU hours per day
sum(increase(nvidia_gpu_utilization[1d])) / 60 / 60

# Cost per model
sum(rate(litellm_proxy_token_usage[1h])) * model_price_per_token

Conclusion

The key architectural principles for AI workloads on Kubernetes:

1. Separate GPU from control plane — avoid resource contention
2. Use the GPU Operator — it automates driver, device plugin, and DCGM
3. Request GPU resources explicitly — pods without GPU requests won’t schedule
4. Monitor everything — GPU metrics, request latency, token usage
5. Start small, scale when needed — K3s + one GPU node is enough to start

The architecture scales from a single Raspberry Pi to a multi-node cluster with GPU workers. The Kubernetes primitives (Deployments, Services, PVs) remain the same regardless of scale.