Running Qwen 3.6 35B A3B on K3s: Lessons Learned

Introduction

After running Qwen 3.6 35B A3B on K3s in production for several weeks, we’ve gathered a set of hard-won lessons. This post shares our operational experience, performance benchmarks, and the tweaks that made the difference between a clunky demo and a reliable service.

Model Selection Rationale

We chose Qwen 3.6 35B A3B because it offers an excellent balance of capability and resource efficiency:

• 35B parameters: Enough capability for complex reasoning tasks
• A3B (3-bit activation): Reduced memory footprint compared to full-precision models
• Open weights: No vendor lock-in, easy to update and iterate
• Mistral-compatible: Works with existing tooling and evaluation pipelines

Hardware Requirements

Minimum Viable Setup

Our single-node setup uses a machine with an NVIDIA L4 GPU (24 GB VRAM) and 32 GB RAM. This handles the model comfortably with room for concurrent requests.

Memory Optimization Techniques

1. Quantization Strategy

We experimented with several quantization approaches:

# Example quantization configuration for vLLM
quantization_config = {
    "quantization": "bitsandbytes",
    "load_in_4bit": True,
    "bnb_4bit_quant_type": "nf4",
    "bnb_4bit_compute_dtype": "float16",
}

4-bit quantization reduced VRAM usage from ~70 GB (FP16) to ~20 GB — fitting comfortably within the L4’s 24 GB limit.

2. vLLM PagedAttention

Using vLLM’s PagedAttention mechanism improved memory utilization:

• Without PagedAttention: ~15 concurrent requests before OOM
• With PagedAttention: ~30+ concurrent requests

3. K3s Resource Limits

Properly configuring Kubernetes resource limits was critical:

resources:
  requests:
    nvidia.com/gpu: "1"
    memory: "8Gi"
    cpu: "2"
  limits:
    nvidia.com/gpu: "1"
    memory: "12Gi"
    cpu: "4"

The GPU limit is the most important — without it, the scheduler can’t place the pod on a GPU node.

Performance Benchmarks

Inference Latency

Key insight: batching dramatically improves throughput but increases P99 latency. For interactive use cases (chat), keep batch size at 1-4.

GPU Utilization

Monitoring with nvidia-smi dmon showed:

• Average GPU compute: 85-92% during active serving
• VRAM usage: ~18-20 GB (well under the 24 GB limit)
• GPU memory fragmentation: minimal with PagedAttention

Common Pitfalls and Fixes

Pitfall 1: GPU Not Detected

Symptom: Pods stuck in Pending state with Insufficient nvidia.com/gpu.

Fix: Verify the GPU operator is running:

kubectl get pods -n gpu-operator
kubectl logs -n gpu-operator -l app.kubernetes.io/component=gpu-feature-discovery

Pitfall 2: OOMKilled on CPU

Symptom: Pod crashes with OOMKilled despite low GPU usage.

Fix: The model loading phase spikes CPU memory. Increase CPU requests:

resources:
  requests:
    memory: "8Gi"  # Increased from 4Gi
    cpu: "2"       # Increased from 1

Pitfall 3: Model Loading Timeout

Symptom: Readiness probe fails because the model takes longer to load than the timeout allows.

Fix: Use liveness probes instead of readiness for initial startup:

livenessProbe:
  httpGet:
    path: /health
    port: 4000
  initialDelaySeconds: 120
  periodSeconds: 30

K3s-Specific Tips

Local PV for Model Cache

Store model weights on a local persistent volume to avoid repeated downloads:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: model-cache
spec:
  capacity:
    storage: 80Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  local:
    path: /data/models
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - gamorcloud01

NetworkPolicy for Model Gateway

Protect your LiteLLM gateway with a NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: litellm-policy
  namespace: ia-services-payed
spec:
  podSelector:
    matchLabels:
      app: litellm
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: traefik
    ports:
    - port: 4000

Monitoring Setup

We use a simple but effective monitoring stack:

1. nvidia-dcgm: GPU metrics (temperature, memory, utilization)
2. Prometheus: Scrapes DCGM and K3s metrics
3. Grafana: Dashboards for GPU health and model performance

The critical Grafana panels to watch:

• GPU Memory Utilization — stay under 85%
• GPU Temperature — alert at 85°C
• Request Latency — track P50 and P99
• GPU Utilization — should be above 70% during active serving

Conclusion

Running Qwen 3.6 35B A3B on K3s is entirely feasible on modest hardware, but it requires careful attention to GPU memory, resource limits, and monitoring. The biggest lesson: start with the hardware constraints and work backward, not the other way around. Quantization, proper resource limits, and monitoring are your best friends.