Introduction

K3s has become the go-to lightweight Kubernetes distribution for edge computing, CI/CD pipelines, and AI experimentation. Its simplicity makes it ideal for teams that want Kubernetes power without the operational complexity of a full production cluster. This post walks you through setting up your first AI workload on K3s.

Why K3s for AI?

Before diving into setup, let’s understand why K3s is a compelling choice for AI experimentation:

  • Minimal resource footprint: Runs on a single Raspberry Pi with 4GB RAM
  • Single binary: No etcd dependency — SQLite or PostgreSQL for storage
  • Offline capability: Can pull images and install addons in disconnected environments
  • Kubernetes compatibility: 99% of K8s APIs work the same way

Prerequisites

You’ll need:

  • A Linux machine (Ubuntu 22.04+, Debian 12+, or similar)
  • At least 4GB RAM (8GB recommended for model serving)
  • Docker or containerd installed
  • Basic familiarity with Kubernetes concepts

Cluster Bootstrap

The fastest way to get started is with the one-line installer:

curl -sfL https://get.k3s.io | sh -

This gives you a fully functional Kubernetes cluster in under 60 seconds. You can verify with:

sudo kubectl get nodes
sudo kubectl get pods -A

GPU Operator Setup

For AI workloads, you’ll need GPU support. K3s makes this straightforward:

# Install the NVIDIA GPU Operator via Helm
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace

Verify GPU detection:

kubectl describe node | grep -A 5 NVIDIA

Deploying a Model Server

Once your GPU is recognized, deploy a model server. We use LiteLLM as our gateway:

# litellm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
  namespace: ia-services-payed
spec:
  replicas: 1
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
      - name: litellm
        image: ghcr.io/anthropics/litellm:main-stable
        ports:
        - containerPort: 4000
        env:
        - name: MODEL_LIST
          value: |
            - model_name: qwen-35b
              litellm_params:
                model: nvidia/nemotron-3-ultra-550b-a55b
        resources:
          limits:
            nvidia.com/gpu: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: litellm
  namespace: ia-services-payed
spec:
  selector:
    app: litellm
  ports:
  - port: 4000
    targetPort: 4000
  type: ClusterIP

Apply it:

kubectl apply -f litellm-deployment.yaml

Testing the Connection

Use curl to verify your model gateway:

curl http://litellm.ia-services-payed.svc.cluster.local:4000/v1/models

Next Steps

Now that your cluster is running, consider:

  • Setting up Traefik ingress for external access
  • Configuring cert-manager for TLS
  • Adding monitoring with Prometheus and Grafana
  • Implementing backup strategies with Velero

Conclusion

K3s lowers the barrier to AI experimentation dramatically. You can go from zero to serving models in under an hour, and the Kubernetes compatibility means your skills transfer to production clusters seamlessly. The key insight: start small, validate your workflows, and scale only when needed.