LiteLLM and MCP Workflows: A Practical Guide

Introduction

In our K3s AI infrastructure, LiteLLM serves as the central model gateway — a proxy that unifies multiple model providers behind a single OpenAI-compatible API. Combined with the Model Context Protocol (MCP), it becomes the backbone for agent-to-model communication. This post documents our production setup and the lessons learned along the way.

Architecture Overview

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Hermes     │────▶│   LiteLLM    │────▶│   NVIDIA     │
│   Agents     │     │   Gateway    │     │   Nemotron   │
└──────────────┘     │              │     └──────────────┘
┌──────────────┐     │  (port 4000) │     ┌──────────────┐
│   Cron Jobs  │────▶│              │────▶│   Mistral    │
└──────────────┘     └──────────────┘     └──────────────┘
┌──────────────┐           ▲               ┌──────────────┐
│   Webhooks   │───────────┘               │   OpenAI     │
└──────────────┘   MCP Endpoint           └──────────────┘
                    /mcp

LiteLLM Deployment on K3s

Configuration

Our LiteLLM deployment uses a model list configuration that routes requests to different providers based on the task:

# litellm-config.yaml
model_list:
  # Primary model for coding tasks
  - model_name: hermes-coder
    litellm_params:
      model: nvidia/nemotron-3-ultra-550b-a55b
      api_base: https://integrate.api.nvidia.com/v1
      api_key: ${NVIDIA_API_KEY}
      timeout: 7200

  # Mistral models for research and analysis
  - model_name: mistral-reasoning
    litellm_params:
      model: mistral/devstral-latest
      api_base: https://api.mistral.ai/v1
      api_key: ${MISTRAL_API_KEY}
      timeout: 3600

  # Lightweight model for quick tasks
  - model_name: hermes-lean
    litellm_params:
      model: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning
      api_base: https://integrate.api.nvidia.com/v1
      api_key: ${NVIDIA_API_KEY}
      timeout: 7200

Kubernetes Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-gateway
  namespace: ia-services-payed
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
        version: main-stable
    spec:
      containers:
      - name: litellm
        image: ghcr.io/anthropics/litellm:main-stable
        ports:
        - containerPort: 4000
        env:
        - name: MODEL_LIST
          valueFrom:
            secretKeyRef:
              name: litellm-config
              key: model-list.yaml
        - name: PROXY_SERVER_KEY
          valueFrom:
            secretKeyRef:
              name: litellm-secrets
              key: api-key
        - name: HTTP_TIMEOUT
          value: "7200"
        - name: REQUEST_TIMEOUT
          value: "3600"
        - name: LOG_PII
          value: "false"
        resources:
          requests:
            memory: "2Gi"
            cpu: "500m"
          limits:
            memory: "4Gi"
            cpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 4000
          initialDelaySeconds: 30
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health
            port: 4000
          initialDelaySeconds: 10
          periodSeconds: 10
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: litellm
  namespace: ia-services-payed
spec:
  selector:
    app: litellm
  ports:
  - port: 4000
    targetPort: 4000
  type: ClusterIP

MCP Integration

MCP Endpoint Configuration

LiteLLM exposes the MCP endpoint at http://litellm.ia-services-payed.svc.cluster.local:4000/mcp. Agents and cron jobs connect to this endpoint to access tools and models:

# Example MCP client connection
from mcp import ClientConnection

# Connect via the K3s service DNS
connection = await ClientConnection.connect(
    "litellm.ia-services-payed.svc.cluster.local",
    port=4000,
    path="/mcp",
)

# List available tools
tools = await connection.list_tools()
for tool in tools:
    print(f"{tool.name}: {tool.description[:80]}...")

MCP Server Registration

Each MCP server registers itself with the LiteLLM gateway:

# Register a new MCP server
curl -X POST http://litellm.ia-services-payed.svc.cluster.local:4000/mcp/register \
  -H "Authorization: Bearer ${LITE...KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "kubernetes-mcp",
    "url": "http://mcp-k8s.forgejo-runners.svc.cluster.local:8080/mcp",
    "timeout": 300
  }'

Multi-Model Routing Strategy

Routing Rules

We use a tiered routing approach:

Health-Based Routing

LiteLLM automatically routes to healthy models:

# If the primary model is unhealthy, LiteLLM falls back
# Check model health
import httpx

async def check_model_health(model_name: str) -> bool:
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            response = await client.post(
                f"{API_BASE}/v1/models",
                headers={"Authorization": f"Bearer {API_KEY}"},
            )
            return response.status_code == 200
    except httpx.TimeoutException:
        return False
    except httpx.RequestError:
        return False

Operational Lessons

Lesson 1: Timeout Configuration Matters

NVIDIA’s API can be slow for large models. We set:

HTTP_TIMEOUT=7200s  # 2 hours — for model loading
REQUEST_TIMEOUT=3600s  # 1 hour — for individual requests

Without these, requests timeout during model cold starts.

Lesson 2: Rate Limiting at the Proxy Layer

Configure LiteLLM’s rate limits to prevent any single consumer from overwhelming the backend:

model_aliases:
  "nvidia/nemotron-3-ultra-550b-a55b": "nemotron-35b"

litellm_settings:
  model_fallbacks:
    "nemotron-35b": ["nemotron-35b", "devstral-large"]
  num_retries: 3
  retry_after: 5

Lesson 3: Logging and Observability

Enable structured logging for debugging:

litellm_settings:
  success_callback: ["langsmith", "opentelemetry"]
  failure_callback: ["langsmith", "opentelemetry"]
  cache: true
  cache_params:
    type: "redis"
    host: "redis-monitoring.svc.cluster.local"
    port: 6379

Security Considerations

API Key Management

Never hardcode API keys. Use Kubernetes secrets:

# Create the secret from existing environment variables
kubectl create secret generic litellm-secrets \
  --from-literal=api-key="${PROXY_SERVER_KEY}" \
  --namespace=ia-services-payed

kubectl create secret generic litellm-config \
  --from-file=model-list.yaml \
  --namespace=ia-services-payed

Network Policies

Restrict who can access the LiteLLM gateway:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: litellm-ingress
  namespace: ia-services-payed
spec:
  podSelector:
    matchLabels:
      app: litellm
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: forgejo-runners
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: cicd
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: flux-system
    ports:
    - port: 4000
  policyTypes:
  - Ingress

Monitoring LiteLLM

Key metrics to track:

1. Request latency — P50, P95, P99 per model
2. Token usage — input/output per model and consumer
3. Error rate — by status code and error type
4. Model health — uptime and response time per backend
5. Cache hit rate — effectiveness of the response cache

Grafana dashboard queries (Prometheus):

# Request rate by model
rate(litellm_proxy_request_count[5m])

# Average latency by model
litellm_proxy_latency_seconds / 1000

# Error rate
sum(rate(litellm_proxy_request_error_count[5m])) / sum(rate(litellm_proxy_request_count[5m]))

Conclusion

LiteLLM + MCP provides a robust, scalable model gateway for K3s-based AI infrastructure. The key takeaways:

• Configure generous timeouts for model loading
• Use Kubernetes secrets for API key management
• Implement health-based model fallbacks
• Monitor token usage and latency per model
• Restrict access with NetworkPolicies

With these practices in place, your model gateway will handle the complexity of multi-model routing while your agents focus on their actual tasks.