Qwen3.8-27B on the DGX Spark: from 9 to more than double the speed with vLLM

The model dropped on a Friday and we couldn’t wait

The news of Qwen3.8-27B came out on a Friday and, as happens to us almost every time, we couldn’t wait until Monday to get it running. This post is the first chronicle of that deployment: we did it on top, by hand, on the infra we already had, because we haven’t documented how we govern everything with Flux CD or how we set up Forgejo yet. We’ll see that in later posts. Here, for now, what interests us most is to tell the part that surprised us the most: how much it costs —or stops costing— to run a 27B model at home, and which lessons from the official recipes took us from 9 to more than double the tokens per second.

It’s an initial post, on purpose. It won’t pretend to be the definitive deployment manual: it’s the picture of what happened, with the numbers we pulled from the logs, and with the feelings we took away from the journey.

The scenario: a DGX Spark inside the cluster

Our K3s cluster has two nodes. The first, gamorcloud01, is the control plane and CPU worker. The second, gx10-db4d, is the GPU node where inference workloads have been living for years. That’s where Qwen3.8-27B landed, in the dgx-ia namespace, as a Kubernetes Deployment running the vllm/vllm-openai:v0.27.1 container.

The model we uploaded is the NVFP4 quantized version — the recipe they published — downloaded from Hugging Face with its own initContainer, and mounted on a PVC so it doesn’t re-download 40 GB every time it restarts:

containers:
  - name: vllm-nvfp4
    image: vllm/vllm-openai:v0.27.1
    command: [vllm, serve, /models/qwen3.8-27b-inferact-nvfp4]
    resources:
      limits:
        nvidia.com/gpu: "1"
        memory: "120Gi"
      requests:
        nvidia.com/gpu: "1"
        memory: "64Gi"

Two things that matter and that usually hurt in this kind of home deployment:

• The startupProbe with failureThreshold: 180. A 27B model takes a while to start up. If the probe is impatient, the cluster kills the pod before it has learned to breathe. We gave it almost an hour of startup margin and then left a normal readinessProbe.
• The initContainer with a .download-complete. The first time it downloads the model and touches a marker; the next times it sees the marker and skips straight to the service. Without this, every restart was a new wait.

All of this without Flux CD yet: we applied it by hand, to be able to see the difference between config and config without GitOps masking it from us. That, for now, is a debt we’ll settle in a post dedicated to how we govern it via GitOps.

The base recipe: and why it gave 9 tokens/s

We started with a conservative configuration, a “first time I try it and I don’t want it to fall over”. It made sense: --gpu-memory-utilization 0.45, without explicit prefix caching, without MTP. It started. It responded. But slowly.

The base figure that got engraved in us was 9 tokens/second. Little, but stable. And that’s where the curiosity began, because the official Qwen3.8 recipes with vLLM spoke of a much more agile scenario than the one we were getting.

The new recipe: three levers, one change

What we did was apply the recommended recipe for Qwen3.8 in vLLM, lever by lever. We were very careful not to touch what was already working well: --kv-cache-dtype fp8, --max-model-len 262144, --enable-chunked-prefill, and the reasoning and tool-calling parsers (--reasoning-parser qwen3, --tool-call-parser qwen3_coder, --enable-auto-tool-choice). We changed only three things:

# 1) MTP (Multi-Token Prediction): Qwen3.8 trae MTP; vLLM 0.27.1 lo soporta.
- --speculative-config
- '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

# 2) Memoria: 0.45 -> 0.90. El KV cache solo usaba 9-9,8% de la GPU;
#    ~55% quedaba sin asignar. (Riesgo: si hay OOM, bajar a 0.85.)
- --gpu-memory-utilization
- "0.90"

# 3) Prefix caching explícito: por defecto va ON en el V1 engine, pero
#    queríamos asegurarlo. Antes teníamos hit rate 0%.
- --enable-prefix-caching

Each of the three solved a different problem, and that’s why I separate them:

1. MTP (Multi-Token Prediction)

Qwen3.8 comes, out of the box, with a layer of predicting several tokens at once (MTP). vLLM 0.27.1 knows how to take advantage of it with --speculative-config and num_speculative_tokens 2. The idea: the model speculates a couple of tokens at once, and the big model confirms or rejects. If we get it right, we win; if it’s wrong, we don’t lose (it’s discarded and it starts over). In the logs we saw a draft acceptance rate between 60% and 82%, and an average acceptance length of 2.3–2.6 tokens. It’s not magic, but it’s an elegant way of advancing.

2. GPU memory: 0.45 → 0.90

This was the one that surprised us most in the diagnosis. With 0.45, the KV cache only used 9-9.8% of the GPU. That is, we had more than half of the card’s memory unassigned to what needed it most: the context cache. Raising to 0.90 doesn’t by itself speed up the token, but it frees the cache to be able to serve more requests and more context, and paves the way for point 3. The only risk, and we wrote it in the manifest itself, is an OOM: if it happened, it drops to 0.85.

3. Explicit prefix caching

This is the one that changes the day. The vLLM V1 engine has prefix caching ON by default, but we had a hit rate of 0% — the cache was activating, but it served no purpose. By adding it explicitly, the hit rate rose to the 84-86% we saw in the logs. Why does it matter so much? Because when two requests share context (the same system prompt, the same conversation, the same tool prefix), instead of recalculating it from scratch, the engine reuses what it had already calculated. In an infra with many chats that share templates, that’s 86% of the work saved.

The numbers: from 9 to more than double, and sometimes 4 to 6 times

Here’s where it’s worth being honest, because “fast” can mean different things, and in the logs you see two speeds that aren’t the same.

In raw throughput (what the recipes measure): the figures come out more than double. The base was 9 tokens/s. With the new recipe, in individual requests (a single one in flight) the logs show ~22–31 tokens/s of generation; and when the engine is at the 4 concurrent requests enabled by --max-num-seqs 4, the peaks rise to 40–56 tokens/s. That is: more than double than the base recipe, just as the recipes said.

In response time (what we perceived in real use): the jump amplifies. The feeling was of a x4 or x6. And it makes sense, because response time doesn’t depend only on raw throughput: the prefix caching at 86% cuts the time to the first token in requests that share context, and MTP speeds up generation. What “looks like more than double” in the statistics translates, in the experience, into a response that arrives 4 to 6 times faster.

We don’t confuse “more than double” with “4–6 times”: it’s the same configuration, measured two ways. Throughput is seen in the logs; the 4–6× feeling is what you notice when using it, and it comes from the cache and speculation reducing the time to the response, not from each token being 6× faster.

The real feeling: against the Qwen3.6 35B

This is where things get interesting, and where it’s worth being very honest about the kind of comparison — because it’s not “model A against model B at the same speed”. They are two different things, and it’s worth separating them.

What we had before — Qwen3.6 35B, a MoE A3B. It’s a mixture of experts model that only activates 3 billion parameters per token. That made it very fast: the cost per token is low because for each token only a few billion parameters work. But at a price: an accuracy that disappointed us. Fast, yes; but in the fine tasks you noticed it lost quality, that it didn’t get it right as well when detail mattered.

The new one — Qwen3.8-27B, unoptimized. Here came the disappointment. Without the recipe, this model was very slow: 9 tokens/s. Slow to the point that, against the 35B MoE, we seemed to have regressed. And yet it had what the 35B didn’t have: much higher accuracy.

And then, with the three optimizations. What really surprised us. Not only did we recover the speed —the new recipe takes us to a speed similar to the 35B MoE (and, in perceived response time against the unoptimized 27B, a 4-6× better)—, but it does it preserving that accuracy. That is: we got the speed of the 35B and the quality of the 27B. The best of both, in the same deployment.

That’s the feeling: it’s not that “a small model crushes a big one”. It’s that the 27B, badly set up, looked like a mess; well set up, it is at once the fastest and the most accurate. The recipe didn’t only speed it up: it gave us back a model that also behaves better. In a home infra, where every second of GPU counts, having speed and accuracy and efficiency in the same deployment is what really changes the experience.

What we took away (and what’s coming)

Three lessons from the journey:

• Don’t trust defaults without measuring them. Prefix caching was “ON” by default and we had a hit rate of 0%. Sometimes the default is de facto disabled. Measure it, don’t assume.
• Unassigned memory is performance memory. The 0.45 of gpu-memory-utilization wasn’t an error, it was an opportunity: half the GPU was asleep.
• Optimizing can do more than changing the model. The 27B without the recipe was slower than the 35B MoE; with the three optimizations it reached a speed similar to the 35B and with much higher accuracy. You don’t always have to change the model: sometimes you have to set it up well.

And what comes —because this post is on purpose the first part—: how we govern these deployments with Flux CD (so next time we don’t do it by hand), and how we set up Forgejo so all this is a PR with human review instead of a kubectl apply in the middle of the night.

In the meantime, the model keeps running on gx10-db4d, in dgx-ia, without falling over since we uploaded it. And, if you ask me, the answer is: yes, you can tell.