cloudburn.dev

DGX Spark: vLLM vs Atlas for Local Inference

Two inference runtimes on the same model: vLLM and Atlas via sparkrun. Memory models, real throughput numbers, and which one makes sense for a personal agent.

Previous in this series: Update: Hermes on DGX Spark - Qwen3.6-35B-A3B-NVFP4 via vLLM
That post covers the three failure modes getting vLLM working on GB10 and what finally fixed them. Start there if you haven’t.

Once the image problem was solved, I had a working inference server. Then I found out there were two ways to get there, and they tell a very different story.

Option A: vLLM

The default choice. Mature, NVIDIA-backed, battle-tested.

docker run -d --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --name vllm_qwen36 \
  -e HF_TOKEN=$HF_TOKEN \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  vllm-node-tf5:latest \
  vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 1 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --max-model-len 131072 \
    --kv-cache-dtype fp8 \
    --attention-backend FLASHINFER \
    --enable-chunked-prefill \
    --max-num-seqs 8 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

Option B: Atlas via sparkrun

Atlas is a pure-Rust inference server with native GB10 support, managed by sparkrun. One command:

uvx sparkrun setup
sparkrun run @atlas/qwen3.6-35b-a3b-nvfp4

No config file. No Docker flags. No image to build. Serves on port 8888.

That contrast is worth sitting with for a moment.

The Memory Model Difference

This matters more than the tok/s numbers alone.

vLLM pre-allocates the entire KV cache pool at startup:

KV memory = max_num_seqs x max_model_len x layers x kv_heads x head_dim x dtype_bytes

At max_model_len: 131072 with gpu_memory_utilization: 0.90, vLLM needs roughly 90GB just for KV cache to support 8 concurrent sequences. At startup, before a single request arrives, about 106GB of your 128GB pool is already spoken for. That’s why max_num_seqs had to drop from 32 to 8 when I moved from 65K to 131K context.

Atlas allocates lazily. KV memory is claimed per-request as tokens are generated and released when the request completes. Startup footprint:

Model weights:  16.3 GB
KV cache:        5.0 GB  (current active requests)
Total:          ~21 GB

Same model. Same 131K context. Five times less memory at rest.

What that means in practice:

  • Atlas can serve 131K context and high concurrency without choosing between them
  • Other processes (OS, Hermes agent process, monitoring) have actual headroom
  • Under light load, the GPU is genuinely available for other tasks

The tradeoff is predictability. vLLM’s pre-allocation means the 9th request when you’ve configured 8 seqs gets queued cleanly, with no surprises. Atlas’s lazy model means if 20 agents simultaneously hit 100K context, you can OOM at runtime with no warning. For personal use with 1-3 concurrent agents, that risk is theoretical. For a production API under real load, vLLM’s behavior under pressure is worth more than the memory savings.

Benchmark Results

Methodology: shell timer around blocking /v1/chat/completions calls, completion_tokens / elapsed_seconds. Thinking mode disabled via chat_template_kwargs: {"enable_thinking": false} on every request. Without that flag, Qwen3 outputs a <think> block before answering and the timing numbers become meaningless.

Throughput

RuntimeContextMax seqstok/s (warm)TTFT
vLLM (enforce_eager)3276832~27407ms
vLLM (CUDA graphs)6553632~7298ms
vLLM (CUDA graphs)1310728~69~100ms
Atlas13107232+~112122ms

Atlas runs about 1.6x faster on decode throughput at equal context. TTFT is slightly higher on Atlas (122ms vs 98ms). That’s expected with MTP speculative decoding overhead on the first token, not a flaw.

enforce_eager disables CUDA graphs and costs roughly 3x throughput. If you hit instability on first run, add it back and accept the penalty. Get stable first, then optimize.

Quality Test

Same prompts, same model, both runtimes:

TestAtlasvLLMResult
Reasoning (bat+ball algebra)passpassIdentical
Code (Sieve of Eratosthenes)passpassIdentical
Tool selection (3 tools, pick correct)passpassIdentical
Instruction following (3 bullets, max 10 words)passpassIdentical

Same model, same quality. The runtime is just the delivery mechanism.

Concurrent Load (3 simultaneous requests)

RuntimePer-req tok/sAggregate tok/sWall time
Atlas single~112n/a4.5s
Atlas 3x concurrent~206112.6s
vLLM 3x concurrent~236911.1s

Under concurrent load, Atlas’s speculative decoding advantage collapses. Draft verification gets contended across requests. vLLM’s pre-allocated batch scheduler handles this better. For Hermes sub-agent fan-out with 3 concurrent children, both runtimes work. vLLM edges ahead under batch load.

When to Use Which

Use Atlas when:

  • Personal agent or home lab (1-5 concurrent users)
  • You want 131K context without giving up concurrency
  • You need other processes to share GPU memory alongside inference
  • You want one-command deploy, zero config
  • You care about non-root operation (Atlas runs as your uid by default; the vLLM eugr image runs as root)

Use vLLM when:

  • Serving a team or production API with real concurrent load
  • You need predictable behavior under pressure, no surprise OOMs
  • You need the broader ecosystem: LoRA, multi-LoRA, more quantization formats, more tool call parsers
  • Enterprise or compliance context where vLLM’s NVIDIA backing and deployment history matters

For my setup (Hermes personal agent, 1-3 concurrent sub-agents, home network), Atlas is the right call. Memory efficiency, non-root operation, and the throughput advantage all point the same direction.

Why Model Size Matters for an Agent

Think of it like hiring:

  • 7B model (qwen2.5:7b on iGPU): handles simple instructions, struggles with multi-step plans, makes tool call mistakes. You spend more time correcting it than it saves you. Roughly Haiku quality.

  • 14B model (qwen2.5:14b on a 4080 Super): solid for most tasks, but complex reasoning and long agent loops expose the ceiling. Between Haiku and Sonnet, closer to Haiku.

  • 35B MoE model (Qwen3.6-35B-A3B on DGX Spark): the jump feels qualitative, not just quantitative. Multi-step plans hold together. Tool calls land right the first time. It pushes back when instructions are ambiguous instead of guessing wrong. Long agent loops don’t collapse. Sonnet-range. This is the threshold where a local agent starts being genuinely useful.

The “35B” label is slightly misleading. It’s a Mixture of Experts model, meaning only 3B parameters activate per token. MoE routing lets it specialize: different experts fire for code vs reasoning vs tool use. That’s why quality is disproportionate to the active compute budget.

The hard constraint is memory. Qwen3.6-35B in NVFP4 takes about 22GB for weights alone. vLLM’s KV cache allocation brings total runtime to roughly 106GB. No consumer GPU reaches that. The best gaming cards top out at 24GB. Atlas’s lazy allocation uses far less at rest, but the weights alone still need the DGX Spark’s unified pool.

Hardware Comparison

Original (iGPU)RTX 4080 SuperDGX Spark
HardwareRadeon 680M16GB VRAMGB10, 128GB unified
Best model that fitsqwen2.5:7b Q4qwen2.5:14b Q4Qwen3.6-35B NVFP4
Quality tierHaikuHaiku+Sonnet-range
InferenceOllamaOllamavLLM / Atlas
tok/sunusable under load~45-55~69-112
TTFTvery highn/a0.10-0.12s
Local / privateyesyesyes

The 4080 Super and 3090 can’t run the 35B model. 106GB runtime won’t fit in 16-24GB VRAM. Capped at roughly 13B Q4 models, which puts them in Haiku+ territory at best.

Groq (cloud, llama-3.3-70b) hits around 300 tok/s. Faster, but remote, rate-limited on the free tier, and your data leaves your network. DGX Spark is the only local option that reaches Sonnet-class quality right now.

Connecting Hermes

Both runtimes expose an OpenAI-compatible API. The config change is minimal:

# hermes config.yaml

# For Atlas (port 8888)
custom_providers:
- name: DGX Spark
  base_url: http://192.168.4.57:8888/v1
  model: nvidia/Qwen3.6-35B-A3B-NVFP4
  api_mode: chat_completions

model:
  default: nvidia/Qwen3.6-35B-A3B-NVFP4
  provider: DGX Spark
  context_length: 131072

delegation:
  model: nvidia/Qwen3.6-35B-A3B-NVFP4
  provider: DGX Spark
  max_concurrent_children: 3
  orchestrator_enabled: true

One thing that bit me: Hermes auto-detects context length from the API. Atlas reports a low number at startup (12288). Override it explicitly with context_length: 131072 or Hermes refuses to start with a “below minimum 64K” error.

Caveats

  • GB10 has no native Marlin FP4 kernel support. Warnings in logs, not errors. Falls back to emulation.
  • 128GB unified memory sounds like a lot until OS, Hermes agent process, and inference KV cache all compete for the same pool. vLLM is aggressive about claiming it. Atlas is polite.
  • nvidia/* models on HuggingFace are gated. You need a valid HF_TOKEN with access approved before downloading.
  • Atlas runs as your user (uid=1000) by default. The vLLM eugr image runs as root. Worth knowing if you care about that surface.
  • Qwen3 defaults to thinking mode. Disable per-request with chat_template_kwargs: {"enable_thinking": false} or the model will spend most of its token budget in <think> blocks before answering.

Debug log: ~/vllm-debug-log.md
Forum thread: NVIDIA Developer Forums