cloudburn.dev

Update: Hermes on DGX Spark - Qwen3.6-35B-A3B-NVFP4 via vLLM

Follow-up to the Proxmox/ROCm iGPU build. Hardware upgrade to DGX Spark (GB10), three distinct failure modes getting vLLM serving an NVFP4 MoE model, and what finally fixed it.

Original post: Running a Local AI Agent on Proxmox with AMD ROCm GPU Passthrough
This is an update. Go read that first if you want the Hermes setup walkthrough.

The Hardware Jump

The original post ran Hermes with qwen2.5:7b on a Radeon 680M iGPU through ROCm in a Proxmox LXC. It worked. It was also painfully slow. Roughly 2-minute response times under agent load, because the iGPU’s memory bandwidth saturates hard when you’re pushing 131K context through a 7B model repeatedly.

I now have access to a DGX Spark: NVIDIA’s GB10 SoC, compute capability SM 12.1, 128GB unified CPU/GPU memory pool. The pitch is to run a real MoE model locally and actually get usable latency from Hermes.

Target model: nvidia/Qwen3.6-35B-A3B-NVFP4, the same Qwen3.6 35B architecture NVIDIA calls out in their Hermes agent blog, quantized to NVFP4 format using ModelOpt. Five times the parameters of the iGPU build, on hardware that can actually feed them.

Getting it serving was not straightforward. Here’s what happened.

What Broke and Why

Problem 1: Silent hang after attention backend selection

Three independent builds, including NVIDIA’s own recipe, hung silently right after this log line:

[EngineCore] INFO Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].

No error. No timeout. GPU allocates memory, spawns 40–160 sleeping threads (wchan=futex_do_wait), utilization drops to 0%, process never progresses.

Root cause: FlashInfer has no pre-compiled CUDA kernels for SM 12.1 (VLLM_HAS_FLASHINFER_CUBIN = False inside those images). When FlashInfer is selected, it falls back to JIT kernel compilation. JIT compilation deadlocks silently on GB10.

Why FlashInfer always won: --kv-cache-dtype fp8 (required for memory budget) disqualifies every other backend except FlashInfer and Triton. vLLM picks FlashInfer first.

Quick check if your image has this problem:

docker run --rm <your-image> python3 -c "import vllm.envs as e; print(e.VLLM_HAS_FLASHINFER_CUBIN)"
# False = you will hang

Workaround (for images without pre-compiled cubins):

--attention-config '{"backend": "TRITON_ATTN"}'
-e VLLM_USE_FLASHINFER_SAMPLER=0

Note: VLLM_ATTENTION_BACKEND env var does not exist in older vLLM builds. It’s silently ignored. Must use the CLI arg.

Problem 2: KeyError in the NVFP4 weight loader

Once past the FlashInfer hang, the model load itself crashed:

KeyError: 'layers.0.mlp.experts.w2_input_scale'

This is in the vLLM weight loader for Qwen3.5 MoE (qwen3_5.py:393). It expected fused scale keys in a specific format. The NVFP4 checkpoint uses a different structure: per-expert, per-projection scales with a VLM wrapper prefix:

model.language_model.layers.X.mlp.experts.Y.down_proj.input_scale
model.language_model.layers.X.mlp.experts.Y.gate_proj.weight_scale
# ...

The model’s declared architecture is Qwen3_5MoeForConditionalGeneration, a vision-language model wrapper, not a plain causal LM. The image’s vLLM didn’t know how to unpack it.

Problem 3: Corrupt model cache (bonus)

All my HuggingFace model caches had .incomplete suffix blobs, partial downloads from before I had a valid token for gated nvidia/* models. The FlashInfer hang was masking this entirely since it died before weight loading was ever attempted. Clean your cache and re-download with a token:

# nvidia/* models are gated -- need HF_TOKEN
find ~/.cache/huggingface/hub -name "*.incomplete" -delete

What Fixed It

eugr_nv (NVIDIA moderator on the developer forums) pushed a new build on June 14, 2026: eugr/spark-vllm-docker, prebuilt-vllm-current release.

Two things in this build solved both problems:

  • vLLM 0.22.1rc1.dev511: updated weight loader handles the Qwen3_5MoeForConditionalGeneration NVFP4 checkpoint format correctly
  • FlashInfer 0.6.13 with pre-compiled SM 12.1 cubins: no more JIT hang, FlashInfer works natively on GB10

Build takes about 4 minutes (downloads prebuilt wheels, no compilation):

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./build-and-copy.sh --tf5

--tf5 is required. The model’s config declares transformers_version: 5.7.0.dev0, and without it vLLM pins an older transformers that can’t load the checkpoint.

Serving Config

# vllm_qwen36_config.yaml
model: nvidia/Qwen3.6-35B-A3B-NVFP4
host: 0.0.0.0
port: 8000
tensor_parallel_size: 1
trust_remote_code: true
gpu_memory_utilization: 0.85
max_model_len: 65536
kv_cache_dtype: fp8
attention_backend: FLASHINFER
docker run -d --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --name vllm_qwen36 \
  -e HF_TOKEN=$HF_TOKEN \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -e VLLM_LOGGING_LEVEL=INFO \
  vllm-node-tf5:latest \
  vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --config /config.yaml

Verified working:

$ curl http://localhost:8000/v1/models
{"data":[{"id":"nvidia/Qwen3.6-35B-A3B-NVFP4","max_model_len":65536,...}]}

$ curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4",
       "messages":[{"role":"user","content":"Say hello in one word"}],
       "max_tokens":10}'
# → real tokens, system_fingerprint: vllm-0.22.1rc1.dev511+gc621af169.d20260614

Connecting Hermes

vLLM exposes an OpenAI-compatible API on port 8000, same interface as Ollama’s /v1 endpoint from the original post. Hermes config change is minimal:

# hermes config -- swap endpoint, bump model
llm:
  provider: openai
  base_url: http://localhost:8000/v1
  model: nvidia/Qwen3.6-35B-A3B-NVFP4
  api_key: none

Context window: max_model_len: 65536 in the serve config above. The original post expanded qwen2.5:7b to 131K context via Ollama Modelfile. Qwen3.6’s native context is 262K; we’re serving 64K here with kv_cache_dtype: fp8 and CUDA graphs active. The KV cache at 64K context is the dominant memory consumer (roughly 86GB at 90% utilisation). The model weights themselves are only ~22GB in NVFP4.

Performance Comparison

Hardware + Model

Original (iGPU)This update (DGX Spark)
HardwareRadeon 680M, shared RAMGB10, 128GB unified pool
Modelqwen2.5:7b (Ollama)Qwen3.6-35B-A3B-NVFP4 (vLLM)
Parameters7B dense35B MoE (3B active/token)
QuantizationQ4_K_M (Ollama default)NVFP4 (ModelOpt)
Context served131K64K
Inference serverOllamavLLM 0.22.1rc1
APIOpenAI-compatible /v1OpenAI-compatible /v1
Setup difficultyMediumHard (3 distinct failure modes)

Why Model Size Matters for an Agent

Think of it like hiring someone for a job:

  • 7B model (qwen2.5:7b on iGPU): smart kid who reads fast. Can follow simple instructions, answer basic questions. Struggles with multi-step plans, forgets context, makes tool call mistakes. You spend more time correcting it than it saves you. Roughly Claude Haiku quality.

  • 14B model (qwen2.5:14b on 4080 Super): solid junior. Handles most tasks, but complex reasoning, ambiguous instructions, and long agent loops expose the ceiling. Gets the job done often enough to be useful, but you notice the gaps. Between Haiku and Sonnet, closer to Haiku.

  • 35B MoE model (Qwen3.6-35B-A3B on DGX Spark): the jump feels qualitative, not just quantitative. Multi-step plans hold together. Tool calls are accurate first time. It pushes back when instructions are unclear instead of guessing wrong. Long agent loops don’t collapse. Sonnet-range. The threshold where local AI becomes genuinely useful as an autonomous agent.

The “35B” is slightly misleading. It’s a Mixture of Experts model, meaning only 3B parameters are active per token. MoE routing lets the model specialize: different experts fire for code vs reasoning vs tool use. That’s why quality is disproportionate to the active compute.

The hard constraint is memory. A 35B MoE model in NVFP4 takes ~106GB at runtime (weights + KV cache). That’s the entire 128GB unified pool of the DGX Spark nearly full. No consumer GPU gets there. The best gaming cards top out at 24GB.

Throughput

These are not apples-to-apples. Each setup runs the best model its hardware can fit:

SetupInferenceModelQuality tiertok/sTTFT
Radeon 680M iGPUOllamaqwen2.5:7b Q4Haikuunusable under agent loadvery high
RTX 4080 Super (16GB)Ollamaqwen2.5:14b Q4Haiku+~45–55n/a
RTX 3090 (24GB)Ollamaup to ~13B Q4Haiku+~40–50n/a
Groq APIcloudllama-3.3-70bSonnet-range~300low
DGX Spark GB10 (128GB)vLLMQwen3.6-35B NVFP4Sonnet-range~720.10s

Groq is faster, but it’s remote, rate-limited on the free tier, and your data leaves your network. The DGX Spark is the only local option that reaches Sonnet-class quality. 0.10s TTFT, no rate limits, no API costs, data stays home.

Caveats

  • enforce_eager is removed in the final config. CUDA graphs are active, which is where the 72 tok/s comes from. If you hit OOM or instability on first run, add it back and expect ~25 tok/s.
  • GB10 has no native Marlin FP4 kernel support (warnings in logs, not errors, falls back to emulation).
  • 128GB unified memory sounds like a lot until you realize OS, Hermes agent process, and vLLM KV cache all share the same pool. Budget accordingly.
  • nvidia/* models on HuggingFace are gated. You need a valid HF_TOKEN with access approved.

Debug log with full attempt history: ~/vllm-debug-log.md
Forum thread documenting the FlashInfer SM 12.1 issue: NVIDIA Developer Forums