One DGX Spark, One Model, Room to Train
Two models on one Spark was a mistake.
I ran it that way for a while — a fast workhorse for the common path, a heavier reasoner for the hard one — and spent most of my time refereeing two vLLM engines fighting over a single pool of memory. Neither could see the real free space. Both wanted hand-tuned fractions. And the heavy model was slow enough that the sync endpoints calling it kept tripping the gateway timeout.
So I dropped it. One model does inference now. Everything left over goes to training. The box got simpler and faster in the same move — and this is the setup, with the reasoning that got me here.
Disclosure: this guide was drafted with an AI assistant, then reviewed, corrected against the running box, and edited by hand. The commands below are the ones actually serving the model on my Spark — verify version-specific flags against your own build.
Why one model, not two
The DGX Spark is a desk-side machine built around NVIDIA's GB10 Grace Blackwell chip: 20 ARM cores, a Blackwell GPU, and the part that decides everything below — 128 GB of LPDDR5X shared between CPU and GPU at 273 GB/s. That bandwidth number, not the capacity, shapes every choice that follows.
The heavy model in the old setup was a 120B (Nemotron-3-Super). I abandoned it. My workload is mostly structured narration of pre-computed quant data, not deep reasoning, and the 120B was the wrong tier for it — async "Thoughts" took around two minutes, and the sync portfolio and stock-narrative endpoints kept hitting the ~100-second gateway timeout and returning 504s. No flag fixes that. It's the model class at 273 GB/s.
So the final design drops the second model entirely. One model serves; the rest of the pool trains.
The model that earns the slot
Qwen3.6-35B-A3B, served as NVFP4.
→ 35B total parameters, but only 3B active per token. As a Mixture-of-Experts model it stores a large network and reads only a sliver per token, so it decodes far faster than a dense heavy model on the same bandwidth.
→ ~23 GB of NVFP4 weights. Small enough to leave most of the pool free.
→ On this box it does the real work: financial-news interpretation, technical-analysis summaries, agentic tool calls, and coding — fast first tokens, clean tool-calling, reasoning when you ask for it per request.
I'm not going to quote a tokens-per-second number I haven't benchmarked on the Spark. Measure your own — there's a way to do it at the end.
The bigger models stay on an API
If you're eyeing a frontier open model like GLM-5.1 — not locally. At 754B total parameters it overflows 128 GB even at 4-bit. Keep that class behind an API and let the Spark own the high-frequency local work plus your training jobs.
The architecture
One model resident behind a vLLM endpoint, training sized to fit the headroom, both sharing one substrate. Inference takes a fixed slice; training takes the rest and yields under load.
┌──────────────────┐ ┌──────────────────┐
Clients ──▶ │ Apps & agents │ │ Frontier API │
│ news/TA/tools/code│ │ hard-reasoning │
└────────┬─────────┘ └──────────────────┘
▼
┌────────────────────────────────┐
│ vLLM OpenAI-compatible server │
│ localhost:8000/v1 · streaming │
└────────────────┬───────────────┘
▼
┌──────────────────────────────────────────────────────────┐
│ DGX Spark · GB10 · sm_121 · 273 GB/s · single node │
│ 128 GB unified memory pool │
│ ┌─────────────────────┐ budget ┌──────────────────┐ │
│ │ Qwen3.6-35B-A3B │ ─────────▶ │ Model training │ │
│ │ Solo inference │ │ leftover pool │ │
│ │ NVFP4 · ~23 GB │ │ LoRA · rerankers │ │
│ │ ~0.40 util resident │ │ embeddings │ │
│ └─────────────────────┘ └──────────────────┘ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Shared substrate: vLLM nightly · Docker · HF cache │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
The one gotcha that cost me a day
The NVFP4 checkpoint is MIXED_PRECISION, and that breaks the stable image.
vllm/vllm-openai:v0.20.0 misreads the checkpoint as plain FP8, decides the MoE experts are unquantized, and marlin refuses them:
moe_backend='marlin' is not supported for unquantized MoE
Both --quantization modelopt and --quantization fp4 fail the same way. The fix is the nightly image — it parses the mixed-precision MoE and loads it on marlin correctly. The catch: nightly is a rolling tag, so once a build works, pin its digest in your runbook instead of trusting the tag.
The sizing math
Account for every gigabyte. The unified pool is shared by the OS, the CUDA context, the resident model, and anything else on the box — there is no separate VRAM to hide in.
| Component | Footprint | Notes |
|---|---|---|
Inference reservation (--gpu-memory-utilization 0.40) | ~48 GB | Qwen3.6 weights ~23 GB + fp8 KV + activations |
| OS + ARM userspace + container runtime | ~10 GB | Fixed overhead |
| Leftover on a dedicated Spark | ~60–70 GB | Your training canvas |
That leftover is a real training canvas — LoRA/QLoRA fine-tunes, a bi-encoder plus cross-encoder reranker for a RAG pipeline, or embedding-model work, all while the inference endpoint stays live.
One honest caveat: that ~60–70 GB assumes a Spark doing nothing but serving this one model. Mine isn't. It also runs the embedding and reranker servers and a full data stack (Postgres, Redis, object store, a vector DB, the finvibe worker), so real training headroom here is closer to ~40–50 GB. Count what's already resident on your box before you size a job.
The real constraint is bandwidth, not capacity. Memory fits comfortably; a heavy training step and an inference request hitting the GPU in the same instant slow each other down at 273 GB/s. Treat training as the yielding workload, and pin the inference fraction low and explicit so training has a known ceiling to allocate against.
Step 1 — Prep the Spark
Confirm the GPU is visible and note the architecture. On the Spark this reports as Blackwell with compute capability 12.1 (sm_121).
# Confirm the GPU and driver
nvidia-smi
# Confirm Docker can see the GPU
docker run --rm --gpus all nvidia/cuda:13.0-base-ubuntu24.04 nvidia-smi
Create a host-side Hugging Face cache so weights survive container restarts and download only once:
mkdir -p ~/.cache/huggingface
export HF_TOKEN="hf_your_token_here"
The principle on Spark is download once, mount everywhere. Never let your first vllm serve also perform a multi-hundred-gigabyte download — stage weights first, then mount the same cache into every container.
Step 2 — Pull the vLLM image that can serve it
Use the nightly image, for the reason above — the stable v0.20.0 cannot load this checkpoint.
# Rolling tag — pin the digest once a build works for you
docker pull vllm/vllm-openai:nightly
A few environment flags select the right backends for NVFP4 on sm_121:
export HF_HUB_OFFLINE=1
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_FP8_MOE_BACKEND=flashinfer_cutlass
export FLASHINFER_DISABLE_VERSION_CHECK=1
export CUTE_DSL_ARCH=sm_121a
Step 3 — Pre-stage the weights
Download the checkpoint into the host cache before serving. Slow once, instant forever after.
# ~23 GB of NVFP4 weights
huggingface-cli download nvidia/Qwen3.6-35B-A3B-NVFP4 \
--local-dir-use-symlinks False
First load from a cold cache, plus the FlashInfer autotune on first boot, runs about 5–6 minutes.
Step 4 — Serve Qwen3.6
One container, one model, the flags that actually load the mixed-precision MoE. This is the command serving my box, lightly trimmed:
docker run -d --name qwen --ipc=host --restart unless-stopped \
--gpus all -p 8000:8000 \
-e HF_HUB_OFFLINE=1 \
-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
-e VLLM_FP8_MOE_BACKEND=flashinfer_cutlass \
-e FLASHINFER_DISABLE_VERSION_CHECK=1 \
-e CUTE_DSL_ARCH=sm_121a \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:nightly \
vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
--served-model-name qwen-fast \
--trust-remote-code \
--dtype auto \
--quantization modelopt \
--kv-cache-dtype fp8 \
--attention-backend flashinfer \
--moe-backend marlin \
--gpu-memory-utilization 0.40 \
--max-model-len 65536 \
--max-num-seqs 8 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--async-scheduling \
--enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
A note on the fraction: --gpu-memory-utilization 0.40 reserves roughly 48 GB (NVIDIA's own recipe used 0.85 on a dedicated box). It's a lever — push it down toward 0.30 to hand more of the pool to training, at the cost of inference concurrency and KV headroom. Pin it explicitly either way, because if vLLM grabs memory dynamically your training process OOMs unpredictably; a fixed fraction gives the training job a clean, known boundary.
Verify it is up:
curl -sS http://localhost:8000/v1/models | jq -r '.data[0].id'
# expected: qwen-fast
Step 5 — Pre-warm the JIT & verify
The first request after boot triggers Inductor and FlashInfer code generation and can take ~25 seconds. Never send that to a real user — fire a throwaway warm-up with a tiny token budget.
curl -sS http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen-fast","messages":[{"role":"user","content":"ping"}],"max_tokens":3}' > /dev/null
echo "warmed"
Watch the shared pool while inference is live:
watch -n 2 nvidia-smi
Step 6 — Train in the leftover
With inference holding its fixed slice, the rest of the pool is yours. Cap the training process so it never reaches for the slice vLLM owns — a hard memory fraction on the training side too.
# LoRA fine-tune in a separate container, capped so it
# cannot touch the inference slice.
docker run --rm --gpus all --ipc=host \
-e PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v $PWD/train:/work -w /work \
nvcr.io/nvidia/pytorch:25.05-py3 \
python finetune_lora.py \
--gpu-mem-fraction 0.45 \
--gradient-checkpointing \
--batch-size 4
Size the training fraction against what's actually free on your box, not the theoretical max — 0.40 inference plus 0.45 training leaves headroom for the OS and CUDA contexts only if nothing else is resident. On a shared box, subtract the other services first.
What fits the leftover well:
→ LoRA / QLoRA fine-tunes of small-to-mid models — adapter training is memory-light and the natural fit for a shared box.
→ Retrieval models for RAG — train or fine-tune a bi-encoder, then a cross-encoder reranker, on your own evidence corpus.
→ Embedding models — domain-adapt an embedding model on your data so retrieval quality climbs.
Keep full from-scratch pre-training off this box. The Spark is sized for fine-tuning and adapter training alongside inference, not pre-training large models from zero. If a job needs the whole pool, run it when inference is idle, or rent cloud GPUs for it.
Step 7 — Manage contention
Two workloads on one GB10 compete for the 273 GB/s bus, not just for capacity. Three habits keep it smooth:
→ Make training the yielding workload. Lower its process priority and keep batch sizes modest so a training step never monopolises the bus while a user request is mid-decode.
→ Schedule heavy runs in low-inference windows. Overnight, or whenever your agents are quiet. A cron-driven training queue pairs naturally with a daily brief pipeline.
→ Hold the memory line. Inference fraction fixed; training capped so the two plus everything else resident sum under 1.0. Treat any OOM in docker logs as your signal to back the training fraction off a notch.
Watch both at once — inference latency in vLLM's metrics, GPU and memory in nvidia-smi. If time-to-first-token spikes whenever a training step lands, that's bandwidth contention. Throttle training; don't go looking for more memory.
Takeaways
→ One model beats two on this box. Solo inference removes the dual-engine memory juggling and the slow heavy-model decode.
→ Qwen3.6 at 3B active is the right driver — fast decode, long context, clean tool-calling and reasoning parsers.
→ Use the nightly image. The mixed-precision NVFP4 checkpoint will not load on v0.20.0; that one fact will save you a day.
→ The leftover pool is a real asset — fine-tune, train retrieval and reranker models, or adapt embeddings, without taking inference down. Size it against what's actually free, not the theoretical max.
→ Bandwidth, not capacity, is the constraint. Treat training as the yielding workload and pin memory fractions explicitly.
→ Keep frontier-scale models on an API. 754B parameters never fit in 128 GB.
If you run this stack, the most useful thing you can do next is measure two numbers under a real workload: your inference decode rate with training idle, and again with a training step running. The gap between them is your true contention cost. What did your Spark actually clock?
Enjoyed this post? Subscribe for more.