GPT-OSS-120B: Topology Benchmark on DGX Spark

The Model

openai/gpt-oss-120b — 117B total parameters, 5.1B active per token. Sparse Mixture-of-Experts with native MXFP4 quantization, designed for Blackwell’s MX-format tensor cores. At 4-bit precision, the full model fits in a single DGX Spark’s 128 GB unified memory with room for KV cache.

This is the model the DGX Spark community rallied around first. Every topology comparison, every routing strategy, every vLLM optimization — GPT-OSS-120B was the proving ground.

Test Setup

• Hardware: 2x NVIDIA DGX Spark (Grace Blackwell GB10, 128 GB unified each)
• Interconnect: ConnectX-7 dual 200GbE QSFP stacking link
• Image: vllm-node-mxfp4 (0.1.dev12777)
• Recipe: openai-gpt-oss-120b.yaml (gpu_memory_utilization=0.70, max_model_len=4096)
• Flags: --enforce-eager --mxfp4-layers moe,qkv,o,lm_head --kv-cache-dtype fp8
• Benchmark: llama-benchy 0.3.5, pp1024 tg128, 50 runs per concurrency level
• Date: 2026-03-22

Topologies Tested

Decode Throughput

Generation speed (tg128) — higher is better.

Legend: ━━ Solo · ━━ Cluster TP=2 · ━━ Shuffle · ━━ Least-busy

Cluster TP=2 dominates c1 through c32 — both GPUs working on each request via NCCL all-reduce. At c64, the synchronization overhead catches up and simple-shuffle overtakes by 20%.

Note that Solo plateaus at c32 (315 t/s) while the multi-node topologies keep scaling. Least-busy also flatlines — backend starvation kicks in.

Prefill Throughput

Prompt processing speed (pp1024) — higher is better.

Legend: ━━ Solo · ━━ Cluster TP=2 · ━━ Shuffle · ━━ Least-busy

The crossover is dramatic. Cluster wins at low concurrency, but at c16+ shuffle pulls away because each node independently processes different prefills. At c64, Solo and Cluster both collapse (memory pressure) while shuffle keeps climbing — each node only sees ~32 concurrent requests.

Time to First Token (TTFT)

End-to-end latency to first generated token (ms) — lower is better.

Legend: ━━ Solo · ━━ Cluster TP=2 · ━━ Shuffle · ━━ Least-busy

TTFT crossover happens at c8. Below that, Cluster’s faster per-request prefill wins. Above c8, shuffle’s independent processing wins because each node starts its prefill without waiting for the other.

Routing Strategy: simple-shuffle vs least-busy

This benchmark included a controlled A/B test — same model, same day, only the LiteLLM routing strategy changed.

simple-shuffle wins across the board at c8+. At c64, the gap is enormous:

Why least-busy fails

least-busy tracks in-flight requests and routes to the least loaded backend. With identical backends and similar request durations, this creates oscillation — Backend A finishes a batch, drops to zero in-flight, ALL new requests rush to A, Backend B starves, cycle repeats.

Variance confirms it:

Never use least-busy for identical backends.

When to Use Which Topology

Technical Notes

• Always use IP addresses — mDNS hostname resolution causes hard crashes at c64
• LiteLLM Docker DNS: Remote backends need IPs (mDNS doesn’t resolve in containers)
• Thermal: GPUs reach 84-90C under sustained c64 load. T.Limit throttle at 53C is normal operation
• Solo recipe is safer for GF1: gpu_mem 0.70 with 4K context stays within thermal margin. The claude recipe (0.75, 131K) has crashed GF1 under load