Three Models, Two Sparks: Cross-Model Benchmark Comparison

The Lineup

Three models, all running on the same hardware: 2x NVIDIA DGX Spark (Grace Blackwell GB10, 128 GB unified each), connected via ConnectX-7 200GbE stacking link. Each benchmarked across Solo, Cluster TP=2, and 2x Solo + simple-shuffle topologies using llama-benchy (pp1024 tg128, 50 runs per concurrency level).

All three are Mixture-of-Experts variants — sparse MoE, standard MoE, and LatentMoE respectively. The active parameter count determines real-world speed more than total parameter count.

Why Total Parameters Don’t Predict Speed

The headline number on a model card — “120B parameters” — tells you almost nothing about inference speed on MoE hardware. What matters is:

1. Active parameters per token — how much compute per forward pass
2. Quantization precision — bits per active parameter
3. Effective bit-work — active params x bits = total work per token

GPT-OSS-120B is the fastest despite having 117B total params — MXFP4 at 4-bit gives the lowest effective compute per token. Nemotron has the most active params (12B) and is 3.5x slower at c1.

Legend: ━━ GPT-OSS-120B · ━━ Gemma4-26B · ━━ Nemotron-3-Super

GPT-OSS leads at low concurrency but Gemma4 catches up by c8 and overtakes at c32+. Nemotron scales the slowest — its 12B active params per token limit how many concurrent requests the hardware can process.

Cluster TP=2: Who Benefits Most?

Tensor parallelism splits each request across both GPUs. The benefit depends on how much NCCL synchronization traffic the model generates relative to the compute saved.

Legend: ━━ GPT-OSS-120B · ━━ Gemma4-26B · ━━ Nemotron-3-Super

Gemma4 benefits most from TP=2 at c1 (+50%), but GPT-OSS-120B maintains higher absolute throughput at every concurrency level until c64, where Gemma4 edges ahead (478 vs 472 t/s).

The Crossover: When Does Shuffle Beat Cluster?

Every model shows the same pattern: Cluster TP=2 wins decode at low concurrency, but 2x Solo + simple-shuffle catches up as concurrency rises. The crossover point varies dramatically by model:

Legend: ━━ GPT-OSS-120B · ━━ Gemma4-26B · ━━ Nemotron-3-Super

Nemotron is the only model where Cluster TP=2 never loses decode. Its 12B active params create enough per-request compute that tensor parallelism always outperforms independent routing, even at c64.

GPT-OSS shows the biggest shuffle advantage at c64 (+20% vs cluster) — its tiny 5.1B active set means per-request compute is cheap, so the overhead of NCCL synchronization becomes the bottleneck before the independent-routing overhead of shuffle does.

Prefill: Shuffle Always Wins at Scale

All three models show the same pattern for prefill throughput: Cluster wins at c1-c2, shuffle overtakes at c4, and the gap widens steadily.

At c64, GPT-OSS shuffle achieves +211% vs cluster prefill. The cluster’s prefill collapses under high concurrency because NCCL synchronization serializes what could be independent parallel work.

TTFT: The Latency Picture

Time to First Token matters most for interactive use. Lower is better.

Gemma4 has the best absolute TTFT: 181ms at c1 (cluster). Its small model memory (25.67 GiB) means less data to move before generating the first token. Nemotron’s 504ms reflects its 69.5 GiB footprint and 12B active compute per prefill token.

Routing Strategy: simple-shuffle Everywhere

Tested on GPT-OSS-120B and Nemotron-3-Super, the result is consistent:

simple-shuffle outperforms least-busy at c8+, across all models. Least-busy causes backend starvation: one node gets all traffic while the other idles. At c64, the gap is 57-68% in decode throughput.

This isn’t model-dependent — it’s a fundamental property of least-busy routing with identical backends. Always use simple-shuffle for 2x Solo mode.

Choosing the Right Model

Choosing the Right Topology

Technical Differences That Matter

For detailed per-model results, see the individual benchmark articles:

• GPT-OSS-120B Topology Benchmark
• Gemma4-26B Topology Benchmark
• Nemotron-3-Super Topology Benchmark