Nemotron-3-Super-120B: Topology Benchmark on DGX Spark

The Model

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 — 120B total parameters, 12B active per token. LatentMoE (Mamba-2 + MoE + Attention hybrid) with NVFP4 quantization. At 69.5 GiB, it nearly fills a single DGX Spark’s memory budget.

Nemotron-3-Super is the slowest model in this benchmark series by raw token speed, but it has unique characteristics: a reasoning-heavy architecture that generates thinking tokens before content, and 12B active params that create an ideal ratio for tensor parallelism — enough compute to justify two GPUs, but small enough NCCL traffic to keep synchronization cheap.

This is also the model that exposed the SM12.1 kernel problem. Without the CUTLASS patch, FlashInfer/Marlin backends run at half speed.

Test Setup

• Hardware: 2x NVIDIA DGX Spark (Grace Blackwell GB10, 128 GB unified each)
• Interconnect: ConnectX-7 dual 200GbE QSFP stacking link
• Image: vllm-experimental (vLLM 0.17.2rc1.dev0 + SM12.1 FP8 CUTLASS patch)
• Patch: saifgithub/vllm-gb10-sm121
• Recipe: nvidia-nemotron-3-super-120b-nvfp4-cutlass.yaml (gpu_memory_utilization=0.70, max_model_len=262144)
• Benchmark: llama-benchy 0.3.5, pp1024 tg128, 50 runs per concurrency level
• Date: 2026-03-30

Topologies Tested

Decode Throughput

Generation speed (tg128) — higher is better. Solo c64 omitted (OOM).

Legend: ━━ Solo · ━━ Cluster TP=2 · ━━ Shuffle · ━━ Least-busy

Cluster TP=2 wins or ties decode at every concurrency level. This is unique among the three models tested — GPT-OSS-120B and Gemma4 both lose to shuffle at c64.

The advantage is largest at c1 (+40%) and narrows to +3% at c32 before converging to a tie at c64. Least-busy tracks Solo almost exactly — confirming it’s effectively a single-node topology due to backend starvation.

Prefill Throughput

Prompt processing speed (pp1024) — higher is better.

Legend: ━━ Solo · ━━ Cluster TP=2 · ━━ Shuffle · ━━ Least-busy

Prefill tells a different story. Cluster wins at c1-c2 where its faster per-request processing matters, but shuffle overtakes at c4 and keeps pulling away. At c64, shuffle leads by 23%.

Note how flat Solo’s prefill curve is — it saturates early at ~1,550 t/s and can’t grow regardless of concurrency. The 69.5 GiB model leaves limited KV cache budget.

Time to First Token (TTFT)

End-to-end latency to first generated token (ms) — lower is better.

Legend: ━━ Solo · ━━ Cluster TP=2 · ━━ Shuffle · ━━ Least-busy

The TTFT crossover is later than the other models — cluster holds the lead through c16 and only loses at c32. This reflects Nemotron’s higher per-request latency: each request is expensive enough that tensor parallelism helps more than request independence.

Least-busy at c64 is catastrophic: 22.8 seconds TTFT, nearly 2x worse than cluster.

The CUTLASS Patch: 2x Performance

This benchmark only exists because of the SM12.1 CUTLASS patch. Without it, Nemotron-3-Super runs at half speed on DGX Spark.

Legend: ━━ FlashInfer+Marlin · ━━ Old v0.17.1 PTX · ━━ CUTLASS SM12.1

Why FlashInfer/Marlin are slow on SM12.1

vLLM’s kernel priority: Marlin > FlashInfer > CUTLASS > Torch. On DGX Spark (SM12.1):

• MarlinFP8 is rejected by default (capability gate >= 89), unless VLLM_TEST_FORCE_FP8_MARLIN=1 forces it — which the original recipe did, actively hurting performance
• FlashInferFP8 is selected next (capability >= 100) — functional but not optimized for SM12.1
• CutlassFP8 would crash without the enable_sm120_family patch (SM12.1 cubins hit asm("trap"))

The fix is to disable FlashInfer/Marlin and let the patched CUTLASS kernels handle everything:

env:
  VLLM_NVFP4_GEMM_BACKEND: "cutlass"
  VLLM_MARLIN_USE_ATOMIC_ADD: "1"
  VLLM_USE_FLASHINFER_MOE_FP4: "0"
  VLLM_DISABLED_KERNELS: "MarlinFP8ScaledMMLinearKernel,FlashInferFP8ScaledMMLinearKernel"

Routing: Least-busy Starvation Confirmed

Third model, same result. Least-busy at c4+ matches Solo throughput exactly — it’s routing almost all traffic to one backend.

Never use least-busy for identical backends. This is now confirmed across GPT-OSS-120B, Gemma4, and Nemotron-3-Super.

When to Use Which Topology

Cluster TP=2 is the recommended default for Nemotron-3-Super — it’s the only model where cluster wins or ties decode at every concurrency level. The 12B active params create ideal tensor parallelism conditions.

The CUTLASS recipe is mandatory — FlashInfer/Marlin backends are 2x slower on SM12.1.

Technical Notes

• GLOO_SOCKET_IFNAME: Required on vLLM v0.17.2+ for cluster mode (enp1s0f0np0). Independent from NCCL_SOCKET_IFNAME — Gloo and NCCL use separate transports
• Solo c64: OOM — the 69.5 GiB model at gpu_mem 0.70 doesn’t leave enough KV cache for 64 concurrent 1024-token prefills
• Reasoning tokens: Nemotron generates thinking tokens before content, making decode sequences longer and amplifying the benefit of faster per-request decode (cluster advantage)
• SM12.1 patch source: saifgithub/vllm-gb10-sm121 — enables enable_sm120_family for CUTLASS FP8 kernels