Found 11 posts
Same hardware, three models, completely different performance profiles. GPT-OSS-120B is fastest despite 117B params. Gemma4 has the best TTFT. Nemotron never loses to shuffle. The right model depends on the workload.
The smallest model benefits most from tensor parallelism: +50% at c1. Cluster TP=2 with --no-ray dominates decode through c32. Also: Ray vs PyTorch distributed -- the 2-8% you're leaving on the table.
The model where cluster never loses. Nemotron-3-Super benefits more from TP=2 than any other model tested -- and the SM12.1 CUTLASS patch doubles performance vs FlashInfer.
Cluster TP=2 dominates decode up to c32, then simple-shuffle takes over. Least-busy routing collapses under load. Full topology comparison with four configurations.
Everything in the series built toward this: Claude Code running on locally served models. Here's what works, what's rough, and where it's heading.
The recipe system from spark-vllm-docker turns twenty minutes of flag archaeology into one command — and makes everything reproducible.
Neither topology dominates — cluster wins decode at every concurrency level, but 2x Solo wins prefill and TTFT under load. The right choice depends on the workload.
The second DGX Spark arrived. Before writing a single line of config: check firmware. Then cables, SSH, Docker, vLLM, model cache — and Claude Code helping build the skills to manage it all.
Claude Code speaks Anthropic. gpt-oss-120b speaks OpenAI with Harmony-style tool calls. LiteLLM sits in the middle and translates — including a custom callback that patches the tool calls neither side gets right.
Post 3 hit 4,158 t/s at c64. llama-benchy puts those numbers under the microscope. Same hardware, two tools, 25x difference in TTFT.
Every reviewer tested single-user latency and called the DGX Spark slow. Nobody tested concurrency. The community found the real number: 3,975 tokens per second.