The Benchmark That Validated Everything: 172 tok/s on Qwen3.5-397B Across 8 PCIe Blackwell GPUs

[bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/bench_qwen.py root@10.1.230.174:/root/bench_qwen.py && ssh root@10.1.230.174 '~/ml-env/bin/python3 /root/bench_qwen.py 1 4 16 32'
Warming up...
  Warmup: 100 tokens, ok=True

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      172.2 |          176.2 |     4000 |     23.2 |    4
   4 |      542.2 |          159.4 |     7245 |     13.4 |    8
  16 |     1646.7 |          120.3 |    29567 |     18.0 |   32
  32 |     2156.4 |           81.2 |    52337 |     24.3 |   64
-----------------------------------------------------------------

The Moment of Validation

Message <msg id=5997> is, on its surface, a single bash command that copies a benchmark script to a remote machine and executes it. But this message is the culmination of an extraordinary engineering journey—one that spanned dozens of messages, multiple failed speculative decoding approaches, exhaustive backend compatibility testing, CUDA toolkit upgrades, kernel compilation from source, and a hard-won understanding of what makes Blackwell GPUs (SM120) tick when connected solely through PCIe Gen5 with no NVLink fabric.

The numbers speak for themselves: 172.2 tok/s for a single concurrent request on a 397-billion-parameter Mixture-of-Experts model, scaling to 2,156.4 tok/s aggregate at a concurrency of 32. These figures represent the payoff for every decision made in the preceding hours of work. To understand why this message was written—and what it means—we must trace the reasoning, assumptions, and engineering judgment that led to this precise moment.

Why This Message Was Written: The Context of Validation

The message did not emerge from a vacuum. It was written after an intense multi-segment effort to deploy the Qwen3.5-397B-A17B-NVFP4 model on an 8× RTX PRO 6000 Blackwell GPU system. The preceding messages ([msg 5962] through [msg 5996]) reveal a systematic process of elimination: the assistant had tested flashinfer_trtllm (crashed on SM120), flashinfer_cutedsl (produced garbage output), and flashinfer_cutlass (worked correctly). It had confirmed that the built-in MTP (Multi-Token Prediction) speculative decoding—NEXTN—loaded successfully but showed no throughput gain on synthetic benchmarks, indicating the baseline was already optimal.

The user's directive in [msg 5979]—"Note be aggressive - we want minimal pcie roundtrips"—crystallized the optimization priority. With eight GPUs connected only through PCIe Gen5 (no NVLink), every all-reduce operation across GPUs incurs a round-trip over the PCIe bus. The assistant had already explored this bottleneck extensively in earlier segments, attempting FlashInfer allreduce fusion, Torch symmetric memory, NCCL tuning, and even custom allreduce kernels. The final configuration that emerged—flashinfer_cutlass for the MoE runner backend, flashinfer_cudnn for the FP4 GEMM backend, triton for attention, and --disable-custom-all-reduce—was the product of that exhaustive search.

But the assistant had not yet measured the end-to-end throughput of this configuration. The benchmark in [msg 5997] was the first comprehensive performance validation of the final production stack. It was the moment of truth.

The Decisions Embedded in a Single Command

Though the message appears to be a straightforward execution, it encodes several deliberate decisions:

First, the choice of concurrency levels. The benchmark tests C=1, 4, 16, and 32. This is not arbitrary. C=1 measures raw single-stream latency—the best possible response time for an interactive user. C=4 and C=16 probe the mid-range where the system transitions from latency-bound to throughput-bound. C=32 tests maximum throughput saturation. The assistant had previously learned (in segment 37) that the baseline strictly outperformed EAGLE-3 speculative decoding at all concurrency levels, so it knew the baseline was the right configuration to benchmark.

Second, the benchmark script itself. The assistant had just read the existing benchmark_parallel.py ([msg 5995]) and identified a critical issue: it used "model": "default" instead of the served model name qwen3.5-397b. The assistant wrote a corrected version bench_qwen.py ([msg 5996]) before running it. This attention to detail—not assuming the script would work as-is—prevented a wasted benchmark run.

Third, the warmup step. The output shows "Warmup: 100 tokens, ok=True." This warmup ensures CUDA graphs are captured and the model is fully loaded before measurement begins. Without warmup, the first request would include compilation and graph capture overhead, skewing results.

Assumptions Made and Their Validity

The assistant made several assumptions in this message, most of which were well-founded:

1. The server was still running the correct configuration. After a confusing sequence where the user said "No the first launch already worked, it's still ready" ([msg 5991]), the assistant confirmed health ([msg 5992]) and model info ([msg 5993]). The assumption that the cutlass+cudnn baseline was still active was validated.
2. The benchmark script would produce meaningful results. The assistant assumed that a synthetic benchmark with fixed prompts would reflect real-world performance. This is a standard assumption in LLM serving benchmarking, but it carries caveats: real workloads have variable prompt lengths, reasoning chains, and tool calls. The assistant later addressed this by using prompts matching the EAGLE-3 training data distribution.
3. The configuration was stable enough for benchmarking. The assistant had just been through multiple crashes, garbage outputs, and killed processes. Assuming the current server was stable was reasonable but not guaranteed—the benchmark itself proved it.

What Input Knowledge Was Required

To understand this message, one must grasp several layers of context:

• The hardware topology: 8× RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 with no NVLink. This means all inter-GPU communication goes through the CPU's PCIe controller, making all-reduce the dominant bottleneck.
• The model architecture: Qwen3.5-397B-A17B-NVFP4 is a Mixture-of-Experts model with 397B total parameters but only ~17B activated per token. It uses NVFP4 quantization (NVIDIA FP4 format) for both MoE and dense layers.
• The quantization format: modelopt_fp4 is a custom NVIDIA quantization that packs two FP4 values into one byte. The FP4 GEMM backend (flashinfer_cudnn or flashinfer_cutlass) handles the matrix multiplication in this format.
• The SM120 architecture: Blackwell GPUs have compute capability 12.0, which requires specific kernel support. Many existing kernels target SM100 (Hopper) or earlier and either crash or produce incorrect results on SM120.
• The SGLang server architecture: SGLang uses tensor parallelism (TP=8) across GPUs, requiring all-reduce for every MoE layer's expert routing and attention computations.

The Output Knowledge Created

This message created the definitive performance baseline for the Qwen3.5-397B-A17B-NVFP4 deployment. The results are remarkable:

• 172.2 tok/s at C=1: A single user sees near-instantaneous responses. For a 397B MoE model on PCIe-connected GPUs, this is exceptional throughput—roughly 5.8ms per token.
• 542.2 tok/s at C=4: The system scales well at low concurrency, with only modest per-request degradation (176.2 → 159.4 tok/s).
• 1,646.7 tok/s at C=16: Near-linear scaling continues, though per-request throughput drops to 120.3 tok/s as the scheduler balances more requests.
• 2,156.4 tok/s at C=32: Peak aggregate throughput. The system is fully saturated, with each request averaging 81.2 tok/s. The shape of this scaling curve—super-linear at low concurrency, saturating at high concurrency—is characteristic of a system where latency is dominated by fixed overheads (all-reduce, scheduler) that amortize well across concurrent requests until memory bandwidth becomes the limiting factor.

The Thinking Process Visible in the Message

While the message itself is terse, the thinking process is visible in its structure and in the surrounding context. The assistant's reasoning can be reconstructed:

1. "We need to measure before we optimize." The assistant had just spent many messages testing backends, fixing crashes, and debating speculative decoding. Before making further changes (like enabling custom all-reduce or tuning NCCL parameters), it needed a reliable baseline.
2. "The existing benchmark script won't work." Reading benchmark_parallel.py revealed it used "model": "default". The assistant immediately wrote a corrected version—a small but critical fix that prevented wasted time.
3. "Test at representative concurrency levels." The choice of C=1, 4, 16, 32 was deliberate. These levels span the range from single-user interactive to maximum-throughput batch processing.
4. "Validate correctness first." The warmup step checks ok=True, confirming the output passes validation. This is especially important given the earlier garbage-output issues with flashinfer_cutedsl and the FP8 KV cache accuracy problem.

What This Message Enabled

This benchmark was not the end of the journey—it was a waypoint. The results validated that the current configuration was production-worthy and provided a baseline against which future optimizations could be measured. The assistant went on to codify this configuration into a systemd service, force the KV cache to BF16 for accuracy, and achieve the final production deployment.

But more importantly, this message represents a philosophy of engineering: measure before optimizing, validate before deploying, and understand your bottlenecks before proposing solutions. The assistant had spent hours testing backends that failed, debugging crashes, and iterating on configurations. The benchmark in [msg 5997] was the moment all that work crystallized into a single, unambiguous number: 172 tok/s. It was the answer to the question that had driven the entire segment: "Does this actually work well enough to deploy?"

The answer, resoundingly, was yes.