The Benchmark That Confirmed a Ceiling: Measuring `--num-continuous-decode-steps 4` on GLM-5-NVFP4

Introduction

In the high-stakes world of large-scale inference optimization, negative results are often the most informative. Message [msg 824] captures one such moment: a single benchmark command and its output that, on its surface, reports modest throughput numbers for the GLM-5-NVFP4 model running on 8 RTX PRO 6000 Blackwell GPUs. But beneath the terse statistics lies the culmination of a multi-hour debugging saga spanning kernel patches, NCCL tuning experiments, allreduce fusion reversions, and a growing recognition that the hardware topology imposes fundamental constraints no amount of software optimization can fully circumvent.

This message is the assistant's systematic measurement of whether --num-continuous-decode-steps 4—a parameter that batches multiple decode steps before yielding control back to the scheduler—can improve throughput by amortizing the overhead of inter-GPU communication. The answer, delivered plainly in the benchmark output, is no: the total token throughput of 2,741.74 tok/s is essentially indistinguishable from the ~2,800 tok/s baseline established without this flag. This article examines why this message was written, the reasoning that led to it, the assumptions that shaped the experiment, and what the result reveals about the state of the system.

The Message

The message consists of a single bash command executed over SSH on the remote inference server, followed by its output:

[assistant] Now let me run the benchmark suite:
[bash] ssh root@10.1.230.174 "source /root/ml-env/bin/activate && echo '=== 512 concurrency ===' && python3 -m sglang.bench_serving --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --base-url http://localhost:8000 --dataset-name random --num-prompts 512 --random-input-len 256 --random-output-len 128 --request-rate inf 2>&1 | grep -E 'Output token throughput|Total token throughput|Peak output|Concurrency:|Mean TPOT'"
=== 512 concurrency ===
Output token throughput (tok/s):         901.74    
Peak output token throughput (tok/s):    2310.00   
Total token throughput (tok/s):          2741.74   
Concurrency:                             356.86    
Mean TPOT (ms):                          409.58

The command invokes SGLang's built-in bench_serving script with 512 random prompts of 256 input tokens and 128 output tokens each, at infinite request rate (maximizing server load). The output is filtered to five key metrics: output token throughput, peak output token throughput, total token throughput, achieved concurrency, and mean time per output token (TPOT).

The Optimization Journey: How We Got Here

To understand why this particular benchmark was run, one must trace the assistant's trajectory through the preceding messages. The context begins with the assistant having achieved a working baseline of approximately 1,867 tok/s at 256 concurrency using the flashinfer_cutlass MoE backend with CUTLASS autotuning enabled for SM120 ([msg 805]). By raising --max-running-requests from 64 to 1024, throughput scaled to ~2,800 tok/s at 512 concurrency and ~3,740 tok/s at 1024 concurrency—a remarkable 4× improvement over the initial ~880 tok/s.

However, GPU power draw remained stubbornly around 250–330W per card, well below the 600W TDP. The assistant correctly identified that the GPUs were compute-starved, not memory-bound: the SMs were active but stalling on inter-GPU allreduce operations required by the Mixture-of-Experts (MoE) architecture. This diagnosis led to the most ambitious intervention: patching FlashInfer's allreduce fusion kernel to add SM120 support, enabling a specialized fused communication primitive that combines the allreduce with the MoE computation to overlap communication with compute.

The allreduce fusion attempt ([msg 799]–[msg 801]) was a dramatic failure. The kernel compiled and initialized successfully—a nontrivial achievement given that SM120 (consumer Blackwell) was not a supported target—but throughput collapsed to 236 tok/s and power dropped to 125W. The assistant correctly diagnosed that cudaGridDependencySynchronize(), the synchronization primitive used by the fusion kernel, likely behaves differently on SM120 than on SM90/SM100, causing excessive serialization or deadlock. The patch was reverted ([msg 802]).

The Reasoning Behind This Message

After reverting the allreduce fusion, the assistant enumerated alternative optimization strategies ([msg 804]): NCCL tuning, the flashinfer_trtllm MoE backend, overlap scheduling, and --num-continuous-decode-steps. The NCCL tuning attempts were rocky—NCCL_ALGO=Tree crashed with an AllGather incompatibility for int8 data ([msg 812]), and NCCL_MAX_NCHANNELS=32 caused a different crash ([msg 815]). After cleaning up leaked semaphores and restarting with conservative NCCL settings (NCCL_MIN_NCHANNELS=8), the assistant finally got a clean server running with --num-continuous-decode-steps 4 ([msg 822]–[msg 823]).

Message [msg 824] is therefore the first systematic measurement after this restart. The assistant's stated intent—"Now let me run the benchmark suite"—belies a deeper purpose: to determine whether batching decode steps can compensate for the allreduce overhead that the fusion kernel failed to address. The --num-continuous-decode-steps parameter instructs the scheduler to run multiple decode iterations back-to-back without yielding, increasing the computational density between communication barriers. In theory, this should improve GPU utilization by reducing the frequency of allreduce operations relative to compute.

Assumptions and Their Validity

The assistant made several assumptions in designing this experiment:

Assumption 1: Continuous decode steps would improve throughput. This assumption was reasonable given the observed symptom—GPUs at 98% utilization but only 55% of TDP power—which suggested that the compute units were idle waiting for communication. Batching decode steps should increase the compute-to-communication ratio. However, the assumption failed to account for the possibility that the bottleneck was not the frequency of allreduce operations but their intrinsic latency. If each allreduce takes tens of milliseconds due to PCIe P2P latency (a known issue in the Proxmox VM environment, as documented in earlier segments), then batching more decode steps between allreduces doesn't help—the GPU still stalls on each individual allreduce call.

Assumption 2: The NCCL settings used (NCCL_MIN_NCHANNELS=8, no algorithm override) were sufficient for the continuous-decode-steps experiment. The assistant had just spent several messages debugging NCCL crashes caused by aggressive tuning parameters. The conservative settings avoided crashes but may have left performance on the table. It's possible that --num-continuous-decode-steps 4 combined with better NCCL tuning (e.g., more channels, different buffer sizes) could have yielded gains, but the assistant chose to isolate the decode-steps variable first.

Assumption 3: 512 concurrency was the right test point. The assistant had previously observed that throughput scaled with concurrency up to 1024. Testing at 512 concurrency provided a middle ground—high enough to stress the system but not so high that scheduler overhead dominated. This was a pragmatic choice, but it meant the result might not generalize to other concurrency levels.

Assumption 4: The benchmark was reproducible. The assistant did not run multiple trials or report variance. The single measurement of 2,741.74 tok/s is treated as definitive, but real-world inference throughput can vary due to autotuning state, memory fragmentation, and thermal conditions.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

SGLang's architecture and server arguments: Understanding what --num-continuous-decode-steps does, how the scheduler works, and how bench_serving measures throughput.
MoE inference communication patterns: The allreduce operations that synchronize expert activations across GPUs in tensor-parallel deployment, and why they become a bottleneck at scale.
NCCL tuning parameters: The meaning of NCCL_MIN_NCHANNELS, NCCL_P2P_LEVEL, and NCCL_IB_DISABLE, and how they affect inter-GPU communication performance.
GPU utilization metrics: The relationship between SM clock speed, power draw, utilization percentage, and actual compute throughput—specifically, why 98% utilization with 330W power draw on a 600W TDP card indicates a compute pipeline stalled on communication.
The GLM-5-NVFP4 model characteristics: A Mixture-of-Experts model with 8 experts, requiring allreduce operations for expert-parallel communication, quantized to NVFP4 format using modelopt.
The hardware topology: 8 RTX PRO 6000 Blackwell GPUs in a Proxmox VM, with each GPU on its own PCIe root complex, preventing direct P2P DMA and forcing all inter-GPU communication through the host's PCIe fabric.

Output Knowledge Created

This message produces several forms of knowledge:

Direct knowledge: The benchmark establishes that --num-continuous-decode-steps 4 does not improve throughput for this specific configuration. The total token throughput of 2,741.74 tok/s (901.74 output tok/s) at 512 concurrency with a mean TPOT of 409.58 ms is essentially identical to the ~2,800 tok/s baseline without the flag.

Comparative knowledge: By maintaining the same test methodology as earlier benchmarks, the assistant enables direct comparison. The 2,741.74 tok/s result can be compared to the ~1,950 tok/s at 256 concurrency, the ~2,800 tok/s at 512 concurrency without continuous decode steps, and the ~3,740 tok/s at 1024 concurrency. This creates a performance profile that reveals the system's scaling behavior.

Negative knowledge: The most valuable output is the negative result—continuous decode steps don't help. This eliminates one optimization path and forces the assistant to consider alternatives. The flashinfer_trtllm MoE backend, mentioned as a next step in the chunk summary, becomes the next candidate.

Diagnostic knowledge: The concurrency metric of 356.86 (achieved concurrency vs. 512 requested) indicates that the server is not fully saturated at this request rate. The gap between peak output throughput (2,310 tok/s) and sustained output throughput (901.74 tok/s) suggests burst-followed-by-queue behavior typical of a system where the bottleneck is not compute throughput but request scheduling or communication latency.

The Thinking Process Visible in the Message

The message itself is terse—a single command and its output—but the reasoning behind it is visible through the surrounding context. The assistant's thinking follows a clear pattern:

Problem identification: GPU power is only 55% of TDP despite 98% utilization. The SMs are stalled on communication.
Hypothesis generation: Allreduce fusion could overlap communication with compute, improving utilization.
Hypothesis testing (failed): Allreduce fusion patch causes throughput to collapse to 236 tok/s.
Hypothesis revision: If fusion doesn't work, try batching decode steps to reduce communication frequency.
Experimental setup: Start server with --num-continuous-decode-steps 4, conservative NCCL settings.
Measurement: Run benchmark at 512 concurrency.
Evaluation: Compare result to baseline. The assistant does not explicitly state the comparison in the message, but the reader can infer from the context that 2,741.74 tok/s is essentially the same as the ~2,800 tok/s baseline. The absence of any commentary like "improvement!" or "still the same" is itself telling—the assistant moves on to the next experiment without dwelling on the negative result. This pattern of hypothesis-driven optimization, with rapid iteration and systematic measurement, is characteristic of the assistant's approach throughout the session. Each message builds on the knowledge gained from previous experiments, and even "failed" experiments produce valuable diagnostic information.

Broader Implications

The failure of both allreduce fusion and continuous decode steps to improve throughput points to a deeper constraint: the PCIe P2P latency inherent in the Proxmox VM's GPU topology. Earlier segments of the conversation ([msg 795]–[msg 823] context) documented the assistant's discovery that each GPU sits on its own PCIe root complex, preventing direct GPU-to-GPU DMA. All inter-GPU communication must traverse the host's PCIe fabric, adding latency that no amount of kernel fusion or batching can eliminate.

This is a hardware topology problem masquerading as a software optimization problem. The assistant's attempts to fix it through kernel patches, NCCL tuning, and scheduler parameters are all addressing the wrong layer—they optimize the communication implementation but cannot change the communication path. The only real solutions would be hardware-level (enabling P2P DMA through ACS override or IOMMU group merging, attempted in Segment 3) or architectural (switching from TP8 to TP4+PP2 to reduce the allreduce domain size).

The message thus serves as a diagnostic waypoint: it confirms that the current optimization strategies have hit a ceiling imposed by the virtualization layer. The assistant's next steps—exploring the flashinfer_trtllm MoE backend—represent a continuation of the search for software-level gains, but the fundamental bottleneck remains unresolved.

Conclusion

Message [msg 824] is a study in disciplined engineering measurement. In a single command, the assistant tests a hypothesis, produces a clear result, and generates actionable knowledge—even though that knowledge is negative. The benchmark output of 2,741.74 total tok/s tells the assistant (and the reader) that --num-continuous-decode-steps 4 is not the path forward. The real value lies not in the number itself but in what it represents: the systematic narrowing of the optimization space, the elimination of one more variable, and the growing clarity that the bottleneck lies deeper than any software parameter can reach.

The Benchmark That Confirmed a Ceiling: Measuring --num-continuous-decode-steps 4 on GLM-5-NVFP4