The 28% Breakthrough: How Doubling Batch Size Unlocked Hidden Throughput in GLM-5-NVFP4 Inference

Introduction

In the long arc of optimizing GLM-5-NVFP4 inference across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment where months of investigation into kernel internals, shared memory limits, PCIe topology, and virtualization overhead suddenly crystallize into a single, measurable victory. Message [msg 937] captures that moment. It is the message where the assistant benchmarks a newly tuned server configuration and discovers a 28% throughput improvement at high concurrency—a jump from 1,640 to 2,095 output tokens per second, and from 3,249 to 4,151 total tokens per second. This article examines that message in depth: why it was written, the decisions it reflects, the assumptions it validates, and the knowledge it creates.

The Message in Full

The assistant writes:

Excellent — significant improvement at 2048 concurrency thanks to max-running-requests 2048:

| Concurrency | Old (max_rr=1024, cds=4) | New (max_rr=2048, cds=8) | Change | |---|---|---|---| | 256 | 1,383 tot / 705 out | 1,384 tot / 705 out | Same | | 512 | 2,205 tot / 1,106 out | 2,216 tot / 1,111 out | Same | | 1024 | 3,190 tot / 1,601 out | 3,250 tot / 1,631 out | +2% | | 2048 | 3,249 tot / 1,640 out | 4,151 tot / 2,095 out | +28% |

At 2048 concurrency, we jumped from 1,640 to 2,095 output tok/s (and 3,249 to 4,151 total tok/s). The peak output hit 7,216 tok/s! This is because we now have ~2048 concurrent tokens, meaning each expert gets ~64 tokens average — double what we had before.

Let me push further with even more concurrency and see where the KV cache limit actually hits:

The assistant then immediately executes a benchmark at 3072 concurrency, which returns 2,062 output tok/s—essentially plateauing at the same level as 2048, confirming the KV cache ceiling.

Why This Message Was Written: The Reasoning and Motivation

This message sits at the convergence of two parallel lines of investigation that had been running for dozens of prior messages. Understanding why it was written requires tracing both threads.

Thread 1: The Shared Memory Dead End

In the messages immediately preceding [msg 937] (specifically [msg 918] through [msg 923]), the assistant had been deep in the weeds of CUTLASS kernel configuration for the Blackwell SM120 architecture. It discovered that certain large tile configurations (M128×N256 and M256×N128) were failing at runtime with the TRT-LLM grouped GEMM launcher. After extensive investigation—reading CUTLASS source files, tracing the StageCountAutoCarveout logic, and examining the sm120_smem_capacity_bytes = 101376 (99KB) constant—the assistant concluded that these tiles simply could not fit in SM120's shared memory. The mainloop alone required ~91KB for a single pipeline stage, leaving insufficient room for the fused epilogue that handles router scales, bias, and scatter operations.

This was a fundamental hardware constraint, not a software bug. The assistant marked this approach as "DEPRIORITIZED" and pivoted.

Thread 2: The cuBLASLt Investigation

Simultaneously, the assistant had dispatched a research sub-agent (via the task tool) to investigate cuBLASLt FP4 integration ([msg 927]). The hope was that NVIDIA's cuBLASLt library might offer a faster FP4 GEMM path on SM120, potentially bypassing the CUTLASS limitations. The sub-agent returned with a thorough analysis ([msg 928]): cuBLASLt FP4 was available via torch._scaled_mm, but it was actually 5-8% slower than FlashInfer's CUTLASS path for large matrix sizes, and critically, it had no grouped GEMM support for MoE (Mixture of Experts) layers. Since GLM-5-NVFP4 uses 256 experts with 8 active per token, the MoE grouped GEMM is the dominant computation—cuBLASLt was a dead end.

The Pivot to Batch Size

With both the tile-fixing and cuBLASLt approaches ruled out, the assistant turned to what it had identified earlier as the highest-impact lever: increasing the effective per-expert batch size. The core insight was elegant and data-driven. With 256 experts and 8 active per token, at 1024 concurrent tokens each expert processes only ~32 tokens. At 2048 concurrent tokens, that doubles to ~64 tokens per expert. Larger batches mean the GPU's tensor cores can amortize their overhead more effectively, and the CUTLASS kernels can select more efficient tile configurations.

The two parameters that control this are --max-running-requests (how many requests can be in-flight simultaneously) and --num-continuous-decode-steps (how many decode steps are batched before re-scheduling). The assistant doubled both: from 1024 to 2048 for max-running-requests, and from 4 to 8 for continuous decode steps.

How Decisions Were Made

The decision-making process visible in this message is a textbook example of hypothesis-driven optimization.

Decision 1: Which Parameters to Tune

The assistant did not blindly tweak parameters. It had already established, through the earlier shared-memory investigation, that the CUTLASS kernel path was already optimal for the given hardware constraints. The cuBLASLt investigation confirmed there was no faster kernel alternative. This narrowed the search space to scheduling parameters—things that control how many tokens are processed together, rather than how individual matrix multiplications are executed.

The choice of max-running-requests and num-continuous-decode-steps was grounded in the model's architecture: GLM-5-NVFP4 has 256 experts with top-8 routing. Each decode step processes a batch of tokens, and each token activates 8 experts. The total expert computation per step scales linearly with the number of tokens. By increasing both the number of in-flight requests and the number of decode steps batched together, the assistant effectively multiplied the per-expert token count.

Decision 2: The Benchmark Protocol

The assistant used a systematic benchmarking approach: testing at concurrency levels of 256, 512, 1024, and 2048, with 128 input tokens and 128 output tokens per request. This provided a clear baseline comparison against the earlier configuration. The use of sglang.bench_serving with --request-rate 999 (effectively infinite request rate, saturating the server) ensured the benchmark measured peak server throughput rather than client-side limitations.

Decision 3: Pushing Past the Limit

After seeing the 28% improvement, the assistant immediately tested 3072 concurrency. This was a deliberate probe to find the system's ceiling. The result (2,062 output tok/s, essentially flat versus 2048) confirmed that the KV cache was the limiting factor—the server had 495,488 tokens of KV cache, and at ~256 tokens per request (128 input + 128 output), the theoretical maximum was ~1,935 concurrent requests. The 3072 benchmark showed concurrency of 2,072, confirming the system was hitting the KV cache wall.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

Assumption 1: The Bottleneck Is Compute, Not Communication

This was validated in earlier work (referenced in the segment summary for chunk 0). The assistant had benchmarked TP4+PP2 (tensor parallelism 4 + pipeline parallelism 2) and found it was 2× slower than TP8 (tensor parallelism 8). This ruled out allreduce latency as the primary bottleneck—if communication were the issue, TP4+PP2 would have been faster because it reduces the allreduce size. The fact that TP8 was faster confirmed the model is compute-bound, meaning the optimization lever should be increasing compute efficiency (i.e., larger batches).

Assumption 2: Larger Batches Improve Throughput Monotonically

The assistant assumed that increasing concurrency would continue to improve throughput until hitting a resource limit (KV cache). This was validated up to 2048 concurrency, where the 28% improvement occurred. At 3072, throughput plateaued, confirming the assumption was correct but bounded.

Assumption 3: The KV Cache Estimate Is Accurate

The assistant calculated the KV cache limit as 495,488 tokens / 256 tokens per request ≈ 1,935 max concurrent requests. This assumed uniform request lengths (128+128 tokens). In practice, the 3072 benchmark showed concurrency of 2,072, slightly above the estimate, likely because some requests completed faster than others, allowing more to be in flight simultaneously.

Assumption 4: The Old Configuration Was Optimal for Its Parameters

The "Old" column in the comparison table (max_rr=1024, cds=4) was assumed to be a fair baseline. The assistant had previously tuned these parameters to maximize throughput, so the comparison is between two locally-optimal configurations rather than a random starting point versus a tuned one.

Mistakes and Incorrect Assumptions

While the message is largely a success story, there are nuances worth examining:

The 3072 Test Was Premature

The assistant immediately jumped to 3072 concurrency after seeing the 2048 results, without first checking whether the KV cache could accommodate it. The result was predictable: throughput plateaued. A more systematic approach would have been to first calculate the KV cache limit explicitly, then test at the predicted maximum (around 1935) rather than overshooting to 3072. However, the assistant's approach was pragmatic—it took only one additional benchmark to confirm the ceiling, and the overshoot provided useful data about how the server behaves under extreme concurrency (elevated TPOT of 905ms, indicating queuing delays).

The "Peak Output" Metric Is Misleading

The assistant celebrated "peak output hit 7,216 tok/s" but this is a transient peak, not sustained throughput. The sustained output throughput at 2048 concurrency was 2,095 tok/s. The peak metric captures a momentary burst when many requests complete simultaneously, which is highly dependent on the specific timing of request arrivals and completions. It is not a reliable indicator of system capacity. The assistant's focus on sustained throughput in the comparison table is correct, but the peak value is presented without caveats.

No Investigation of Diminishing Returns Below 2048

The comparison table shows that at 256 and 512 concurrency, the new configuration produced identical throughput to the old one. At 1024, there was only a 2% improvement. The 28% gain only appeared at 2048. This suggests that the benefits of num-continuous-decode-steps 8 only materialize when there are enough tokens in flight to fill the larger decode batches. The assistant did not explore why the smaller concurrency levels showed no improvement—was the old configuration already saturating the GPUs at those levels? Was the overhead of larger decode steps (e.g., increased scheduling latency) offsetting the benefits at lower concurrency? These questions remain unanswered.

Input Knowledge Required

To fully understand this message, the reader needs:

GLM-5-NVFP4 architecture knowledge: The model uses 256 experts with top-8 routing, meaning each token activates 8 experts. The MoE grouped GEMM is the dominant computation. FP4 quantization means weights are stored in 4-bit floating point format, requiring specialized kernel support.
SGLang server parameters: --max-running-requests controls how many concurrent requests the server will accept before queuing. --num-continuous-decode-steps controls how many decode iterations are batched together before the scheduler re-evaluates which requests to process. Both affect the effective batch size seen by the GPU kernels.
CUTLASS and GPU kernel concepts: Understanding why larger per-expert batches improve throughput requires familiarity with GPU occupancy, tensor core utilization, and the relationship between matrix dimensions and kernel efficiency. The earlier investigation into SM120's 99KB shared memory limit provides essential context for why the tile-fixing approach failed.
KV cache fundamentals: The server's KV cache capacity (495,488 tokens) determines the maximum number of concurrent requests. Each request consumes KV cache slots proportional to its sequence length. When the cache is full, new requests must wait for existing ones to complete.
Benchmarking methodology: The use of sglang.bench_serving with random input/output lengths, the meaning of "output token throughput" versus "total token throughput" (which includes input tokens), and the significance of "Mean TPOT" (time per output token) and "Concurrency" (average number of requests being processed simultaneously).

Output Knowledge Created

This message creates several important pieces of knowledge:

Empirical Validation of the Batch-Size Hypothesis

The 28% throughput improvement at 2048 concurrency provides strong evidence that GLM-5-NVFP4's MoE layers benefit significantly from larger per-expert batch sizes. This is not merely a server configuration tweak—it is a validation of the compute-bound bottleneck analysis. If the model were communication-bound, increasing batch size would not improve throughput because the allreduce overhead per token would remain constant or increase.

The KV Cache Ceiling

The 3072 benchmark establishes the practical throughput ceiling for this hardware configuration: approximately 2,100 output tok/s and 4,100 total tok/s. Further improvements would require either increasing KV cache capacity (by allocating more GPU memory to the cache via --mem-fraction-static) or reducing per-request KV cache consumption (e.g., via sliding window attention or KV cache quantization).

A Benchmarking Methodology for Future Tuning

The comparison table provides a template for evaluating server configuration changes: test across a range of concurrency levels, report both output and total throughput, and compute the change relative to a consistent baseline. The assistant's approach of testing at 256, 512, 1024, and 2048 concurrency reveals that the improvement is not uniform—it only appears at the highest concurrency level. This is valuable information for anyone tuning similar deployments.

The Peak vs. Sustained Throughput Distinction

The message implicitly raises an important methodological point: peak throughput (7,216 tok/s) and sustained throughput (2,095 tok/s) are very different metrics. The peak value captures a transient burst, while the sustained value reflects steady-state performance. The assistant correctly uses sustained throughput for the comparison table but highlights the peak as an additional data point.

The Thinking Process Visible in the Message

The assistant's reasoning is remarkably transparent in this message. Several cognitive patterns are visible:

Pattern 1: Quantitative Comparison

The assistant immediately constructs a comparison table, mapping the old and new configurations across four concurrency levels. This is not just reporting—it is thinking in structure. The table reveals the pattern that the improvement is concentrated at the highest concurrency level, which the assistant then explains with the per-expert batch size analysis.

Pattern 2: Causal Explanation

The assistant does not stop at "we got 28% more throughput." It provides a causal mechanism: "This is because we now have ~2048 concurrent tokens, meaning each expert gets ~64 tokens average — double what we had before." This connects the configuration change to the architectural property of the model (256 experts, top-8 routing) to the observed outcome. It is a complete causal chain.

Pattern 3: Immediate Next-Step Probing

The assistant's first action after presenting the results is to push further: "Let me push further with even more concurrency and see where the KV cache limit actually hits." This reflects a systematic optimization mindset—find the improvement, then find the ceiling. The 3072 test is not random exploration; it is a deliberate search for the system's upper bound.

Pattern 4: Selective Enthusiasm

The assistant expresses excitement ("Excellent — significant improvement," "The peak output hit 7,216 tok/s!") but immediately channels that enthusiasm into the next experiment. The emotional tone is positive but not complacent—the improvement is celebrated, but the focus quickly shifts to finding the next bottleneck.

Conclusion

Message [msg 937] is a turning point in the GLM-5-NVFP4 optimization journey. After extensive investigation into kernel internals, shared memory constraints, and alternative computation paths—all of which led to dead ends—the assistant achieves a concrete, measurable improvement by doubling the effective batch size. The 28% throughput gain validates the compute-bound bottleneck analysis and demonstrates that for MoE models on Blackwell GPUs, the most impactful optimization lever may not be in the kernel code at all, but in the scheduling parameters that control how many tokens the GPU processes at once.

The message also reveals the assistant's methodology: systematic benchmarking across multiple concurrency levels, causal reasoning connecting configuration changes to architectural properties, and immediate probing for the next bottleneck. It is a model of hypothesis-driven optimization—one that turns a 28% improvement into not just a victory lap, but a launchpad for further investigation.