Tuning Continuous Decode Steps: Pushing GPU Utilization in GLM-5-NVFP4 Inference

The Message in Context

In message [msg 940], the assistant makes a focused, experimental move: it decides to test higher values of --num-continuous-decode-steps (specifically 16, with a plan to also try 32) in an ongoing effort to squeeze more throughput out of the GLM-5-NVFP4 model running on 8x RTX PRO 6000 Blackwell GPUs. The message is short, containing a single bash command that kills the running SGLang server and launches a new one with the updated parameter. But this seemingly simple action sits at a critical juncture in a much deeper investigation into GPU utilization, kernel efficiency, and the architectural bottlenecks of MoE inference on Blackwell hardware.

The Reasoning and Motivation

To understand why this message was written, we need to trace the thread of reasoning that led here. The assistant had just completed a systematic benchmarking run comparing the old server configuration (max-running-requests=1024, num-continuous-decode-steps=4) against a new configuration (max-running-requests=2048, num-continuous-decode-steps=8). The results were striking: at 2048 concurrency, throughput jumped from 1,640 to 2,095 output tok/s — a 28% improvement (see [msg 937]).

This improvement came from a simple insight: with 256 experts and only 8 active per token, the per-expert batch size during decode is tiny. At 1024 concurrent tokens, each expert processes roughly 32 tokens. At 2048 concurrent tokens, that doubles to ~64 tokens per expert. Larger per-expert batches mean the GPU's tensor cores can achieve higher utilization on the FP4 GEMM operations that dominate decode time.

But there was a puzzle. Despite achieving 100% GPU utilization (as reported by nvidia-smi), the power draw remained stubbornly at ~235W per GPU — barely 39% of the 600W TDP (see [msg 938]). This indicated that while the GPU schedulers were busy, the actual compute units were not fully saturated. The FP4 GEMM kernels were running far below their theoretical peak: the CUTLASS kernels plateau at ~1,300 TFLOPS (70% of dense peak) only for very large matrices, but during actual decode, per-expert batch sizes of 16–64 tokens achieve merely 0.8–55 TFLOPS — 0.02–3% of peak (see chunk 0 summary).

The num-continuous-decode-steps parameter controls how many decode iterations the server performs before re-checking the request queue for new work. Higher values mean the server commits to a fixed batch of requests for multiple consecutive decode steps, which keeps the batch "hot" — no new requests are inserted mid-stream, so the batch composition remains stable and the GPU can work through its decode steps without interruption. This reduces scheduling overhead and, crucially, allows the system to maintain larger effective batch sizes per step.

The assistant had already seen benefit from moving from 4 to 8 continuous decode steps. The natural next question was: would 16 or 32 steps yield even more improvement? The message captures this moment of extrapolation — the assistant is testing whether the relationship between continuous decode steps and throughput is monotonic, or whether it plateaus (or even reverses) at higher values.

The Decision Process

The decision to try num-continuous-decode-steps=16 (and later 32) was driven by a clear chain of reasoning visible in the preceding messages:

  1. The bottleneck is compute, not communication. A TP4+PP2 benchmark had confirmed the model is compute-bound, ruling out allreduce latency as the primary issue (chunk 0).
  2. Per-expert batch size is the key lever. With 8 active experts per token and 256 total experts, the effective batch per expert is concurrent_tokens * (8/256). Doubling concurrent tokens doubles per-expert batch size.
  3. The KV cache has headroom. At 495K tokens of KV cache capacity and ~256 tokens per request, the system can handle ~1,935 concurrent requests. The max-running-requests=2048 setting was already pushing toward this limit.
  4. Continuous decode steps compound the effect. By batching more decode steps together, the server maintains larger effective batches for longer, reducing the overhead of re-scheduling and allowing the GPU to stay in a steady-state compute pattern. The assistant's decision was methodical: kill the old server, create a new launch script with the updated parameter, and restart. The bash command uses pkill -9 -f sglang to ensure a clean shutdown, then writes a new script (run_tp8_cds16.sh) that mirrors the previous configuration except for the --num-continuous-decode-steps 16 flag. This controlled experiment design — changing exactly one variable at a time — reflects good scientific practice.

Assumptions Made

The message rests on several assumptions, some explicit and some implicit:

Explicit assumption: "The idea is that with 8 steps, we do 8 decode iterations without checking for new requests, which keeps the batch hot." The assistant assumes that "keeping the batch hot" — i.e., maintaining a stable set of requests across multiple decode steps — is the mechanism driving the improvement, and that more steps will amplify this effect.

Implicit assumption of monotonicity: The assistant assumes that going from 8 to 16 steps will continue to improve throughput, or at least not harm it. This is not guaranteed — higher continuous decode steps mean the server is less responsive to new requests, which could increase queueing delay and reduce the effective concurrency. If the server doesn't check for new requests often enough, it might underfill the batch, leading to lower throughput.

Implicit assumption of KV cache stability: The assistant assumes that the KV cache can sustain the increased decode depth. With 16 continuous decode steps, each request generates 16 new tokens before the server re-checks the queue. This means the KV cache grows by 16 tokens per request per batch cycle, which could accelerate cache pressure.

Implicit assumption of clean restart: The assistant assumes that killing the server with pkill -9 -f sglang and immediately restarting will work cleanly. In practice, this had caused issues before — earlier attempts to restart had failed because the heredoc didn't write properly (see [msg 931]), and orphaned shared memory objects had caused warnings (see [msg 926]).

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Knowledge of SGLang server parameters: What --num-continuous-decode-steps does and how it affects scheduling behavior. This is a relatively advanced parameter that controls the trade-off between throughput and latency.
  2. The previous benchmark results: The 28% improvement at 2048 concurrency (from [msg 937]) provides the baseline against which this experiment will be measured.
  3. The hardware context: 8x RTX PRO 6000 Blackwell GPUs with 600W TDP each, but only drawing ~235W during inference. This power gap is the core problem the assistant is trying to solve.
  4. The model architecture: GLM-5-NVFP4 uses 256 experts with 8 active per token, which creates the sparse MoE compute pattern that makes per-expert batch size so critical.
  5. The KV cache constraint: 495K tokens of KV cache capacity limits the maximum number of concurrent requests to ~1,935 (at 256 tokens per request).
  6. The server configuration: TP8 (tensor parallelism across 8 GPUs), flashinfer attention backend, trtllm NSA backends, flashinfer_cutlass MoE runner — each of these choices constrains the optimization space.

Output Knowledge Created

This message produces several forms of output:

Immediate output: A new server process running with --num-continuous-decode-steps 16, ready for benchmarking. The launch script is written to disk at /root/run_tp8_cds16.sh, creating a reproducible configuration.

Experimental data (to be produced): The subsequent benchmark results will reveal whether 16 continuous decode steps improve throughput beyond the 2,095 output tok/s achieved with 8 steps. This data will inform the final optimization plan.

Documentation artifact: The launch script itself serves as a record of the configuration tested, contributing to the systematic exploration documented in the glb5improvement-xx.md files mentioned in the chunk summary.

The Thinking Process

The reasoning visible in this message is a clear example of iterative optimization through controlled experimentation. The assistant:

  1. Observes a phenomenon: Throughput improved when going from cds=4 to cds=8.
  2. Forms a hypothesis: More continuous decode steps keep the batch hotter, improving throughput.
  3. Designs an experiment: Test cds=16 (and later cds=32) while keeping all other parameters identical.
  4. Executes precisely: Kills the old server, creates a new script with exactly one parameter changed, and restarts. The phrase "the idea is that with 8 steps, we do 8 decode iterations without checking for new requests, which keeps the batch hot" reveals the assistant's mental model of the system. It sees the scheduling loop as a source of overhead — every time the server checks for new requests, it potentially disrupts the smooth flow of decode operations. By batching more decode steps together, the server can achieve a more efficient steady state. The mention of "Let me try 16 and 32" also reveals that the assistant is thinking in powers of two (4 → 8 → 16 → 32), suggesting a systematic sweep rather than random exploration. This binary search pattern is common in performance tuning, where parameters often have logarithmic effects.

Potential Mistakes and Incorrect Assumptions

While the reasoning is sound, several potential issues deserve scrutiny:

The "hot batch" hypothesis may have diminishing returns. The benefit of continuous decode steps comes from maintaining batch stability. But after a certain point, the batch is already as "hot" as it can get — the GPU is already saturated with work, and additional steps just mean the server is less responsive to new requests. The 28% improvement from cds=4 to cds=8 might not replicate from cds=8 to cds=16.

Increased latency for individual requests. Higher continuous decode steps mean a request that arrives just after the batch starts must wait longer before being admitted. This increases the Time Per Output Token (TPOT) for individual requests, even if aggregate throughput improves. The assistant's benchmarks measure throughput but don't directly measure tail latency, which could degrade significantly.

The KV cache pressure is real. With 16 continuous decode steps, each request generates 16 tokens before the next scheduling point. If the batch is near capacity, this could cause the KV cache to overflow, forcing the server to either reject requests or stall. The assistant had already observed that at 3072 concurrency, throughput plateaued at ~2,062 tok/s — the same as 2048 — because the KV cache was full (see <msg id=937-938>).

The restart might fail silently. Earlier in the conversation, the assistant had trouble with heredoc-based script creation (the file wasn't written in [msg 931]). The same issue could recur here, and the assistant doesn't verify that the new server is actually running before proceeding to benchmarks.

Conclusion

Message [msg 940] captures a pivotal moment in a systematic optimization campaign. The assistant has identified that GPU utilization is the key bottleneck, has found that increasing per-expert batch size through higher concurrency and continuous decode steps yields real improvements, and is now probing the limits of this approach. The experiment with num-continuous-decode-steps=16 represents a methodical, hypothesis-driven approach to performance tuning — change one variable, measure the effect, and iterate. Whether or not this particular experiment succeeds, the reasoning process it embodies is the essence of systems optimization: understand the bottleneck, form a hypothesis, test it precisely, and learn from the result.