The Systematic Optimization Campaign: From Documentation to Execution on Blackwell GPUs

Introduction

In the demanding world of large language model inference optimization, the gap between theory and practice is measured in crashed servers, marginal gains, and hard-won negative results. This article synthesizes a critical chunk of an opencode coding session—spanning messages 958 through 1062—in which an AI assistant methodically worked through a prioritized optimization plan for deploying the GLM-5-NVFP4 model (a 744-billion-parameter Mixture-of-Experts language model quantized to NVFP4 precision) across eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The chunk captures a complete arc: from documenting optimization hypotheses, through systematic testing of Tier 1 approaches, to the discovery that the fundamental bottleneck lies not in communication patterns but in the hardware's inability to efficiently execute the small per-expert matrix multiplications that define MoE inference.

This is a story of disciplined engineering. The assistant wrote ten improvement documents in parallel, tested three distinct optimization strategies (Piecewise CUDA Graphs, MSCCLPP allreduce, Single Batch Overlap), ruled them all out as transformative, pivoted to Expert Parallelism (EP8), launched it successfully, benchmarked it, watched it crash under moderate load, and diagnosed the root cause—all within a single continuous session. The overarching narrative is one of systematic hypothesis testing: each optimization was evaluated against a rigorous baseline across four concurrency levels (1, 10, 256, 1024), and each was abandoned when the data failed to support the hypothesis.

The Documentation Phase: Ten Files in One Round

The chunk opens with the assistant completing the documentation phase of the optimization campaign. Earlier in the session, the assistant had dispatched six parallel research sub-agents to explore expert parallelism, CUDA graphs, SM120 kernel optimization, compute/communication overlap, alternative model formats, and memory bandwidth optimization. Their findings had been synthesized into a prioritized three-tier plan with eleven distinct optimization approaches, each to be documented in a file called glb5improvement-XX.md. The first document—on piecewise CUDA graphs—had already been written. The remaining ten existed only as notes in the assistant's context.

Message 963 ([chunk 8.0][6]</chunk>) is the moment when those notes become permanent artifacts. The assistant writes all ten documents in a single round of parallel tool calls, dispatching write() operations for files covering MSCCLPP, Single Batch Overlap, Expert Parallelism, Allreduce Fusion, L2 Cache Pinning, Persistent Grouped GEMM, FP4 Structured Sparsity, Torch Compile, DP2+TP4, and Column-Major X-Sharing. This is not merely a housekeeping task; it is a strategic decision to externalize knowledge from the assistant's context into structured, persistent documents that can guide the execution phase.

The parallelization of these writes reveals a sophisticated understanding of the tool system's capabilities. The assistant recognizes that the ten writes are independent—no document depends on the content of another—and dispatches them simultaneously. This reduces the number of rounds needed from ten (or more) to one, demonstrating an efficiency-first mindset that characterizes the entire chunk.

The Pivot to Execution: Baseline First

With documentation complete, the assistant pivots to execution. Message 964 ([chunk 8.0][7]</chunk>) announces this transition: "All 10 improvement docs written. Now let me move to execution." But the assistant does not immediately jump to testing optimizations. Instead, it first establishes a baseline benchmark across four concurrency levels: 1, 10, 256, and 1024 concurrent requests. This methodological discipline—measure before you change—is a hallmark of rigorous engineering.

The baseline is established using the current best configuration: TP8 (Tensor Parallelism across 8 GPUs), CUDA graphs disabled, and a chunked prefill scheduling parameter (cds16). The assistant runs the benchmark suite, collects throughput numbers at each concurrency level, and records them as the reference point against which all subsequent optimizations will be measured. This baseline would prove invaluable: when MSCCLPP and SBO each delivered only ~2% improvement, the baseline provided the statistical confidence to declare those approaches non-transformative.

Tier 1, Attempt 1: Piecewise CUDA Graphs (Blocked)

The first Tier 1 optimization to be tested is Piecewise CUDA Graphs—a technique that uses CUDA graph capture for parts of the computation while allowing dynamic tensor shapes for others. The expected impact is 10–20% throughput improvement, making it the most promising of the quick-to-test options.

What follows is an extended debugging saga spanning messages 975 through 1014 ([chunk 8.0][18]–[57]</chunk>). The assistant launches the server with --enable-cuda-graph --enable-piecewise-cuda-graphs, only to encounter a crash during model loading. The error trace points to a failure in torch.compile(fullgraph=True)—a requirement for piecewise CUDA graphs—which is incompatible with FlashInfer's FP4 JIT compilation code.

The assistant embarks on a systematic debugging effort. It traces the error through multiple layers of the software stack: from the SGLang server startup, through the PyTorch Dynamo compilation pipeline, into the FlashInfer FP4 quantization code. The root cause is identified: FlashInfer's fp4_quantize function calls subprocess.run to get the CUDA version, and this subprocess call creates a graph break that violates fullgraph=True. The assistant patches get_cuda_version to avoid the subprocess call and adds @torch.compiler.disable to fp4_quantize to make it opaque to Dynamo.

But the patch is not enough. The fullgraph=True requirement means that any graph break—even a function decorated with @torch.compiler.disable—causes compilation to fail. The assistant discovers that torch.compiler.disable does not exempt a function from the fullgraph constraint; it merely prevents Dynamo from tracing into it, which itself constitutes a graph break. This is an architectural incompatibility: piecewise CUDA graphs require a single, contiguous graph, but FlashInfer's FP4 JIT code is fundamentally dynamic and cannot be captured in a static graph.

After extensive debugging, the assistant accepts the blocker and moves on. The effort is not wasted: the debugging produces deep knowledge about the interaction between PyTorch Dynamo, FlashInfer's JIT compilation system, and the fullgraph constraint. This knowledge prevents future wasted effort on the same approach and informs the assistant's understanding of the software stack's limitations.

Tier 1, Attempt 2: MSCCLPP (~2% Improvement)

With piecewise CUDA graphs blocked, the assistant pivots to MSCCLPP (Microsoft Collective Communication Library)—a custom allreduce implementation using IPC-based one-shot communication that bypasses NCCL's ring algorithm. The hypothesis is that allreduce latency is a significant bottleneck, and MSCCLPP's lower-latency communication will improve throughput.

The MSCCLPP investigation (messages 1015–1040, [chunk 8.0][58]–[83]</chunk>) is itself a mini-saga. The assistant discovers that MSCCLPP is not a separate package but is built into sgl_kernel.allreduce. It configures the SGLANG_MSCCLPP_MAX_BYTES environment variable (initially with a formatting error—writing "4MB" instead of the bytes value, requiring a fix to "4194304"). It restarts the server multiple times, runs benchmarks at all four concurrency levels, and collects the results.

The verdict is clear: MSCCLPP yields only ~2% improvement over baseline across all concurrency levels. The assistant documents this result in message 1040 ([chunk 8.0][83]</chunk>) as "negligible" and "not transformative." The allreduce is simply not the bottleneck; the MoE expert GEMMs dominate the compute profile.

Tier 1, Attempt 3: Single Batch Overlap (~2% Improvement)

The third Tier 1 optimization is Single Batch Overlap (SBO), a technique that overlaps the MoE combine operation with the down-projection GEMM to hide latency. The assistant tests SBO in combination with MSCCLPP (since both are communication-side optimizations), running the same benchmark suite.

Message 1047 ([chunk 8.0][90]</chunk>) presents the definitive comparison table:

| Concurrency | Baseline | MSCCLPP | MSCCLPP+SBO | |---|---|---|---| | 1 | 9.17 | 9.29 | 9.14 | | 10 | 38.03 | 37.13 | 38.22 | | 256 | 352.79 | 359.17 | 357.57 | | 1024 | 1520.55 | 1548.68 | 1539.19 |

The numbers tell a stark story. Across all concurrency levels, the three configurations are within 2% of each other—well within the noise floor of measurement. The assistant draws the decisive conclusion: "Communication optimizations (MSCCLPP, SBO) have negligible impact. The entire bottleneck is the MoE expert GEMMs being memory-bandwidth-bound due to small per-expert matrix sizes."

This is a textbook example of a well-executed negative result. The assistant had a hypothesis, tested it rigorously at multiple concurrency levels, and accepted the data when it refuted the hypothesis. The result is not failure; it is knowledge. The assistant now knows that communication is not the bottleneck, and the search space narrows to optimizations that increase per-expert compute efficiency.

The Pivot to Expert Parallelism

With communication-side optimizations ruled out, the assistant pivots to Expert Parallelism (EP8)—the most promising remaining approach. The reasoning is grounded in the bottleneck diagnosis: if the problem is small per-expert GEMMs, then the solution must make those GEMMs larger. EP8 does this by distributing different experts across different GPUs, so that each GPU computes its assigned experts on the full token batch rather than computing sharded experts on a fraction of tokens.

The assistant begins with code reconnaissance (messages 1048–1053, [chunk 8.0][91]–[96]</chunk>), searching the SGLang server arguments file for relevant parameters: moe_a2a_backend, ep_size, expert_parallel, and --ep. The grep output confirms that the infrastructure exists: ep_size defaults to 1 but can be set to match tp_size, and the flashinfer_cutlass MoE runner backend supports EP configurations. The assistant verifies that --moe-a2a-backend flashinfer automatically sets ep_size = tp_size (8), and that there are no obvious blockers.

The EP8 server launches successfully (messages 1054–1057, [chunk 8.0][97]–[100]</chunk>). The server logs show "TP0 EP0", "TP1 EP1", etc., confirming that each GPU has both a tensor parallelism rank and an expert parallelism rank. Per-GPU memory drops from 60.89 GB to 59.50 GB—a modest but welcome reduction—and KV cache capacity increases slightly. The assistant notes these observations with cautious optimism.

EP8 Benchmarking: Disappointment and Crash

The EP8 benchmarks (messages 1058–1059, [chunk 8.0][101]–[102]</chunk>) deliver disappointing results. At concurrency 1, EP8 achieves 118ms TPOT vs 107ms baseline—a 10% regression. At concurrency 10, EP8 achieves 125ms vs 110ms baseline—a 14% regression. The assistant's analysis is immediate and insightful: "This makes sense at low concurrency—the all-to-all overhead for EP exceeds the benefit from larger expert GEMMs when each expert processes very few tokens."

The assistant does not abandon EP based on these low-concurrency results. Instead, it formulates a conditional hypothesis: EP should improve at high concurrency, where the all-to-all overhead is amortized across more tokens and the GEMMs become more compute-efficient. It launches benchmarks at 256 and 1024 concurrent requests.

The 256-concurrency benchmark crashes. The 1024-concurrency benchmark crashes immediately with a ClientPayloadError: Response payload is not completed—the server process died mid-stream.

Diagnosing the EP8 Crash

Messages 1060–1061 ([chunk 8.0][103]–[104]</chunk>) document the post-mortem. The assistant searches the server log for error patterns, hypothesizing OOM as the likely cause. The grep command targets five categories: "Error", "error", "CUDA out of memory", "OOM", "RuntimeError", and "NCCL".

The critical finding is a warning from FlashInfer's autotuner:

[Autotuner]: Skipping tactic <...MoERunner object...> 15, due to failure while profiling:
[TensorRT-LLM][ERROR] Assertion failed: Failed to initialize cutlass TMA WS grouped gemm.
Error: Error Internal

The error references a specific generated kernel file: cutlass_kernel_file_gemm_grouped_sm120_M256_BS_group0. This is a CUTLASS grouped GEMM kernel targeting the SM120 architecture with an M256 tile size—the variant needed for larger batch sizes. The "TMA WS" refers to Tensor Memory Accelerator Warp Specialization, a Blackwell-specific feature. The error originates from TensorRT-LLM's CUTLASS integration, not from a simple CUDA memory allocation failure.

This discovery has profound implications. The M256 tile configuration—required for efficient processing of large token batches—cannot be compiled or executed on SM120. This means that EP8 at scale may be fundamentally broken on Blackwell GPUs. The server might work for small batches (where smaller tile sizes like M64 or M128 are used) but crash when batch sizes grow and the autotuner attempts the unsupported M256 variant.

The assistant now faces a critical decision: can the autotuner issue be fixed by patching FlashInfer's kernel generation code, or is EP8 fundamentally incompatible with SM120 for this model? The answer will determine the direction of the entire optimization campaign.

Synthesis: What This Chunk Teaches Us

The Value of Systematic Negative Results

The most striking feature of this chunk is the assistant's willingness to accept negative results. Piecewise CUDA Graphs was blocked by a framework incompatibility. MSCCLPP delivered ~2%. SBO delivered ~2%. EP8 regressed low-concurrency performance and crashed under load. Each of these results could have been rationalized away—"maybe if I tweak the parameter," "maybe with a different backend," "maybe with more memory." Instead, the assistant accepts each result, documents it, and moves to the next hypothesis.

This discipline is rare and valuable. In optimization work, the sunk cost fallacy is a constant temptation: the more effort invested in setting up an experiment, the harder it is to abandon it when the results are negative. The assistant resists this temptation consistently, letting data guide decisions rather than intuition or investment.

The Confirmation of the Bottleneck Diagnosis

The negative results collectively confirm the bottleneck diagnosis that had emerged from earlier profiling: the MoE expert GEMMs are memory-bandwidth-bound due to small per-expert matrix sizes. Communication optimizations cannot help because communication is not the bottleneck. Expert Parallelism cannot help (at least not stably) because the all-to-all overhead and autotuner failures introduce new problems.

This confirmation is itself a form of progress. The assistant now knows that the bottleneck is fundamental to the hardware-software combination: Blackwell's SM120 architecture has 99KB of shared memory per SM, lacks TMEM support, and cannot efficiently execute the small FP4 GEMMs that MoE inference requires. Only optimizations that fundamentally increase per-expert compute efficiency—such as L2 cache pinning, persistent kernel designs, or custom CUDA kernels—can unlock meaningful gains.

The Methodology as a Template

The chunk establishes a methodology that could serve as a template for any ML inference optimization campaign:

  1. Document all hypotheses in a structured, comparable format before testing.
  2. Establish a rigorous baseline across multiple concurrency levels.
  3. Test hypotheses in priority order, starting with the lowest-effort, highest-expected-impact options.
  4. Test at multiple concurrency levels to capture both latency-sensitive and throughput-sensitive regimes.
  5. Accept negative results and let data guide the next decision.
  6. Diagnose crashes systematically, using targeted log searches to identify root causes.
  7. Document each result so that future efforts can build on accumulated knowledge.

The Hardware Constraint as the Ultimate Limit

The chunk ultimately reveals that the optimization space is bounded by hardware constraints that no amount of software tuning can overcome. The SM120 architecture's 99KB shared memory limit prevents the use of larger CUTLASS tiles that would improve FP4 throughput. The lack of TMEM prevents the use of Blackwell's advanced tensor memory features. The broken cudaGridDependencySynchronize on SM120 prevents allreduce fusion. The autotuner failure on M256 prevents EP8 from scaling to high concurrency.

These are not bugs that can be fixed by patching configuration files. They are architectural constraints that require either different hardware (B200 Blackwell with SM100 and TMEM), different model architectures (fewer experts, larger per-expert matrices), or fundamentally different kernel designs (persistent kernels, custom CUDA code). The assistant's systematic testing has mapped the boundaries of the feasible optimization space, and the map shows that the remaining territory requires deep engineering investment.

Conclusion

This chunk of the opencode session is a masterclass in systematic ML inference optimization. The assistant writes ten documents in parallel, establishes a rigorous baseline, tests three Tier 1 optimizations (one blocked, two marginal), pivots to Expert Parallelism, launches it successfully, benchmarks it, watches it crash, and diagnoses the root cause—all within a single continuous session. The overarching narrative is one of disciplined hypothesis testing: each optimization is evaluated against data, and each is abandoned when the data fails to support the hypothesis.

The most important output of this chunk is not a performance improvement—it is knowledge. The assistant now knows that communication optimizations are not transformative, that EP8 is unstable under load due to autotuner failures on SM120, and that the core bottleneck remains the small per-expert GEMMs that are memory-bandwidth-bound on Blackwell's constrained architecture. This knowledge narrows the search space dramatically and sets the stage for the next phase of the campaign: deeper kernel-level optimizations that directly address the hardware constraints.

In the end, the chunk demonstrates that the most valuable optimization tool is not a clever kernel or a faster communication library—it is a systematic methodology for generating, testing, and accepting or rejecting hypotheses. The assistant's willingness to let data guide decisions, even when the data is disappointing, is the true engine of progress in this complex optimization journey.