Testing the Optimization Frontier: Systematic Hypothesis Evaluation on Blackwell GPUs

Introduction

In the previous segment of this opencode coding session, the AI assistant had completed a comprehensive optimization plan for the GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts language model quantized to NVFP4 precision—deployed across eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The plan was organized into three tiers, with eleven distinct optimization approaches, each documented in a file called glb5improvement-XX.md. The first document had been written, and the remaining ten existed only as notes in the assistant's context.

Segment 8 captures the transition from planning to execution. The assistant writes all remaining improvement documents, establishes a rigorous baseline, and then systematically tests the Tier 1 optimizations in priority order. What follows is a story of disciplined hypothesis testing: each optimization is evaluated against data at multiple concurrency levels, and each is either blocked by framework incompatibilities, abandoned when the data shows marginal impact, or crashes under load. The overarching narrative is one of systematic knowledge generation—the assistant learns what does not work, and in doing so, narrows the search space toward what might.

This article synthesizes the work across messages 958–1062 of the conversation, drawing on the detailed chunk article [chunk 8.0][1] and the raw conversation data to reconstruct the assistant's reasoning, methodology, and discoveries.

The Documentation Phase: Ten Files in One Round

The segment opens with the assistant completing the documentation phase. Message 963 ([msg 963]) shows the assistant dispatching ten write() calls in a single parallel round, producing files for optimization approaches 02 through 11:

Establishing the Baseline: Methodological Discipline

Before testing any optimization, the assistant runs a baseline benchmark across four concurrency levels: 1, 10, 256, and 1024 concurrent requests. This is the current best configuration: TP8 (Tensor Parallelism across 8 GPUs), CUDA graphs disabled, and chunked prefill scheduling with --num-continuous-decode-steps 16.

The baseline numbers become the reference point for all subsequent comparisons:

| Concurrency | Output tok/s | |---|---| | 1 | 9.17 | | 10 | 38.03 | | 256 | 352.79 | | 1024 | 1520.55 |

This methodological discipline—measure before you change—is a hallmark of rigorous engineering. The baseline provides the statistical foundation for declaring optimizations transformative, marginal, or ineffective. When MSCCLPP and Single Batch Overlap each deliver only ~2% improvement, the baseline gives the assistant the confidence to declare them non-transformative rather than chasing noise.

Tier 1, Attempt 1: Piecewise CUDA Graphs (Blocked)

The first Tier 1 optimization to be tested is Piecewise CUDA Graphs—a technique that uses CUDA graph capture for parts of the computation while allowing dynamic tensor shapes for others. The expected impact is 10–20% throughput improvement, making it the most promising of the quick-to-test options.

The assistant launches the server with --enable-cuda-graph --enable-piecewise-cuda-graphs in message 975 ([msg 975]), only to encounter a crash during model loading. The error trace points to a failure in torch.compile(fullgraph=True)—a requirement for piecewise CUDA graphs—which is incompatible with FlashInfer's FP4 JIT compilation code.

What follows is an extended debugging saga spanning messages 975 through 1014 ([msg 975] to [msg 1014]). The assistant traces the error through multiple layers of the software stack: from the SGLang server startup, through the PyTorch Dynamo compilation pipeline, into the FlashInfer FP4 quantization code. The root cause is identified: FlashInfer's fp4_quantize function calls subprocess.run to get the CUDA version, and this subprocess call creates a graph break that violates fullgraph=True.

The assistant attempts a patch: modifying get_cuda_version to avoid the subprocess call and adding @torch.compiler.disable to fp4_quantize to make it opaque to Dynamo. But the patch is insufficient. The fullgraph=True requirement means that any graph break—even a function decorated with @torch.compiler.disable—causes compilation to fail. The assistant discovers that torch.compiler.disable does not exempt a function from the fullgraph constraint; it merely prevents Dynamo from tracing into it, which itself constitutes a graph break.

This is an architectural incompatibility. Piecewise CUDA graphs require a single, contiguous graph, but FlashInfer's FP4 JIT code is fundamentally dynamic and cannot be captured in a static graph. The assistant documents the blocker in message 1014 ([msg 1014]) and moves on.

The debugging effort is not wasted. It produces deep knowledge about the interaction between PyTorch Dynamo, FlashInfer's JIT compilation system, and the fullgraph constraint. This knowledge prevents future wasted effort on the same approach and informs the assistant's understanding of the software stack's limitations.

Tier 1, Attempt 2: MSCCLPP (~2% Improvement)

With piecewise CUDA graphs blocked, the assistant pivots to MSCCLPP—Microsoft Collective Communication Library, a custom allreduce implementation using IPC-based one-shot communication that bypasses NCCL's ring algorithm. The hypothesis is that allreduce latency is a significant bottleneck, and MSCCLPP's lower-latency communication will improve throughput.

The MSCCLPP investigation (messages 1015–1040, [msg 1015] to [msg 1040]) is itself a mini-saga. The assistant first discovers that MSCCLPP is not a separate package but is built into sgl_kernel.allreduce. It configures the SGLANG_MSCCLPP_MAX_BYTES environment variable—initially with a formatting error (writing "4MB" instead of the bytes value, requiring a fix to "4194304"). It restarts the server multiple times, runs benchmarks at all four concurrency levels, and collects the results.

The verdict is delivered in message 1040 ([msg 1040]):

| Concurrency | Baseline | MSCCLPP | Delta | |---|---|---|---| | 1 | 9.17 tok/s | 9.29 tok/s | +1.3% | | 10 | 38.03 tok/s | 37.13 tok/s | -2.4% | | 256 | 352.79 tok/s | 359.17 tok/s | +1.8% | | 1024 | 1520.55 tok/s | 1548.68 tok/s | +1.8% |

The assistant's conclusion is definitive: "MSCCLPP result: ~2% improvement across the board, negligible. The allreduce is not the bottleneck — MoE expert GEMMs dominate." This is a textbook example of a well-executed negative result. The assistant had a hypothesis, tested it rigorously at multiple concurrency levels, and accepted the data when it refuted the hypothesis.

Tier 1, Attempt 3: Single Batch Overlap (~2% Improvement)

The third Tier 1 optimization is Single Batch Overlap (SBO), a technique that overlaps the MoE combine operation with the down-projection GEMM to hide latency. The assistant tests SBO in combination with MSCCLPP (since both are communication-side optimizations), running the same benchmark suite.

Message 1047 ([msg 1047]) presents the definitive comparison table:

| Concurrency | Baseline | MSCCLPP | MSCCLPP+SBO | |---|---|---|---| | 1 | 9.17 | 9.29 | 9.14 | | 10 | 38.03 | 37.13 | 38.22 | | 256 | 352.79 | 359.17 | 357.57 | | 1024 | 1520.55 | 1548.68 | 1539.19 |

The numbers tell a stark story. Across all concurrency levels, the three configurations are within 2% of each other—well within the noise floor of measurement. The assistant draws the decisive conclusion: "Communication optimizations (MSCCLPP, SBO) have negligible impact. The entire bottleneck is the MoE expert GEMMs being memory-bandwidth-bound due to small per-expert matrix sizes."

This conclusion is a watershed moment. The assistant has now ruled out all communication-side optimizations as transformative. The bottleneck is confirmed to be the small per-expert GEMMs—each expert in the GLM-5 model is relatively small, and the FP4 precision means the matrices are memory-bandwidth-bound rather than compute-bound. No amount of communication optimization can fix this.

The Pivot to Expert Parallelism

With communication-side optimizations ruled out, the assistant pivots to Expert Parallelism (EP8)—the most promising remaining approach. The reasoning is grounded in the bottleneck diagnosis: if the problem is small per-expert GEMMs, then the solution must make those GEMMs larger. EP8 does this by distributing different experts across different GPUs, so that each GPU computes its assigned experts on the full token batch rather than computing sharded experts on a fraction of tokens.

The assistant begins with code reconnaissance (messages 1048–1053, [msg 1048] to [msg 1053]), searching the SGLang server arguments file for relevant parameters: moe_a2a_backend, ep_size, expert_parallel, and --ep. The grep output confirms that the infrastructure exists: ep_size defaults to 1 but can be set to match tp_size, and the flashinfer_cutlass MoE runner backend supports EP configurations. The assistant verifies that --moe-a2a-backend flashinfer automatically sets ep_size = tp_size (8), and that there are no obvious blockers.

The EP8 server launches successfully in message 1054 ([msg 1054]). The server logs show "TP0 EP0", "TP1 EP1", etc., confirming that each GPU has both a tensor parallelism rank and an expert parallelism rank. Per-GPU memory drops from 60.89 GB to 59.50 GB—a modest but welcome reduction—and KV cache capacity increases from 490,880 to 517,632 tokens. The assistant notes these observations with cautious optimism.

EP8 Benchmarking: Disappointment and Crash

The EP8 benchmarks (messages 1058–1059, [msg 1058] to [msg 1059]) deliver disappointing results. At concurrency 1, EP8 achieves 118ms TPOT vs 107ms baseline—a 10% regression. At concurrency 10, EP8 achieves 125ms vs 110ms baseline—a 14% regression.

The assistant's analysis is immediate and insightful: "This makes sense at low concurrency—the all-to-all overhead for EP exceeds the benefit from larger expert GEMMs when each expert processes very few tokens." The all-to-all communication required for EP adds latency that, at low concurrency, outweighs the computational benefit of larger per-expert matrices.

The assistant does not abandon EP based on these low-concurrency results. Instead, it formulates a conditional hypothesis: EP should improve at high concurrency, where the all-to-all overhead is amortized across more tokens and the GEMMs become more compute-efficient. It launches benchmarks at 256 and 1024 concurrent requests.

The 256-concurrency benchmark crashes. The 1024-concurrency benchmark crashes immediately with a ClientPayloadError: Response payload is not completed—the server process died mid-stream.

Diagnosing the EP8 Crash

Messages 1060–1061 ([msg 1060] to [msg 1061]) document the post-mortem. The assistant searches the server log for error patterns, hypothesizing OOM as the likely cause. The grep command targets five categories: "Error", "error", "CUDA out of memory", "OOM", "RuntimeError", and "NCCL".

The critical finding is a warning from FlashInfer's autotuner:

[Autotuner]: Skipping tactic <...MoERunner object...> 15, due to failure while profiling:
[TensorRT-LLM][ERROR] Assertion failed: Failed to initialize cutlass TMA WS grouped gemm.
Error: Error Internal

The error references a specific generated kernel file: cutlass_kernel_file_gemm_grouped_sm120_M256_BS_group0. This is a CUTLASS grouped GEMM kernel targeting the SM120 architecture with an M256 tile size—the variant needed for larger batch sizes. The "TMA WS" refers to Tensor Memory Accelerator Warp Specialization, a Blackwell-specific feature. The error originates from TensorRT-LLM's CUTLASS integration, not from a simple CUDA memory allocation failure.

This discovery has profound implications. The M256 tile configuration—required for efficient processing of large token batches—cannot be compiled or executed on SM120. This means that EP8 at scale may be fundamentally broken on Blackwell GPUs. The server might work for small batches (where smaller tile sizes like M64 or M128 are used) but crash when batch sizes grow and the autotuner attempts the unsupported M256 variant.

The assistant now faces a critical decision: can the autotuner issue be fixed by patching FlashInfer's kernel generation code, or is EP8 fundamentally incompatible with SM120 for this model? The answer will determine the direction of the entire optimization campaign.

Synthesis: What This Segment Teaches Us

The Value of Systematic Negative Results

The most striking feature of this segment is the assistant's willingness to accept negative results. Piecewise CUDA Graphs was blocked by a framework incompatibility. MSCCLPP delivered ~2%. SBO delivered ~2%. EP8 regressed low-concurrency performance and crashed under load. Each of these results could have been rationalized away—"maybe if I tweak the parameter," "maybe with a different backend," "maybe with more memory." Instead, the assistant accepts each result, documents it, and moves to the next hypothesis.

This discipline is rare and valuable. In optimization work, the sunk cost fallacy is a constant temptation: the more effort invested in setting up an experiment, the harder it is to abandon it when the results are negative. The assistant resists this temptation consistently, letting data guide decisions rather than intuition or investment.

The Confirmation of the Bottleneck Diagnosis

The negative results collectively confirm the bottleneck diagnosis that had emerged from earlier profiling: the MoE expert GEMMs are memory-bandwidth-bound due to small per-expert matrix sizes. Communication optimizations cannot help because communication is not the bottleneck. Expert Parallelism cannot help (at least not stably) because the all-to-all overhead and autotuner failures introduce new problems.

This confirmation is itself a form of progress. The assistant now knows that the bottleneck is fundamental to the hardware-software combination: Blackwell's SM120 architecture has 99KB of shared memory per SM, lacks TMEM support, and cannot efficiently execute the small FP4 GEMMs that MoE inference requires. Only optimizations that fundamentally increase per-expert compute efficiency—such as L2 cache pinning, persistent kernel designs, or custom CUDA kernels—can unlock meaningful gains.

The Methodology as a Template

The segment establishes a methodology that could serve as a template for any ML inference optimization campaign:

  1. Document all hypotheses in a structured, comparable format before testing.
  2. Establish a rigorous baseline across multiple concurrency levels.
  3. Test hypotheses in priority order, starting with the lowest-effort, highest-expected-impact options.
  4. Test at multiple concurrency levels to capture both latency-sensitive and throughput-sensitive regimes.
  5. Accept negative results and let data guide the next decision.
  6. Diagnose crashes systematically, using targeted log searches to identify root causes.
  7. Document each result so that future efforts can build on accumulated knowledge.

The Hardware Constraint as the Ultimate Limit

The segment ultimately reveals that the optimization space is bounded by hardware constraints that no amount of software tuning can overcome. The SM120 architecture's 99KB shared memory limit prevents the use of larger CUTLASS tiles that would improve FP4 throughput. The lack of TMEM prevents the use of Blackwell's advanced tensor memory features. The broken cudaGridDependencySynchronize on SM120 prevents allreduce fusion. The autotuner failure on M256 prevents EP8 from scaling to high concurrency.

These are not bugs that can be fixed by patching configuration files. They are architectural constraints that require either different hardware (B200 Blackwell with SM100 and TMEM), different model architectures (fewer experts, larger per-expert matrices), or fundamentally different kernel designs (persistent kernels, custom CUDA code). The assistant's systematic testing has mapped the boundaries of the feasible optimization space, and the map shows that the remaining territory requires deep engineering investment.

Conclusion

Segment 8 of this opencode session is a masterclass in systematic ML inference optimization. The assistant writes ten documents in parallel, establishes a rigorous baseline, tests three Tier 1 optimizations (one blocked, two marginal), pivots to Expert Parallelism, launches it successfully, benchmarks it, watches it crash, and diagnoses the root cause—all within a single continuous session. The overarching narrative is one of disciplined hypothesis testing: each optimization is evaluated against data, and each is abandoned when the data fails to support the hypothesis.

The most important output of this segment is not a performance improvement—it is knowledge. The assistant now knows that communication optimizations are not transformative, that EP8 is unstable under load due to autotuner failures on SM120, and that the core bottleneck remains the small per-expert GEMMs that are memory-bandwidth-bound on Blackwell's constrained architecture. This knowledge narrows the search space dramatically and sets the stage for the next phase of the campaign: deeper kernel-level optimizations that directly address the hardware constraints.

In the end, the segment demonstrates that the most valuable optimization tool is not a clever kernel or a faster communication library—it is a systematic methodology for generating, testing, and accepting or rejecting hypotheses. The assistant's willingness to let data guide decisions, even when the data is disappointing, is the true engine of progress in this complex optimization journey.

References