The Shared Memory Ceiling: Diagnosing CUTLASS Tile Failures on Blackwell SM120

In the middle of a deep optimization session for deploying the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant produces a message that serves as both a technical synthesis and a pragmatic pivot. Message [msg 898] is deceptively short — it contains a precise diagnosis of why certain CUTLASS FP4 GEMM tile configurations fail to initialize on the SM120 architecture, followed by a routine server health check. But this single message crystallizes hours of investigation into GPU kernel efficiency, shared memory constraints, and the architectural differences between Blackwell's compute capabilities and its predecessors. It is the moment where deep kernel analysis gives way to practical benchmarking, and where the assistant demonstrates both its technical depth and its ability to prioritize forward progress over perfect understanding.

The Message in Full

The assistant writes:

I see StageCountAutoCarveout combined with KernelScheduleAuto for SM120. The auto carveout means CUTLASS auto-computes how many pipeline stages fit in the available SMEM after subtracting the epilogue storage. With 100KB SMEM and larger tiles like 128×256, the auto-calculation probably determines it needs more than 100KB for even a single stage, causing the init failure.

>

Now let me check the server: [bash] sleep 30 && ssh root@10.1.230.174 'tail -5 /root/sglang-server.log; curl -s http://localhost:8000/health 2>/dev/null || echo "NOT_READY"' [2026-02-19 14:32:46] INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) [2026-02-19 14:32:47] INFO: 127.0.0.1:53272 - "GET /model_info HTTP/1.1" 200 OK [2026-02-19 14:32:48 TP0] Prefill batch, #new-seq: 1, #new-token: 64, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, input throughput (token/s): 0.00, cuda graph: False [2026-02-19 14:32:49] INFO: 127.0.0.1:53288 - "POST /generate HTTP/1.1" 200 OK [2026-02-19 14:32:49] The server is fired up and...

The message is split into two distinct halves: a kernel-level root cause analysis, and an operational status check. Understanding why both halves belong in the same message requires tracing the investigation that led here.

The Investigation That Preceded This Moment

In the messages leading up to [msg 898], the assistant had been systematically dissecting why the FP4 GEMM kernels on SM120 were underperforming. The GLM-5-NVFP4 model uses a Mixture-of-Experts architecture with 256 experts, of which 8 are activated per token. During the decode phase of inference, each expert sees only a small batch of tokens — approximately 16 tokens per expert at 512 concurrent requests, or 32 at 1024 concurrency. These tiny GEMMs (dimensions like 16×2048×6144) have arithmetic intensity of roughly 16 ops/byte, which is 72 times below the ~1,158 ops/byte needed to saturate the GPU's compute units. The result is catastrophic underutilization: the GPUs achieve only 0.04% of their dense FP4 peak during decode.

The assistant had also discovered that the CUTLASS autotuner for SM120 only successfully compiled four tile configurations: all variations of 128×128 tiles (with different K dimensions and cluster configurations). Two promising larger tiles — M128×N256 and M256×N128 — failed during initialization with the error "Failed to initialize cutlass TMA WS grouped gemm." These larger tiles would have allowed better work distribution across streaming multiprocessors (SMs) and higher compute density, potentially improving throughput by 30-50%.

By examining the generated CUDA kernel files in /root/.cache/flashinfer/0.6.3/120a/generated/cutlass_instantiations/, the assistant identified that the failing tiles used the FINALIZE epilogue mode rather than NONE, and that the launcher template at moe_gemm_tma_ws_launcher.inl was the key to understanding why.

The Root Cause: Shared Memory Arithmetic

Message [msg 898] opens with the assistant's synthesis of this investigation. The key insight is the interaction between three CUTLASS template parameters: StageCountAutoCarveout, KernelScheduleAuto, and the available shared memory on SM120.

On NVIDIA's SM100 architecture (used in the Blackwell B200), each SM has 228 KB of shared memory, which provides ample room for the pipeline stages, epilogue storage, and TMA descriptors required by warp-specialized GEMM kernels. SM120 (used in the RTX PRO 6000 Blackwell), by contrast, has only 99 KB of shared memory per SM — a reduction of more than half. This is a critical architectural difference that directly impacts which tile configurations can fit.

StageCountAutoCarveout is a CUTLASS mechanism that automatically calculates how many pipeline stages can fit in the available shared memory after subtracting the storage required by the epilogue (the output stage of the GEMM). The "carveout" refers to the shared memory reserved for the epilogue before the stage count is computed. For a tile configuration like M128×N256×K128, the combined storage for the mainloop pipeline stages, the epilogue shared storage, and the TMA descriptors may exceed the 99 KB budget. When CUTLASS's auto-calculation determines that even a single pipeline stage cannot fit, the kernel fails to initialize — which is exactly the error the autotuner reported.

The assistant's analysis correctly identifies that the KernelScheduleAuto parameter compounds the problem. On SM100, CUTLASS uses explicit 1SM or 2SM warp-specialized schedules that are hand-optimized for that architecture's shared memory capacity. On SM120, the auto scheduler must make these decisions at compile time without the same level of architectural tuning, and it likely selects configurations that demand more shared memory than is available.

What the Assistant Assumed — and What It Got Right

The assistant makes one minor imprecision: it refers to "100KB SMEM" when the actual shared memory limit on SM120 is 99 KB. This is a negligible rounding error that does not affect the analysis. The more significant assumption is that the auto-carveout calculation is the sole cause of the initialization failure. While this is the most plausible explanation given the evidence, the assistant did not experimentally verify it — for example, by manually computing the shared memory requirements for each tile configuration, or by patching the launcher to force a different stage count. Instead, the assistant treats this as a sufficient explanation and moves on.

This decision reflects a pragmatic tradeoff. The assistant could spend additional cycles definitively proving the root cause, but the practical implication is already clear: larger tiles don't fit in SM120's shared memory, and no amount of configuration tweaking within the existing codebase will change that. The path to improved performance lies elsewhere — in higher concurrency, better kernel schedules, or alternative GEMM implementations. The assistant's willingness to accept a plausible hypothesis and pivot to benchmarking is a hallmark of effective optimization work: know when to dig deeper and when to move forward.

The Pivot: From Kernel Analysis to Server Benchmarking

The second half of the message is a complete change of focus. After 30 seconds of sleep (to allow the server to finish loading), the assistant checks the TP8 SGLang server that was launched in [msg 890]. The log output confirms the server is running: Uvicorn is listening on port 8000, the model info endpoint responds successfully, and a test generate request has already been processed. The server is "fired up and..." — the message cuts off, but the implication is clear: benchmarking can begin.

This transition is significant because it marks the shift from investigation to measurement. The assistant had spent the preceding messages (roughly [msg 876] through [msg 897]) analyzing kernel code, inspecting generated CUDA files, reading launcher templates, and building a mental model of the SM120 architecture's limitations. With the root cause understood and the server confirmed operational, the assistant can now run the actual throughput benchmarks that will quantify the real-world impact of these architectural constraints.

The Broader Context: Why This Matters

Message [msg 898] sits at a critical juncture in the conversation. The overall session (Segment 7) is focused on understanding why the GLM-5-NVFP4 model underperforms on Blackwell GPUs and finding optimization paths. The assistant had already established that the model is compute-bound (not communication-bound) by comparing TP4+PP2 against TP8 configurations. The deep dive into CUTLASS kernel efficiency revealed that the GPUs draw only ~235W out of 600W TDP during inference, and that the CUTLASS kernels plateau at ~1,300 TFLOPS (70% of dense peak) only for very large matrices — while actual decode workloads achieve merely 0.8–55 TFLOPS (0.02–3% of peak).

The shared memory limitation identified in this message is therefore not just an interesting technical detail; it is a fundamental constraint on the optimization space. It means that the largest available CUTLASS tile configurations cannot be used, which limits the compute density of the grouped GEMM kernels that power the MoE expert computations. The assistant's subsequent work — tuning --max-running-requests to 2048 and --num-continuous-decode-steps to 8 — achieved a 28% throughput improvement, but the shared memory ceiling remains a hard limit that no amount of parameter tuning can bypass without changes to the kernel implementations themselves.

Input Knowledge and Output Knowledge

To fully understand this message, a reader needs substantial background in GPU computing: the role of shared memory in CUDA kernels, the concept of pipeline stages in GEMM implementations, how TMA (Tensor Memory Accelerator) warp specialization works on NVIDIA architectures, and the CUTLASS template library's metaprogramming approach to kernel generation. Knowledge of the specific model architecture (GLM-5-NVFP4's 256-expert MoE, FP4 quantization) and the SGLang serving framework is also necessary to appreciate why the tile size matters for this particular workload.

The output knowledge created by this message is twofold. First, it establishes a specific, testable hypothesis for why the larger tile configurations fail on SM120: the auto-carveout calculation determines that the shared memory requirement exceeds the available 99 KB budget. Second, it confirms that the TP8 server is operational and ready for benchmarking, enabling the next phase of the investigation. The message thus serves as both a conclusion to the kernel analysis thread and a launch point for the performance measurement thread.

Conclusion

Message [msg 898] exemplifies the kind of thinking that distinguishes effective system optimization from aimless tinkering. The assistant identifies a root cause with sufficient confidence to act, without over-investing in perfect proof. It recognizes when analysis has reached diminishing returns and when it is time to gather empirical data. And it communicates its findings with precision — the mention of StageCountAutoCarveout and KernelScheduleAuto is not jargon but a specific, falsifiable claim about the mechanism of failure. In the broader narrative of the conversation, this message marks the moment when the assistant stops asking "why is this broken?" and starts asking "how well can we do with what works?" — a shift from diagnosis to optimization that ultimately yields a 28% throughput improvement in the sessions that follow.