The CuteDSL Experiment: When JIT-Generated Kernels Fall Short at Scale

Introduction

In the high-stakes world of large language model inference optimization, every microsecond counts. When deploying a massive 8-GPU setup running the GLM-5-NVFP4 model — a 256-expert Mixture-of-Experts architecture quantized to NVIDIA's FP4 format — the choice of which kernel backend drives the MoE (Mixture-of-Experts) computation can make or break throughput. Message [msg 1253] captures a pivotal moment in this optimization journey: the moment a promising new backend, flashinfer_cutedsl, is put through its paces and found to have a surprising, non-monotonic performance profile that defies simple expectations.

This article examines that single message in depth, unpacking the reasoning, assumptions, and discoveries embedded within its concise text and tool calls.

The Scene: A High-Stakes Optimization Hunt

By the time we reach [msg 1253], the assistant and user have been engaged in an intensive, multi-session effort to maximize inference throughput for the GLM-5-NVFP4 model on a machine equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs (compute capability SM120). The baseline configuration using flashinfer_cutlass as the MoE runner backend had achieved respectable throughput: approximately 1,603 tok/s at 1024 concurrency, with a single-stream TPOT (time per output token) around 95ms. But the team was far from satisfied — theoretical analysis suggested the potential for far higher performance.

The flashinfer_cutedsl backend represented a fundamentally different approach. Instead of using pre-optimized, hand-tuned CUTLASS kernels, CuteDSL uses JIT (Just-In-Time) compilation to generate custom CUDA kernels tailored to the specific GEMM (General Matrix Multiply) shapes encountered at runtime. The promise was compelling: by generating kernels optimized for the exact dimensions of the problem — including the number of active experts, batch size, and tile shapes — CuteDSL could potentially outperform the more generic CUTLASS kernels. This is especially relevant for MoE architectures where the GEMM shapes vary dramatically with batch size and routing patterns.

The Subject Message: Analysis, Hypothesis, and Verification

The message at [msg 1253] is the assistant's response after receiving the results of a comprehensive benchmark suite comparing CuteDSL against the CUTLASS baseline. Let us examine it in full:

Same numbers — 515 tok/s, not a JIT issue. CuteDSL is genuinely slower at high concurrency. This is an interesting finding: the CUTLASS backend likely has better work distribution for grouped GEMMs with many experts (256) at large batch sizes, while CuteDSL may be generating suboptimal kernels for the specific GEMM shapes.

>

Wait, I need to double-check — the concurrency 10 result of 64.29 tok/s is suspicious. Baseline was 38.46. Let me re-verify at concurrency 10:

>

[bash] ssh root@10.1.230.174 'source /root/ml-env/bin/activate && echo "=== Re-running Concurrency 10 ===" && python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8000 --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --dataset-name random --random-input-len 512 --random-output-len 128 --num-prompts 40 --request-rate 100000 --max-concurrency 10 2>&1 | grep -E "Output token throughput|Peak output|TPOT|Mean ITL|Concurrency:"'

>

=== Re-running Concurrency 10 === Output token throughput (tok/s): 64.64 Peak output token throughput (tok/s): 90.00 Concurrency: 8.47 Mean TPOT (ms): 127.88 Median TPOT (ms): 129.63 P99 TPOT (ms): 157.25 Mean ITL (ms): 125.38

This message is deceptively brief, but it contains a rich tapestry of reasoning, hypothesis formation, and experimental verification. Let us unpack each layer.

The Reasoning Process: From Data to Insight

The message opens with a definitive conclusion: "Same numbers — 515 tok/s, not a JIT issue." This refers to the re-run of the concurrency-256 benchmark from the previous message ([msg 1252]). The assistant had initially wondered whether the poor performance at high concurrency was an artifact of JIT compilation warmup — perhaps the CuteDSL kernels for large batch sizes hadn't been compiled yet during the first run, and the benchmark was measuring compilation overhead rather than pure inference. The re-run disproves this hypothesis: the throughput is identical (515.77 vs 515.58 tok/s), confirming that the performance is stable and reproducible, not a warmup artifact.

This is a critical methodological moment. The assistant demonstrates scientific rigor by distinguishing between a genuine performance characteristic and a measurement artifact. Many optimization efforts have been derailed by chasing "improvements" that were actually just warmup effects or caching noise. By explicitly testing this hypothesis, the assistant saves valuable time and establishes a reliable baseline for further analysis.

Having ruled out JIT warmup, the assistant then formulates a new hypothesis: "the CUTLASS backend likely has better work distribution for grouped GEMMs with many experts (256) at large batch sizes, while CuteDSL may be generating suboptimal kernels for the specific GEMM shapes." This is a nuanced insight. The GLM-5-NVFP4 model has 256 experts, and at high concurrency, the batch size per expert grows significantly. CuteDSL's JIT compilation generates kernels based on the observed GEMM dimensions, but the generated kernels may not be as efficient as the hand-tuned CUTLASS templates for these specific shapes. The CUTLASS backend, by contrast, has been extensively optimized for a wide range of tile sizes and may dispatch work more efficiently across the GPU's compute units.

The Suspicious Data Point and the Decision to Re-Verify

The assistant then notices something anomalous: the concurrency-10 result shows CuteDSL achieving 64.29 tok/s versus a baseline of 38.46 tok/s — a 67% improvement. This is suspiciously good. In the context of the overall trend (where CuteDSL underperforms at high concurrency), a dramatic win at low concurrency seems inconsistent. The assistant's instinct is to verify: "Wait, I need to double-check — the concurrency 10 result of 64.29 tok/s is suspicious."

This moment reveals a key aspect of the assistant's thinking: pattern recognition. The performance curve across concurrency levels should be relatively smooth and monotonic. A sudden 67% improvement at one specific concurrency level, surrounded by degradation at neighboring levels, is a red flag. It could indicate:

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

MoE Architecture: The GLM-5-NVFP4 model uses a Mixture-of-Experts design with 256 experts. During inference, each token is routed to a subset of experts (typically 2-8), and the expert computation dominates the forward pass. The GEMM operations within each expert are the primary bottleneck.

FP4 Quantization: The model uses NVIDIA's FP4 format (NVFP4), which packs two 4-bit values into each byte. This requires specialized kernels that can operate on packed 4-bit data, adding complexity to the GEMM implementation.

CUTLASS vs CuteDSL: CUTLASS is NVIDIA's collection of hand-tuned CUDA templates for matrix operations, optimized for specific tile sizes and GPU architectures. CuteDSL (part of the FlashInfer library) is a JIT-compilation approach that generates custom kernels at runtime based on the exact problem dimensions. The trade-off is between pre-optimized generality and shape-specific specialization.

SM120 Architecture: The Blackwell GPU (compute capability 12.0) has specific constraints on tile sizes, cluster shapes, and instruction throughput. Both CUTLASS and CuteDSL must generate kernels compatible with these constraints.

Benchmarking Methodology: Understanding the sglang.bench_serving tool, its parameters (request-rate, max-concurrency, num-prompts), and how to interpret throughput metrics (tok/s, TPOT, ITL) is essential.

Assumptions Made

The message rests on several assumptions, some explicit and some implicit:

The baseline comparison is valid: The assistant assumes that the baseline benchmarks (from earlier runs with flashinfer_cutlass) were conducted under identical conditions — same model, same GPU configuration, same input lengths, same server parameters. Any difference in server configuration (e.g., different max-running-requests or num-continuous-decode-steps) could confound the comparison.

JIT compilation is deterministic: The assistant assumes that re-running the benchmark produces the same kernels (because the GEMM shapes are the same). This is reasonable for a deterministic JIT compiler, but if CuteDSL uses heuristics or random sampling in kernel generation, results could vary between runs.

The benchmark is representative: The random-input-len 512 / random-output-len 128 workload may not capture all relevant operating points. Real-world usage might have different input/output length distributions, which could favor one backend over another.

GPU state is consistent: The assistant assumes that the GPU is in a similar state across benchmark runs — no thermal throttling, no memory fragmentation, no other processes competing for compute resources.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

CuteDSL's non-monotonic performance profile: The discovery that CuteDSL outperforms CUTLASS at low concurrency (10-64) but underperforms at high concurrency (256-1024) is a non-obvious finding. It suggests that the JIT-generated kernels are competitive for small batch sizes but fail to scale.

The JIT warmup hypothesis is ruled out: By re-running the concurrency-256 benchmark and getting identical results, the assistant eliminates JIT compilation overhead as an explanation for the poor high-concurrency performance. The issue is in the kernels themselves, not in their compilation.

A validated data point at concurrency 10: The re-run confirms that CuteDSL's 67% improvement at concurrency 10 is real, strengthening the evidence for the non-monotonic profile.

A refined hypothesis about GEMM work distribution: The assistant's hypothesis — that CUTLASS has better work distribution for grouped GEMMs with many experts at large batch sizes — provides a direction for further investigation. If correct, it points toward either improving CuteDSL's kernel generation for large batch sizes or accepting that CUTLASS is the better backend for this specific workload.

The Significance of the Finding

The non-monotonic performance profile of CuteDSL is significant for several reasons. First, it challenges the assumption that JIT-compiled kernels are universally superior to hand-tuned templates. In the MoE context, where the GEMM shapes vary dramatically with batch size, the optimal backend may depend on the operating point. A production system might even benefit from switching backends dynamically based on current load.

Second, it highlights the importance of comprehensive benchmarking across the full operating range. A naive evaluation that only tested low concurrency would have concluded that CuteDSL is a clear winner (67% improvement). A test that only tested high concurrency would have concluded it's a clear loser (50% degradation). Only by testing the full spectrum does the true, nuanced picture emerge.

Third, it demonstrates the value of scientific skepticism in performance analysis. The assistant's instinct to question the suspicious concurrency-10 result — rather than simply accepting it — prevented a potentially misleading conclusion. The re-run confirmed the result, but the act of verification strengthened the overall analysis.

Conclusion

Message [msg 1253] is a masterclass in performance analysis methodology. It demonstrates how to distinguish genuine performance characteristics from measurement artifacts, how to formulate and test hypotheses, and how to approach anomalous data points with healthy skepticism. The discovery that CuteDSL has a non-monotonic performance profile — better at low concurrency, worse at high concurrency — is a valuable contribution to the optimization effort, steering future work away from a dead end and toward more productive avenues.

In the broader context of the GLM-5-NVFP4 deployment, this message represents a turning point. The CuteDSL experiment, while ultimately not the solution for high-throughput scenarios, provided crucial insights into the behavior of JIT-compiled kernels on SM120 hardware. The path forward would focus on other optimization strategies — expert parallelism, CUDA graphs, and kernel fusion — armed with a deeper understanding of the trade-offs between kernel generation approaches.