The CuteDSL Benchmark: When Expectations Meet Reality in FP4 Inference

Introduction

In the high-stakes world of large language model inference optimization, few moments are as revealing as the head-to-head benchmark of two competing kernel backends. Message 1251 of this opencode session captures precisely such a moment: the assistant has just deployed a server running the GLM-5-NVFP4 model with the flashinfer_cutedsl MoE runner backend — a JIT-compiled kernel generator that promises custom-tuned matrix multiplications — and is now systematically benchmarking it against the established flashinfer_cutlass baseline. The message is a snapshot of scientific inquiry in progress: a hypothesis being tested, data being gathered, and the first hints of a surprising outcome emerging from the numbers.

This article examines message 1251 in depth: its context, its reasoning, its assumptions, and the knowledge it both consumes and produces. The message sits at a critical inflection point in a multi-day optimization effort spanning segments 5 through 10 of the conversation, where the team has been chasing ever-higher inference throughput on a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs.

The Message in Full

The assistant writes:

Interesting — cutedsl at concurrency 1: 10.07 tok/s, TPOT 97.64ms. The baseline was 10.36 tok/s, TPOT 95.14ms. So cutedsl is about 2.5% slower at batch=1.

>

Now let me test at higher concurrency levels where cutedsl should shine. Let me run the full suite:

>

[bash command to run benchmarks at concurrency levels 10, 64, 256, 1024]

>

=== Concurrency 10 === Output token throughput (tok/s): 64.29 Peak output token throughput (tok/s): 90.00 Concurrency: 8.48 Mean TPOT (ms): 128.95 Median TPOT (ms): 130.14 P99 TPOT (ms): 163.57 Mean ITL (ms): 125.80 --- === Concurrency 64 === Output token throughput (tok/s): 249.23 Peak output token throughput (tok/...

The message is truncated at concurrency 64, with the remaining results spilling into the subsequent message (msg 1252). But even in this partial form, the message is dense with meaning.

Context: The Long Road to CuteDSL

To understand why this message matters, we must trace the path that led here. The conversation had been engaged in a multi-week optimization campaign for the GLM-5-NVFP4 model, a 78-layer Mixture-of-Experts transformer quantized to NVFP4 (NVIDIA's 4-bit floating point format). The model is deployed on eight RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism (TP8).

Throughout segments 5 through 9, the team had explored a dizzying array of optimization techniques: FlashInfer CUTLASS MoE autotuning, expert parallelism (EP8), piecewise CUDA graphs, MSCCLPP allreduce, Opportunistic Expert Activation (OEA), and numerous server parameter tunings. Each approach yielded incremental gains, but the core bottleneck remained stubbornly resistant: the FP4 GEMM (General Matrix Multiply) kernels running on SM120 (Blackwell's compute architecture) were far from theoretical peak efficiency.

By segment 10, the team had computed the theoretical maximum single-stream performance at 309 tok/s, while actual performance languished at 10.36 tok/s — a staggering 30x gap representing just 3.4% efficiency. A comprehensive system audit had uncovered and fixed numerous misconfigurations (suboptimal CPU governor, outdated kernel, deep C-states, PCIe MaxReadReq stuck at 512 bytes), but the fundamental GEMM bottleneck remained.

Enter flashinfer_cutedsl. This backend uses NVIDIA's CuteDSL (a domain-specific language for CUDA kernel generation) to JIT-compile custom grouped GEMM kernels specifically tailored to the exact problem dimensions at runtime. Unlike the pre-tuned CUTLASS kernels which are compiled ahead of time for common tile sizes, CuteDSL can theoretically generate optimal kernels for any shape — including the unusual M=1, N=256, K=6144 dimensions that dominate single-stream decode in this model.

The Reasoning Behind the Message

The assistant's reasoning in message 1251 reveals a clear mental model of how these backends work. The key insight is stated explicitly: "Now let me test at higher concurrency levels where cutedsl should shine." This hypothesis is grounded in the architecture of the two backends:

  1. CUTLASS kernels are pre-compiled for specific tile configurations (e.g., 16×8, 32×8, 64×16). At batch size 1 (M=1), the GEMM operation is a thin matrix multiply where one dimension is tiny. Pre-compiled CUTLASS kernels may not have optimal tile shapes for this regime, but they have very low launch overhead since they're already loaded.
  2. CuteDSL kernels are JIT-compiled at runtime. The first time a particular problem shape is encountered, the system generates and compiles a custom kernel. This has a one-time compilation cost, but the resulting kernel should be exactly tuned to the problem dimensions. At higher concurrency, the effective batch size (M) grows as multiple requests are batched together, making the GEMMs "fatter" and giving CuteDSL more opportunity to generate efficient tile configurations. The assistant's earlier exploration (messages 1233-1241) had confirmed that CuteDSL was available in the installed FlashInfer 0.6.3, that it supported FP4 block-scaled GEMMs via grouped_gemm_nt_masked, and that SM120 (compute capability 12.0) was not blocked by any architecture check. The server was launched with --moe-runner-backend flashinfer_cutedsl and successfully loaded the model. The initial single-stream test (msg 1246) showed 101ms/token with CuteDSL versus 97ms/token with CUTLASS — a 4% regression that the assistant attributed to JIT overhead and suboptimal M=1 tiling. Message 1251 confirms this with a proper benchmark: 10.07 tok/s vs 10.36 tok/s baseline, a 2.5% gap. The assistant's reasoning is that this gap will reverse at higher concurrency.

Assumptions Embedded in the Message

Message 1251 rests on several assumptions, some explicit and some implicit:

The primary assumption: That CuteDSL's JIT-compiled kernels will outperform pre-compiled CUTLASS kernels at larger batch sizes. This is a reasonable hypothesis — custom-generated kernels should in theory be better than generic ones — but it assumes that the JIT compiler's optimization passes are effective for the specific FP4 block-scaled GEMM format used by NVFP4 quantization. It also assumes that the compilation overhead doesn't negate the gains.

The secondary assumption: That the benchmark methodology is sound. The assistant uses sglang.bench_serving with random input/output lengths (512 input, 128 output tokens) and a fixed request rate of 100,000 (effectively unlimited). The number of prompts is set to conc * 4 to ensure adequate samples. These are reasonable parameters inherited from earlier benchmarks, but they embed assumptions about workload representativeness.

The implicit assumption: That the concurrency level directly translates to batch size (M) in the GEMM operations. In practice, SGLang's continuous batching and the --num-continuous-decode-steps 16 parameter mean that requests are batched in complex ways — not all requests in flight are necessarily processed together. The relationship between concurrency and effective M is nonlinear.

The assumption of comparability: The assistant compares CuteDSL results against a "baseline" established in earlier benchmarks. But those earlier benchmarks were run on a different kernel version (before the 6.14.11 upgrade) and potentially with different server parameters. The baseline numbers may not be directly comparable, though the assistant treats them as such.

The Input Knowledge Required

To fully understand message 1251, one needs knowledge spanning several domains:

Model architecture: GLM-5-NVFP4 is a 78-layer Mixture-of-Experts transformer with FP4 quantization. Each layer contains attention (QKV projections, attention computation, output projection) and a MoE block with gating, expert computation, and output combination. The FP4 format uses block-scaled quantization where groups of elements share a scale factor.

Hardware characteristics: The eight RTX PRO 6000 Blackwell GPUs have SM120 compute capability, 96GB of HBM3 memory each, and are connected via NVLink with P2P access. The theoretical memory bandwidth is approximately 1800 GB/s per GPU.

Software stack: SGLang is the serving framework, FlashInfer provides the kernel library, and CuteDSL is a JIT compilation backend within FlashInfer. The model is loaded with modelopt_fp4 quantization format and uses tensor parallelism across 8 GPUs.

Benchmarking methodology: The sglang.bench_serving tool measures throughput and latency under varying concurrency. Key metrics include Output token throughput (tok/s), TPOT (Time Per Output Token), ITL (Inter-Token Latency), and concurrency (average number of concurrent requests).

Prior optimization history: The assistant has been through numerous optimization cycles, establishing a baseline of 10.36 tok/s at concurrency 1, scaling to 1,603 tok/s at concurrency 1024 with the CUTLASS backend.

The Output Knowledge Created

Message 1251 produces several pieces of new knowledge:

  1. CuteDSL single-stream performance: 10.07 tok/s, 97.64ms TPOT — establishing that CuteDSL is marginally worse than CUTLASS at batch=1.
  2. CuteDSL at concurrency 10: 64.29 tok/s, 128.95ms mean TPOT — this is actually a significant improvement over the baseline of 38.46 tok/s at concurrency 10 (a 67% gain, though this comparison is revealed in msg 1252).
  3. CuteDSL at concurrency 64: 249.23 tok/s — also an improvement over the baseline of 218.74 tok/s (a 14% gain).
  4. The partial result pattern: Even in this truncated message, we see that CuteDSL is scaling better than expected at moderate concurrency levels. The concurrency=10 result of 64.29 tok/s is particularly striking — nearly doubling the baseline. These results are provisional. The full picture (revealed in msg 1252) shows that the trend reverses dramatically at higher concurrency: CuteDSL achieves only 515.58 tok/s at concurrency 256 (vs 718.83 baseline, a 28% loss) and 796.93 tok/s at concurrency 1024 (vs 1,603.04 baseline, a 50% loss). The message thus captures only the beginning of the story — the hopeful early data points before the disappointing conclusion.

The Thinking Process Visible in the Message

The assistant's thinking process is laid bare in the structure of message 1251. It follows a clear scientific pattern:

  1. Observe: "cutedsl at concurrency 1: 10.07 tok/s... baseline was 10.36 tok/s"
  2. Quantify the gap: "So cutedsl is about 2.5% slower at batch=1"
  3. Form a hypothesis: "Now let me test at higher concurrency levels where cutedsl should shine"
  4. Design an experiment: Run benchmarks at concurrency levels 10, 64, 256, 1024
  5. Execute and collect data: The bash command runs the full suite, and the assistant presents the results as they stream in The phrase "where cutedsl should shine" reveals the assistant's expectation. This is not a neutral "let me check" — it's a hypothesis-driven investigation. The assistant believes, based on architectural understanding, that CuteDSL's JIT compilation will pay dividends at larger batch sizes. The early results at concurrency 10 and 64 appear to confirm this hypothesis. But there's a subtle tension in the message. The assistant has already seen the single-stream regression (2.5% slower). The decision to proceed with the full benchmark suite despite this initial negative signal shows scientific discipline — a willingness to test a hypothesis even when early data is unfavorable.

Mistakes and Incorrect Assumptions

With the benefit of hindsight (knowing from msg 1252 that CuteDSL performs worse at high concurrency), we can identify several issues:

The JIT warmup assumption was partially wrong: The assistant assumed that poor high-concurrency performance might be due to JIT compilation happening for the first time at those batch sizes. Message 1252 tests this by re-running concurrency 256 and finds essentially identical results (515.77 tok/s vs 515.58 tok/s), confirming that JIT warmup was not the issue.

The scaling hypothesis was inverted: CuteDSL performed better at low concurrency (10, 64) and worse at high concurrency (256, 1024). This is the opposite of what the assistant expected. The hypothesis was that CuteDSL would excel at larger batch sizes, but the data shows it excels at smaller ones.

The baseline may not be directly comparable: The "baseline" numbers (10.36 tok/s at concurrency 1, 38.46 at concurrency 10, etc.) were established in earlier segments, potentially with different system configurations. The kernel upgrade and other system changes between segments may have shifted the baseline, making the comparison inexact.

The benchmark parameters may mask effects: Using --num-continuous-decode-steps 16 means the server batches 16 decode steps before checking the scheduler. This interacts with concurrency in complex ways that may favor one backend over another depending on the exact scheduling dynamics.

Broader Significance

Message 1251 is a microcosm of the entire optimization effort. It captures the moment when a promising new approach is put to the test and delivers surprising results. The CuteDSL experiment, like many optimization attempts in this session, yields a mixed verdict: better in some regimes, worse in others. The assistant's disciplined response — gathering data, comparing systematically, and investigating anomalies — is the scientific method applied to systems optimization.

The message also illustrates a fundamental truth about ML inference optimization: theoretical advantages don't always translate to practical gains. CuteDSL's ability to generate custom kernels is theoretically superior, but the actual performance depends on the quality of the generated code, the interaction with the serving framework's batching logic, and the specific characteristics of the FP4 block-scaled format. In this case, the pre-compiled CUTLASS kernels, despite their generic nature, proved more effective at high throughput — a reminder that in systems engineering, the best solution is not always the most sophisticated one.

For the reader following this optimization journey, message 1251 is a turning point. It represents the last major backend exploration before the team pivots to deeper analysis of the FP4 GEMM kernel overhead itself. The CuteDSL experiment, while not delivering the hoped-for breakthrough, provides crucial data that helps narrow the search space. The bottleneck, it turns out, is not in the choice of MoE backend but in the fundamental efficiency of FP4 computation on SM120 — a problem that would require deeper kernel-level investigation to resolve.