A Benchmark That Wasn't: The F-String That Broke the FP4 Kernel Comparison

Introduction

In the middle of an intense deep-dive into FP4 GEMM performance on NVIDIA's latest Blackwell-generation RTX PRO 6000 GPUs, the assistant dispatched a carefully constructed benchmark to compare two competing kernel implementations head-to-head. The command—a Python script tunneled through SSH into a remote server—was designed to answer a straightforward question: which FP4 matrix multiplication backend runs faster on SM120 hardware? But the answer never came. Instead, the remote terminal returned a terse Python error: NameError: name 'Desc' is not defined. The benchmark collapsed before it could produce a single data point.

This message ([msg 871]) is a study in the fragility of complex engineering workflows, the hidden pitfalls of multi-level string escaping, and the relentless iterative nature of performance engineering. Even in failure, it reveals the assistant's reasoning, assumptions, and methodology for comparing GPU kernel implementations.

Context: The FP4 Performance Investigation

To understand why this message was written, one must trace back through the preceding nine messages ([msg 862] through [msg 870]), which constitute a focused investigation into FP4 GEMM performance on the SM120 architecture. The assistant had been working on deploying the GLM-5-NVFP4 model across eight RTX PRO 6000 Blackwell GPUs, and performance was a persistent concern.

The investigation had already produced several critical findings. First, the assistant discovered that FlashInfer's CUTLASS FP4 backend does produce non-zero results on SM120—contradicting a recently filed bug report (FlashInfer issue #2577) that claimed the backend returned all zeros. The issue turned out to be a missing alpha and out tensor in the function call, not a fundamental kernel bug ([msg 866]).

Second, a systematic benchmark of FlashInfer's mm_fp4 across matrix sizes representative of actual inference workloads revealed a sobering picture ([msg 867]). For single-token decode—the most common operation during text generation—the FP4 GEMM achieved a mere 0.3–0.8 TFLOPS, less than 0.1% of the theoretical ~2,000 TOPS peak. Even at batch 64, utilization hovered at 1–3%. The kernels only began to show reasonable performance at batch sizes of 512 or higher, plateauing around 1,200–1,300 TFLOPS (about 65% of peak) at batch sizes of 4K or more ([msg 868]).

Third, the assistant identified the root cause of this poor utilization: the Mixture-of-Experts architecture of GLM-5-NVFP4, which has 256 experts with 8 activated per token. During decode, each expert processes at most batch_size × 8 / 256 tokens. With 512 concurrent requests, each expert gets roughly 16 tokens—placing the operation squarely in the 15–55 TFLOPS range where utilization is catastrophically low ([msg 869]).

Fourth, a comparison with BF16 matrix multiplication showed that FP4 was only about 2× faster than BF16 (437 TFLOPS vs 225 TFLOPS for a 512×2048×6144 GEMM), far short of the theoretical 4× advantage. This confirmed that the FP4 kernels were not optimally tuned for SM120 ([msg 870]).

The Discovery of sgl_kernel

The immediate trigger for message [msg 871] came at the end of message [msg 870]. While probing the available FP4 functions, the assistant discovered that sgl_kernel—a companion library to the SGLang inference engine—exposed a function called cutlass_scaled_fp4_mm. The signature was clean and simple: it took FP4 input tensors, block scales, an alpha scaling factor, and an output dtype, and returned the result directly (no pre-allocated output tensor required).

This was a promising alternative to FlashInfer's mm_fp4, which required a pre-allocated out tensor and an alpha tensor, and which had demonstrated multiple backend failures (cuDNN rejected SM120, trtllm explicitly unsupported, and only CUTLASS worked at all). The assistant's reasoning was straightforward: if sgl_kernel had a better-tuned CUTLASS implementation for SM120, it could provide an immediate performance improvement without any architectural changes to the inference pipeline.

The Benchmark Design

The assistant designed a head-to-head comparison across eight matrix sizes that mapped directly to real inference workloads. The sizes fell into two categories:

  1. MoE expert GEMMs (N=2048, K=6144): These represent the up-projection in a Mixture-of-Experts layer, where the hidden dimension (6144) is projected to the intermediate size (2048). Batch sizes ranged from 16 (typical decode) to 1024.
  2. Attention projection GEMMs (N=768, K=6144): These represent the query/key/value projections in attention, where the hidden dimension is projected to the per-tensor-parallel-rank size (6144 → 768 with TP8). Batch sizes ranged from 16 to 256. The methodology was sound: warmup iterations to eliminate cold-start effects, 100 measurement iterations with torch.cuda.synchronize() barriers for accurate timing, and a formatted table output showing the latency of each backend and the speedup ratio. The decision to use time.perf_counter() rather than CUDA events was a pragmatic choice for a remote SSH command—it avoids the complexity of CUDA event synchronization while still providing microsecond-level precision when combined with explicit torch.cuda.synchronize() calls.

The Failure

The benchmark never ran. Python raised a NameError on line 23, where the assistant attempted to print a formatted table header:

print(f"{'Desc':40s} | {'sgl_kernel':>12s} | {'flashinfer':>12s} | {'speedup':>8s}")

The error message—name 'Desc' is not defined—indicates that Python interpreted Desc as a variable name rather than as a string literal. This is a classic f-string pitfall that occurs when the quoting gets mangled during string interpolation.

The root cause lies in the multi-level escaping required to embed this Python script inside a bash command passed over SSH. The command structure was:

ssh root@HOST 'source /root/ml-env/bin/activate && python3 -c "
    ... Python code with escaped quotes ...
"'

The Python code itself contained f-strings delimited by double quotes, which had to be escaped as \" to survive inside the outer double-quoted Python argument. Inside those f-strings, single quotes were used for string literals like 'Desc'. Under normal circumstances, this should work: the single quotes are protected by the double-quoted f-string, and the double quotes are escaped for the shell.

However, the specific syntax {'Desc':40s} appears to have triggered a parsing issue. In Python f-strings, the expression 'Desc' is a valid string literal, and :40s is a valid format specifier. But when the shell preprocesses the command, the interaction between the outer single quotes (for the SSH command), the inner double quotes (for the Python argument), and the escaped double quotes (for the f-string) can produce unexpected results. The single quotes around 'Desc' may have been consumed or altered during one of the escaping layers, leaving Desc exposed as a bare identifier.

Assumptions and Their Consequences

The assistant made several assumptions that proved incorrect:

Assumption 1: The f-string syntax would survive the escaping layers. This was the most consequential assumption. The assistant had successfully used similar f-string patterns in earlier benchmarks ([msg 867], [msg 868]) without issue, so there was reasonable confidence. However, those earlier benchmarks used simpler format strings without single-quoted string literals inside the f-string expressions. The addition of {'Desc':40s} introduced a quoting conflict that the earlier patterns had avoided.

Assumption 2: The benchmark would complete without errors. The assistant wrote the script as a single monolithic command, assuming it would execute successfully. A more defensive approach would have been to write the Python script to a file first, then execute it—eliminating the escaping complexity entirely.

Assumption 3: sgl_kernel.cutlass_scaled_fp4_mm and flashinfer.mm_fp4 are directly comparable. While both functions perform FP4 matrix multiplication, they have different interfaces (sgl_kernel returns a new tensor, flashinfer writes to a pre-allocated output) and potentially different numerical behavior. The benchmark assumed that any performance difference would be attributable to kernel optimization rather than interface overhead or numerical differences.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

Output Knowledge Created

Despite the failure, this message contributes valuable knowledge:

  1. A documented benchmark methodology for comparing FP4 kernel implementations, including appropriate matrix sizes, warmup procedures, and timing techniques.
  2. Confirmation that multi-level escaping in SSH commands is fragile, especially when f-strings with nested quotes are involved. This is a practical lesson for anyone doing remote GPU benchmarking.
  3. A reference point for the expected performance range: Even though the comparison didn't run, the earlier benchmarks ([msg 867], [msg 868]) established that FlashInfer's CUTLASS FP4 achieves 0.3–1,300 TFLOPS depending on batch size, and that sgl_kernel's implementation is a candidate for further investigation.
  4. The set of matrix sizes that matter for GLM-5-NVFP4 inference: (M=16–1024, N=768 or 2048, K=6144), which directly correspond to the attention and MoE layer dimensions under TP8.

The Thinking Process

The assistant's reasoning is visible in the structure of the benchmark. The choice to test both MoE and attention projection sizes reflects a deep understanding of the model architecture. The inclusion of batch size 16 as "typical decode" shows awareness that during actual inference, the per-expert batch size is determined by concurrent_requests × experts_per_token / total_experts, not by the overall request batch size.

The decision to benchmark sgl_kernel specifically, rather than other backends, was informed by the discovery in [msg 870] that cuDNN and trtllm backends both fail on SM120. The assistant had systematically eliminated alternatives before arriving at this comparison.

The use of torch.cuda.synchronize() between warmup and measurement phases indicates awareness of CUDA's asynchronous execution model. Without these barriers, the timing would include kernel launch overhead from previous iterations, producing unreliable results.

The 100-iteration measurement count represents a trade-off between statistical reliability and execution time. For small GEMMs that complete in ~30 microseconds, 100 iterations provides a 3-millisecond total measurement window—enough to average out noise from GPU clock fluctuations and memory contention.

Broader Significance

This failed benchmark is a microcosm of the challenges in high-performance ML engineering. The work involves not just understanding model architectures and GPU hardware, but also managing the tooling complexity of remote execution, multi-language integration (Python + bash + SSH), and the subtle interactions between quoting conventions.

The failure also illustrates an important principle: in performance engineering, the infrastructure for measuring performance is itself a significant engineering challenge. The assistant spent nine messages building up the measurement infrastructure—verifying kernel correctness, establishing baseline numbers, identifying bottlenecks—before attempting this head-to-head comparison. The failure of the comparison script is not a setback but a debugging opportunity. The fix is straightforward (adjust the f-string quoting or write the script to a file), and the methodology remains sound.

In the broader narrative of the GLM-5-NVFP4 deployment, this message represents a moment of attempted acceleration. The assistant had identified a potential optimization path (sgl_kernel's FP4 kernels) and moved quickly to evaluate it. The failure is a reminder that even well-reasoned engineering decisions can be derailed by mundane technical details—and that persistence through these failures is what distinguishes effective performance engineering.