The Benchmark That Wasn't: Shell Quoting, FP4 GEMMs, and the Quest for Blackwell GPU Utilization

Introduction

In the middle of an intensive optimization session for GLM-5-NVFP4 inference on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant sent a command that appears, at first glance, to be a routine benchmark. Message [msg 872] is a single bash tool invocation: an SSH command executing a Python script on a remote machine to compare the performance of two FP4 matrix-multiplication implementations — sgl_kernel.cutlass_scaled_fp4_mm versus flashinfer.mm_fp4. The script fails with a NameError. On the surface, this is a trivial error. But the message is far more interesting than its outcome suggests. It sits at a critical juncture in a deep investigation into why Blackwell GPUs are delivering far below their theoretical peak, and it reveals the messy, iterative reality of performance engineering at the frontier of AI hardware.

Context: The Performance Investigation

To understand [msg 872], we must understand what came before it. The assistant had been working for hours to optimize GLM-5-NVFP4, a Mixture-of-Experts model with 256 experts, running on eight RTX PRO 6000 Blackwell GPUs. Earlier benchmarking had established that the model was compute-bound, not communication-bound — a TP4+PP2 configuration was 2× slower than TP8, ruling out allreduce latency as the primary bottleneck. The real problem was deeper: the FP4 GEMM kernels were running at a fraction of their theoretical peak.

In [msg 867] and [msg 868], the assistant had benchmarked the FlashInfer CUTLASS FP4 kernel across a range of matrix sizes. The results were sobering:

| Batch Size | TFLOPS | % of ~2000 TOPS Peak | |------------|--------|---------------------| | 1 token | 0.3–0.8 | <0.1% | | 64 tokens | 15–55 | 1–3% | | 256 tokens | 81–219 | 4–11% | | 512 tokens | 161–437 | 8–22% | | 4K+ tokens | 1,190–1,313 | 60–66% |

The kernels plateaued at around 1,300 TFLOPS — about 65% of the estimated 2,000 TOPS peak for Blackwell. But the truly alarming numbers were the small-batch results. During decode, each MoE expert processes at most batch_size / num_experts_per_token tokens. With 256 experts, 8 activated per token, and 512 concurrent requests, each expert gets roughly 16 tokens on average. That puts inference squarely in the 1–3% of peak regime. The small GEMM sizes were catastrophic for utilization.

This analysis set the stage for [msg 872]. The assistant had just discovered that sgl_kernel — a separate library bundled with SGLang — contained its own FP4 GEMM implementation (cutlass_scaled_fp4_mm). The natural next question: is sgl_kernel's kernel faster than FlashInfer's? If so, swapping the backend could yield an immediate throughput improvement without any architectural changes.

The Message: A Benchmark Designed to Compare Two FP4 Kernels

The message (reproduced below with secrets redacted) is a single bash command that SSHs into the remote machine and executes a Python script via a heredoc:

ssh root@[REDACTED] 'source /root/ml-env/bin/activate && python3 << '"'"'PYEOF'"'"'
import torch
import sgl_kernel
import flashinfer
import time

device = "cuda:0"
torch.cuda.set_device(device)

sizes = [
    (16, 2048, 6144, "Expert batch=16 (decode)"),
    (64, 2048, 6144, "Expert batch=64"),
    (256, 2048, 6144, "Expert batch=256"),
    (512, 2048, 6144, "Expert batch=512"),
    (1024, 2048, 6144, "Expert batch=1024"),
    (16, 768, 6144, "Attn proj batch=16"),
    (64, 768, 6144, "Attn proj batch=64"),
    (256, 768, 6144, "Attn proj batch=256"),
]

alpha = torch.tensor(1.0, dtype=torch.float32, device=device)

header = f"{'Description':35s} | {'sgl_kernel':>12s} | {'flashinfer':>12s} | {'ratio':>8s}"
print(header)
print("-" * len(header))

for M, N, K, desc in sizes:
    a_fp4 = torch.randint(0, 255, (M, K//2), dtype=torch.uint8, device=device)
    b_fp4 = torch.randint(0, 255, (K//2, N), dtype=torch.uint8, device=device)
    a_scale = torch.randint(1, 127, (M, K//16), dtype=torch.uint8, device=device).view(torch.float8_e4m3fn)
    b_scale = torch.randint(1, 127, (K//16, N), dtype=torch.uint8, device=device).view(torch.float8_e4m3fn)
    
    # sgl_kernel benchmark
    for _ in range(5):
        sgl_kernel.cutlass_scaled_fp4_mm(a_fp4, b_fp4, a_scale, b_scale, alpha, torch.bfloat16)
    torch.cuda.synchronize()
    
    iters = 100
    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(iters):
        sgl_kernel.cutlass_scaled_fp4_mm(a_fp4, b_fp4, a_scale, b_scale, alpha, torch.bfloat16)
    torch.cuda.synchronize()
    sgl_us = (time.perf_counter() - start) / iters * 1e6
    
    # flashinfer benchmark
    out = torch.zeros(M, N, dtype=torch.bfloat16, device=device)
    for _ in range(5):
        flashinfer.mm_fp4(a_fp4, b_fp4, a_scale, b_scale, alpha=alpha, out=out, backend="cutlass")
    torch.cuda.synchronize()
    
    start = time.perf_counter()
    for _ in range(iters):
        flashinfer.mm_fp4(a_fp4, b_fp4, a_scale, b_scale, alpha=alpha, out=out, backend="cutlass")
    torch.cuda.synchronize()
    fi_us = (time.perf_counter() - start) / iters * 1e6
    
    speedup = fi_us / sgl_us
    print(f"{desc:35s} | {sgl_us:9.1f} us | {fi_us:9.1f} us | {speedup:7.2f}x")
PYEOF'

The script is well-structured. It defines a set of matrix sizes that correspond to realistic inference workloads — MoE expert up-projections (N=2048) and attention projections (N=768), with batch sizes ranging from 16 to 1024. For each size, it runs 5 warmup iterations followed by 100 timed iterations, measuring microseconds per call and computing a speedup ratio. This is a textbook microbenchmark: warmup to eliminate cold-start effects, multiple iterations for statistical stability, CUDA synchronization for accurate timing.

The benchmark compares two libraries:

The Follow-Up: A Better Approach

In the very next message ([msg 873]), the assistant abandoned the inline approach entirely and switched to writing a script file:

ssh root@[REDACTED] 'cat > /tmp/bench_fp4.py << "PYEOF"
...
PYEOF
echo "Script written"'

This approach uses cat to write the Python script to a temporary file, with the heredoc delimiter quoted (&#34;PYEOF&#34;) to prevent any shell expansion. The Python code is then executed by running the script file. This cleanly separates the shell quoting from the Python code, eliminating the quoting conflict entirely. It's a textbook solution to the "script inside a command" problem.

The assistant's thinking process here is visible and instructive. Rather than continuing to fight with quoting, it recognized the fundamental limitation of the inline approach and chose a different strategy. This is a pattern seen throughout the session: when a straightforward approach fails, the assistant doesn't just retry with minor tweaks — it reconsiders the approach fundamentally.

What the Benchmark Was Trying to Discover

Despite the failure, the benchmark's design reveals the assistant's investigative strategy. The matrix sizes were carefully chosen:

  1. MoE expert sizes (N=2048, K=6144): These correspond to the up-projection and down-projection matrices in GLM-5's MoE layers. With 256 experts and 8 activated per token, each expert's GEMM operates on a small subset of tokens. The batch sizes (16, 64, 256, 512, 1024) cover the range from worst-case decode (16 tokens per expert) to near-peak utilization (1024 tokens per expert).
  2. Attention projection sizes (N=768, K=6144): These correspond to the QKV projections in the attention mechanism, which are dense (not MoE) and thus see the full batch. The smaller N dimension (768 with TP8) makes these GEMMs more memory-bound.
  3. The ratio metric: By computing fi_us / sgl_us, the benchmark directly answers the question "how much faster is sgl_kernel than flashinfer?" A ratio > 1 means sgl_kernel is faster; < 1 means flashinfer is faster. This is a clean, actionable metric. The assistant assumed that both libraries implement the same FP4 GEMM operation — which is true at the mathematical level — but they may differ in kernel tuning parameters like tile sizes, thread block configurations, and shared memory usage. The CUTLASS framework allows extensive customization of these parameters, and different libraries may have made different choices. The benchmark was designed to detect these differences empirically.

Assumptions and Potential Pitfalls

The benchmark makes several assumptions that are worth examining:

  1. Representative data: Random FP4 data with random scales may not perfectly represent real model weights and activations. Real weights have structure (e.g., outliers in certain channels) that could affect kernel performance differently than uniform random data.
  2. Isolated measurement: The benchmark measures each GEMM in isolation, without the surrounding attention, MoE routing, and layer normalization operations. In practice, kernel launch overhead, memory traffic patterns, and GPU pipeline occupancy interact in complex ways that isolated benchmarks can't capture.
  3. Single-GPU measurement: The benchmark runs on GPU 0 only. In a TP8 configuration, the GEMM dimensions are smaller (distributed across GPUs), which changes the arithmetic intensity and could shift the relative performance of the two kernels.
  4. Warmup sufficiency: Five warmup iterations may not be enough to fully "warm up" the GPU frequency and memory clock. Modern GPUs have power management that ramps up over hundreds of microseconds.
  5. The auto backend: The benchmark explicitly uses backend=&#34;cutlass&#34; for FlashInfer. The auto backend (which tries cuDNN first) had already been shown to fail on SM120 ([msg 866]). This is a correct choice, but it means the benchmark doesn't test cuDNN's FP4 implementation, which might be faster for some sizes.

Input Knowledge Required

To understand this message fully, one needs:

Output Knowledge Created

Even though the benchmark failed to produce numerical results, the message created valuable knowledge:

  1. Confirmation that sgl_kernel has a usable FP4 GEMM: The import succeeded, meaning cutlass_scaled_fp4_mm is available and compatible with the installed CUDA/PyTorch versions. This is non-trivial — earlier in the session, many kernel libraries failed to load on SM120.
  2. The API signature is compatible: The function takes (a, b, block_scale_a, block_scale_b, alpha, out_dtype) — the same pattern as FlashInfer's mm_fp4. This means swapping between them is mechanically straightforward.
  3. The benchmark infrastructure works: The script structure (warmup, timed loop, synchronization, multiple sizes) is reusable. The assistant reused it successfully in subsequent messages by switching to a file-based approach.
  4. Shell quoting is the enemy: The failure reinforces a lesson that will pay dividends in future messages. The assistant's next attempt ([msg 873]) uses the file-based approach successfully.

The Broader Investigation

This message is a small piece of a much larger puzzle. The assistant was systematically investigating why Blackwell GPUs were underperforming for GLM-5 inference. The FP4 GEMM benchmark was just one thread in a multi-pronged investigation that included:

Conclusion

Message [msg 872] is a study in the gap between intention and execution in performance engineering. The assistant had a clear hypothesis, a well-designed benchmark, and the right tools. The failure was purely mechanical — a shell quoting issue that prevented the Python code from reaching the remote interpreter intact. The assistant's response to the failure was exemplary: diagnose the root cause (shell parsing of f-strings), recognize the fundamental limitation of the inline approach, and switch to a file-based strategy.

In the broader context of the session, this message represents the iterative, sometimes frustrating process of working at the frontier of AI hardware. Blackwell GPUs are new, the FP4 kernels are experimental, and the tooling is still evolving. Every benchmark that fails, every kernel that crashes, every quoting error is a data point that shapes the assistant's understanding of the system. The assistant's persistence — trying different approaches, learning from failures, and methodically working through the stack — is the real story here.

The FP4 GEMM comparison would eventually be answered in subsequent messages, but the answer (that both libraries use similar CUTLASS kernels and neither is dramatically faster) was less important than the investigative process itself. In performance engineering, knowing which questions to ask is often more valuable than getting immediate answers.