When Shell Escaping Bites: A Micro-Tale of Debugging in the Trenches of ML Performance Analysis

Introduction

In the high-stakes world of large language model inference optimization, every microsecond counts. When you're trying to squeeze maximum throughput from a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs running a 5-billion-parameter Mixture-of-Experts model, the difference between a well-tuned kernel and a mediocre one can mean thousands of tokens per second in lost performance. But before you can optimize, you must measure — and before you can measure, you must get your benchmark script to actually run.

This article examines a single message ([msg 873]) from an extended coding session where an AI assistant was deep in the process of characterizing FP4 GEMM (General Matrix Multiply) kernel performance on Blackwell GPUs. The message itself is deceptively simple: the assistant, after two failed attempts to run a benchmark comparison script via SSH, identifies the root cause as shell escaping issues and switches to a file-based approach. But this seemingly trivial moment reveals a wealth of insight about the challenges of remote ML development, the assumptions we make about tooling, and the iterative debugging process that characterizes real-world systems work.

The Context: A Deep Dive into FP4 Kernel Efficiency

To understand why message [msg 873] matters, we need to appreciate the investigation that led to it. The assistant had been working on deploying the GLM-5-NVFP4 model using SGLang across 8 GPUs, and had achieved impressive throughput of around 3,740 tokens per second after enabling FlashInfer CUTLASS MoE autotuning. However, GPU power utilization was low — the cards were drawing only ~235W out of a 600W TDP — suggesting significant room for improvement.

The investigation in messages [msg 865] through [msg 872] had been a systematic deep-dive into FP4 GEMM kernel performance. The assistant had:

  1. Verified that CUTLASS FP4 kernels work on SM120 (Blackwell's compute capability). Earlier attempts with flashinfer's mm_fp4 function had failed due to missing arguments (alpha and out tensors), but once properly called, the kernels produced non-zero results.
  2. Benchmarked FP4 GEMM at various sizes relevant to GLM-5's architecture (hidden_size=6144, moe_intermediate_size=2048, with tensor parallelism dividing these across GPUs). The results were sobering: at single-token decode, the kernel achieved only 0.3 TFLOPS — a mere 0.02% of the theoretical ~2,000 TOPS peak. Even at batch=64, it reached only 15-55 TFLOPS (1-3% of peak). The kernels only began to approach saturation at batch sizes of 4K+, plateauing around 1,200-1,300 TFLOPS (about 65% of peak).
  3. Identified the fundamental bottleneck: During inference, each MoE expert processes at most batch_size / num_experts_per_token tokens. With 256 experts, 8 activated per token, and 512 concurrent requests, each expert gets roughly 16 tokens on average. This puts the GEMM sizes squarely in the 15-55 TFLOPS range — catastrophically low utilization. The model is memory-bandwidth-bound during decode because the per-expert batch sizes are tiny, an inherent consequence of the MoE architecture with 256 experts scattering tokens too thinly.
  4. Discovered sgl_kernel as a potential alternative. The SGLang kernel library exposed a cutlass_scaled_fp4_mm function that might be faster than flashinfer's mm_fp4. The assistant had already verified the function signature and was eager to benchmark it head-to-head against flashinfer.## The Message: A Shell Escaping Nightmare Message [msg 873] is the assistant's response to two consecutive failures. In [msg 871], the assistant attempted to run a Python benchmark script via SSH using a one-liner with python3 -c "...". This failed with NameError: name 'Desc' is not defined — the f-string f"{'Desc':40s} | ..." was being mangled by the shell's quote parsing. The single and double quotes inside the Python string were conflicting with the outer shell quoting, causing the shell to interpret parts of the Python code before Python ever saw it. In [msg 872], the assistant tried a different approach: a heredoc with python3 << 'PYEOF'. But this also failed with the same NameError: name 'Description' is not defined. The issue here was subtler: the heredoc delimiter 'PYEOF' was itself wrapped in a complex SSH quoting structure (ssh ... 'cat > /tmp/bench_fp4.py << "PYEOF" ... PYEOF'), and the shell was still interfering with the Python string literals. Message [msg 873] shows the assistant's diagnosis and solution, quoted in full:
[assistant] The f-string is being parsed by the shell. Let me use a script file:
[bash] ssh root@10.1.230.174 'cat > /tmp/bench_fp4.py << "PYEOF"
import torch
import sgl_kernel
import flashinfer
import time

device = "cuda:0"
torch.cuda.set_device(device)

sizes = [
    (16, 2048, 6144, "Expert batch=16 (decode)"),
    (64, 2048, 6144, "Expert batch=64"),
    (256, 2048, 6144, "Expert batch=256"),
    (512, 2048, 6144, "Expert batch=512"),
    (1024, 2048, 6144, "Expert batch=1024"),
    (16, 768, 6144, "Attn proj batch=16"),
    (64, 768, 6144, "Attn proj batch=64"),
    (256, 768, 6144, "Attn proj batch=256"),
]

alpha = torch.tensor(1.0, dtype=torch.float32, device=device)

print(f"Description                         | sgl_kernel   | flashinfer   |    ratio")
print("-" * 80)

for M, N, K, desc in sizes:
    a_fp4 = torch.randint(0, 255, (M, K//2), dtype=torch.uint8, device=device)
    b_fp4 = torch.randint(0, 255, (K//2, N), dtype=torch.uint8, device=device)
    a_scale = torch.randint(1, 127, (M, K//16), dtype=torch.uint8, device=device).view(torch.float8_e4m3fn)
    b_scale = torch.randint(1, 127, (K//16, N), dtype=torch.uint8, device=device).view(torch.float8_e4m3fn)

    # sgl_kernel benchmark
    for _ in range(5):
        sgl_kernel.cutlass_scaled_fp4_mm(a_fp4, b_fp4, a_scale, b_scale, alpha, torch.bfloat16)
    torch.cuda.synchronize()

    iters = 100
    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(iters):
        sgl_kernel.cutlass_scaled_fp4_mm(a_fp4, b_fp4, a_scale, b_scale, alpha, torch.bfloat16)
    torch.cuda.synchronize()
    sgl_us = (time.perf_counter() - start) / iters * 1e6

    # flashinfer benchmark
    out = torch.zeros(M, N, dtype=torch.bfloat16, device=device)
    for _ in range(5):
        flashinfer.mm_fp4(a_fp4, b_fp4, a_scale, b_scale, alpha=alpha, out=out, backend="cutlass")
    torch.cuda.synchronize()

    start = time.perf_counter()
    for _ in range(iters):
        flashinfer.mm_fp4(a_fp4, b_fp4, a_scale, b_scale, alpha=alpha, out=out, backend="cutlass")
    torch.cuda.synchronize()
    fi_us = (time.perf_counter() - start) / iters * 1e6

    speedup = fi_us / sgl_us
    print(f"{desc:35s} | {sgl_us:9.1f} us | {fi_us:9.1f} us | {speedup:7.2f}x")
PYEOF
echo "Script written"'
Script written

The assistant correctly identifies that the root cause is shell escaping — the f-strings containing curly braces {} and the nested quote characters are being interpreted by the shell before Python ever receives them. The solution is elegantly simple: write the Python script to a file on the remote machine first, then execute it. This avoids all the quoting gymnastics of inline Python execution.

The command uses a heredoc within the SSH command to write the script to /tmp/bench_fp4.py. The key difference from the previous attempt is that the heredoc is now properly isolated: the &#34;PYEOF&#34; delimiter is quoted to prevent shell expansion within the heredoc body, and the entire SSH command is wrapped in single quotes to prevent the local shell from interfering. The script file approach completely sidesteps the quoting problems that plagued the inline execution attempts.

Why This Matters: The Hidden Complexity of Remote Execution

To an outside observer, this message might look like a trivial workaround for a mundane syntax error. But it represents a critical juncture in the investigation. The assistant had spent considerable effort building up a detailed understanding of FP4 kernel performance, and was now at the point where comparing sgl_kernel against flashinfer could reveal whether there was a faster path forward. Every failed attempt to run the benchmark was time lost — and in a session where each SSH round-trip takes seconds, even simple debugging can consume minutes.

The decision to switch to a file-based approach reveals several assumptions and insights:

  1. The assistant assumes the Python code itself is correct. The benchmark script had been tested mentally and appeared logically sound. The failures were purely mechanical — shell escaping issues, not semantic bugs.
  2. The assistant recognizes the limits of inline Python execution. While python3 -c &#34;...&#34; is convenient for quick tests, it becomes fragile when the Python code contains complex string formatting, f-strings with curly braces, or nested quotes. The f-strings in the benchmark script — f&#34;{desc:35s} | {sgl_us:9.1f} us | ...&#34; — contain curly braces that conflict with shell brace expansion and quoting rules.
  3. The file-based approach is more robust for multi-line scripts. Writing to a temp file eliminates quoting issues entirely, at the cost of an extra command to execute the file afterward. The trade-off is worth it for scripts longer than a few lines.
  4. The assistant is operating under time pressure. Rather than debugging the shell escaping further (which could involve trial-and-error with different quoting strategies), the assistant chooses the most direct path to getting the benchmark running. This pragmatic decision prioritizes the investigation's momentum over understanding the exact quoting failure.

Input Knowledge Required

To fully understand this message, the reader needs:

Mistakes and Incorrect Assumptions

The most notable mistake is in the previous message ([msg 872]), where the assistant attempted a heredoc approach but still got the quoting wrong. The error NameError: name &#39;Description&#39; is not defined suggests that the Python code was corrupted during transmission — the header variable assignment was likely mangled by shell interpretation. The assistant correctly identifies the root cause in [msg 873] ("The f-string is being parsed by the shell") and switches to a file-based approach.

However, there's a subtle mistake in the solution itself: the assistant writes the script to /tmp/bench_fp4.py but never executes it in this message. The echo &#34;Script written&#34; at the end confirms the file was created, but the actual benchmark results won't appear until the next round. This means the assistant can't act on the results within this message — it must wait for the next round to see the output. This is a consequence of the synchronous round-based architecture of the assistant, where all tool calls in a round are dispatched together and results arrive only in the next message.

Another potential issue: the script uses torch.cuda.set_device(device) with device = &#34;cuda:0&#34;, but the model was deployed across 8 GPUs. Benchmarking on a single GPU may not reflect the performance characteristics of the distributed inference pipeline, where communication overhead and load balancing also play roles. The assistant is aware of this — the preceding investigation had already established that the model is compute-bound rather than communication-bound — but the benchmark results should still be interpreted with this caveat.

Output Knowledge Created

This message creates several pieces of output knowledge:

  1. A benchmark script (/tmp/bench_fp4.py) on the remote machine that compares sgl_kernel.cutlass_scaled_fp4_mm against flashinfer.mm_fp4 across a range of matrix sizes relevant to GLM-5 inference. The script is reusable and can be modified to test additional sizes or configurations.
  2. Confirmation that the file-based approach works for complex Python scripts with f-strings. The echo &#34;Script written&#34; output confirms successful file creation. This establishes a pattern for future remote execution: write scripts to temp files rather than using inline Python.
  3. A documented debugging process. The sequence of failed attempts ([msg 871], [msg 872]) followed by the successful workaround ([msg 873]) serves as a case study in diagnosing shell escaping issues. The assistant's explicit diagnosis — "The f-string is being parsed by the shell" — captures the root cause succinctly.
  4. The benchmark results themselves (which appear in subsequent messages) will provide a direct performance comparison between two FP4 GEMM implementations. This is valuable knowledge for anyone optimizing GLM-5 inference on Blackwell GPUs.

The Thinking Process

The assistant's reasoning is visible in the message's structure and content. The opening line — "The f-string is being parsed by the shell. Let me use a script file" — shows a clear diagnosis followed by a decisive action. The assistant doesn't waste time speculating about other possible causes (Python version issues, library incompatibilities, etc.) because the error pattern is unambiguous: the NameError in the previous attempts pointed to corrupted Python code, and the most likely cause was shell interference with the f-string syntax.

The choice of a heredoc within SSH rather than a multi-step approach (e.g., scp to copy a local file) reflects the assistant's preference for self-contained commands. A single SSH command that both creates and verifies the script is more efficient than separate commands for writing and checking. The echo &#34;Script written&#34; provides immediate confirmation without requiring an additional ls or cat command.

The assistant also shows awareness of the broader investigation context. The benchmark script is carefully designed with sizes that mirror the actual inference workload: expert batch sizes of 16-1024 tokens, attention projection sizes of 16-256 tokens, all with the model's actual hidden dimension (6144) and intermediate size (2048). This is not a generic benchmark — it's targeted at understanding a specific performance bottleneck in a specific deployment.

Broader Significance

This message, while seemingly minor, illustrates a fundamental truth about systems work: the infrastructure for measurement is often the hardest part. The assistant had already done the hard intellectual work of identifying the performance bottleneck (small per-expert batch sizes leading to low FP4 GEMM utilization), formulating a hypothesis (sgl_kernel might be faster than flashinfer), and designing a benchmark to test it. But the actual execution was derailed by something as mundane as shell quoting.

This pattern is universal in ML engineering. The most sophisticated analysis is worthless if you can't run the experiment. The ability to diagnose and fix infrastructure issues quickly — whether they're shell escaping, CUDA version mismatches, or network configuration problems — is what separates effective practitioners from those who get stuck.

The message also demonstrates the value of iterative debugging. The assistant didn't get the benchmark right on the first try, or even the second. But each failure provided information that narrowed down the root cause. The first failure (f-string parsing in python3 -c) suggested a quoting issue. The second failure (heredoc still failing) confirmed that the issue was more subtle than simple quote mismatches. The third attempt (file-based approach) succeeded by completely sidestepping the problem.

In the end, the benchmark did run successfully ([msg 874]), revealing that sgl_kernel was consistently 2-3x faster than flashinfer for the small batch sizes critical to inference. This finding directly informed the assistant's subsequent optimization strategies. But none of that would have been possible without first solving the shell escaping problem — a problem that has nothing to do with machine learning and everything to do with the messy reality of getting code to run on remote machines.