The Benchmark That Broke: A Shell Escaping Failure in the Pursuit of 100 tok/s

Introduction

In the high-stakes world of large language model deployment, performance optimization is a relentless pursuit. When a user demands 100 tokens per second from a 744-billion-parameter Mixture-of-Experts model running across eight Blackwell GPUs, every millisecond counts. Message [msg 1994] captures a pivotal moment in this optimization journey — a moment of triumph that immediately collides with a mundane but frustrating technical obstacle. The assistant, having just doubled throughput from 20 tok/s to 43 tok/s by enabling CUDAGraph, attempts to run a comprehensive multi-concurrency benchmark to measure the full impact. But the benchmark script never runs. Instead, the shell returns a cryptic parse error: zsh:35: parse error near 'tok/s'. This message is a case study in how even the most sophisticated AI engineering work can be derailed by something as simple as a shell escaping conflict, and how the recovery from such failures reveals deeper truths about the engineering process.

The Performance Optimization Arc

To understand message [msg 1994], we must first understand what led to it. The session had been a marathon of debugging and optimization. The GLM-5 model, quantized to GGUF Q4_K_XL format, had been successfully loaded onto eight RTX PRO 6000 Blackwell GPUs using vLLM, but initially produced completely incoherent garbage output ([msg 1977]). Two bugs were identified and fixed: an MLA attention custom op output buffer disconnect where a phantom tensor caused the output buffer to remain all zeros, and a GGUF shard ordering bug where fused projection layers had their KV and Q columns swapped ([msg 1978]).

With correct output achieved, the user's next demand was clear: 20 tok/s was too slow. The target was at least 100 tok/s ([msg 1984]). The assistant profiled the single-request decode performance and found that 42% of the 50.5ms per decode step was consumed by NCCL allreduce overhead — 157 separate allreduce calls totaling 21ms ([msg 1986]). The obvious solution was CUDAGraph, which batches kernel launches and eliminates the per-call overhead. But initial attempts produced garbage output again ([msg 1979]).

A subagent investigation concluded that the "phantom tensor" diagnosis was incorrect and reverted the MLA attention code back to the upstream custom op path ([msg 1991]). The assistant then hypothesized that the shard ordering fix alone might have been the true root cause of the garbage output, and that the direct-call patch had been unnecessary ([msg 1992]). Testing confirmed this: with the upstream custom op path restored and CUDAGraph enabled, the model produced correct output at 43 tok/s — a 2.15x improvement ([msg 1993]).

The Benchmark Attempt

Message [msg 1994] is the assistant's attempt to capitalize on this success. The todo list had been updated: "Run full benchmark suite with CUDAGraph" was now the active task. The assistant needed to quantify the improvement across multiple concurrency levels — 1, 2, 10, 64, 256, and 1024 concurrent requests — to produce a comprehensive performance report.

The approach was to execute a Python benchmark script inline via SSH, using the same pattern that had worked successfully in previous benchmarks ([msg 1981]). The script would send completion requests to the vLLM server running on the remote machine, measure latencies, and print formatted results. The assistant constructed a Python one-liner with f-strings for formatted output:

print(f'C={conc:4d} | {len(ok):3d}/{num_req:3d} ok ({errs} err) | wall={wall:6.1f}s | {tot_out/wall:7.1f} tok/s | lat avg={statistics.mean(lats):6.2f} p50={lats[len(lats)//2]:6.2f} p99={lats[int(len(lats)*0.99)]:6.2f}')

This f-string contains curly braces with format specifiers like {conc:4d}, {tot_out/wall:7.1f}, and crucially, the literal text tok/s. The entire Python script was wrapped in a shell string passed to ssh:

ssh root@10.1.230.174 '/root/ml-env/bin/python3 -c "..."'

The Failure: zsh Brace Expansion

The shell on the local machine was zsh, not bash. Zsh has a feature called brace expansion that interprets curly braces in certain contexts. When zsh encountered the f-string containing {tot_out/wall:7.1f}, it attempted to expand the brace expression {tot_out/wall:7.1f} as a brace expansion, which produced a parse error because the syntax didn't match valid brace expansion patterns.

The error message zsh:35: parse error near 'tok/s' indicates that zsh reached line 35 of the command (the complex print statement) and failed to parse the expression. The tok/s text, which was meant to be a literal unit string inside the Python f-string, became the point of failure because the preceding brace expansion had already mangled the command structure.

This is a classic shell escaping problem. The assistant had successfully used similar inline Python scripts earlier ([msg 1981]), but those scripts used simpler print statements without nested f-string format specifiers. The earlier benchmark script used basic print(f"Throughput: ...") without the complex column-aligned formatting that required curly braces with colons and format specifiers. The new script's formatting was more ambitious — it aimed for a clean, tabular output — and that ambition triggered the shell escaping issue.

Assumptions and Their Consequences

The assistant made several assumptions in crafting this message:

Assumption 1: The inline SSH pattern would work as before. This was the most consequential assumption. The assistant had executed Python scripts via SSH multiple times successfully. But each previous instance used simpler formatting. The assistant did not consider that the more complex f-string would interact differently with the shell.

Assumption 2: The shell would pass the Python code unchanged. The assistant assumed that wrapping the command in single quotes would prevent any shell interpretation. However, the nested quoting structure — single quotes around the entire command, double quotes around the Python code, and curly braces inside f-strings — created a complex escaping environment that zsh interpreted differently than expected.

Assumption 3: The benchmark would succeed on the first attempt. The assistant was in an eager, forward-moving state. The CUDAGraph optimization had just been validated, and the natural next step was to run the full benchmark suite. There was no testing of the script locally or in a simplified form before the full deployment.

Assumption 4: The remote server's shell behavior matched the local environment. The assistant was running SSH from a local machine (likely the development container) where zsh was the default shell. The remote server might have used bash, but the local zsh was intercepting the command before it was even sent.

Mistakes and Incorrect Assumptions

The primary mistake was not anticipating the shell escaping conflict. The assistant had the knowledge that zsh brace expansion could interfere with curly braces — this is a well-known issue when using Python f-strings or format strings in zsh. But in the momentum of the optimization work, this detail was overlooked.

A secondary mistake was the lack of incremental testing. The assistant could have tested a simplified version of the print statement first, or written the script to a file on the remote server before executing it. The rush to get benchmark results led to a shortcut that backfired.

However, it's important to note that this mistake was not a failure of reasoning about the model or the optimization. The assistant's understanding of the performance characteristics, the CUDAGraph mechanism, and the benchmark methodology was correct. The failure was purely in the execution layer — a tool-use error rather than a conceptual error.

Input Knowledge Required

Understanding this message requires substantial context:

  1. The GLM-5 model architecture: A 744B Mixture-of-Experts model with Multi-head Latent Attention (MLA), which requires specific attention backend support in vLLM.
  2. GGUF quantization: The model was quantized to Q4_K_XL format, a 4-bit block quantization scheme that requires dequantization during inference.
  3. vLLM deployment stack: The serving infrastructure, including tensor parallelism across 8 GPUs, the Triton MLA attention backend, and CUDAGraph for kernel launch optimization.
  4. The debugging history: The two bugs that were fixed (MLA custom op output buffer disconnect and GGUF shard ordering) and the realization that the shard ordering fix was the true root cause.
  5. NCCL allreduce overhead: The profiling result showing 42% of decode time spent on 157 allreduce calls, which CUDAGraph eliminates by batching.
  6. Shell escaping: The interaction between zsh brace expansion, Python f-strings, and SSH command construction.
  7. The benchmark methodology: Using concurrent HTTP requests to measure throughput and latency at different concurrency levels.

Output Knowledge Created

Although the benchmark script failed to execute, the message produced valuable knowledge:

  1. The inline SSH pattern has limits: Complex Python scripts with f-strings containing format specifiers cannot be reliably executed inline through zsh. This is a practical engineering lesson that directly informed the next step.
  2. The need for script files: The failure led directly to message [msg 1995], where the assistant wrote the benchmark to a file (benchmark.py) and executed it cleanly. This is a more robust pattern for complex scripts.
  3. Documentation of the attempted approach: The message serves as a record of what was tried and why it failed, which is valuable for debugging and for understanding the session's timeline.
  4. Confirmation of the optimization state: Even in failure, the message confirms that the assistant had achieved CUDAGraph working at 43 tok/s and was proceeding to benchmark — a positive state that was obscured by the execution error.

The Thinking Process Visible

The assistant's reasoning is visible in the structure of the benchmark script itself. The choice of concurrency levels (1, 2, 10, 64, 256, 1024) reveals a systematic approach to performance characterization. The warmup phase (two requests before measurement) shows awareness of cold-start effects. The inclusion of P50 and P99 latency metrics indicates a sophisticated understanding of tail latency behavior.

The formatting of the output — a tabular display with column headers and aligned data — shows the assistant's intention to produce a clean, readable benchmark report. This attention to presentation quality is characteristic of the assistant's thorough approach throughout the session.

The error itself reveals something about the assistant's cognitive state: it was operating in a "flow" mode, executing the next logical step without pausing to verify the tool invocation would work. This is a common pattern in complex problem-solving sessions where momentum builds and small procedural checks are skipped.

Resolution and Aftermath

The immediate aftermath of message [msg 1994] was message [msg 1995], where the assistant immediately recognized the problem: "The f-string with curly braces is fighting zsh. Let me write a script file instead." The assistant then wrote benchmark.py to the filesystem and executed it via SSH as a file, bypassing the shell escaping issue entirely.

The benchmark results, when they eventually came, showed that CUDAGraph had indeed improved throughput, and further tuning with NCCL_PROTO=LL pushed single-request decode to ~57 tok/s. The 100 tok/s target remained elusive due to the PCIe hardware bottleneck — a fundamental constraint that no amount of software optimization could fully overcome.

Conclusion

Message [msg 1994] is a microcosm of the engineering process: a moment of success followed immediately by a mundane failure, a quick diagnosis, and an adaptive response. The assistant's attempt to benchmark the CUDAGraph optimization was correct in intent and methodology, but failed in execution due to a shell escaping issue. The failure was not a conceptual error but a tool-use error — a reminder that even the most sophisticated AI systems must contend with the gritty realities of command-line interfaces and shell parsing.

The message also illustrates an important principle: the most valuable failures are those that teach us something about our tools and processes. The shell escaping failure taught the assistant (and by extension, the reader) that inline Python scripts with complex f-strings are unreliable through zsh SSH commands, and that writing script files is a more robust approach. This is a small but practical lesson that improves future engineering work.

In the broader arc of the session, this message marks the transition from debugging to benchmarking — from "does it work?" to "how fast is it?" The failure was a speed bump, not a roadblock. Within one message, the assistant had diagnosed the problem and pivoted to a working solution. That adaptability is the hallmark of effective engineering, whether human or AI.