When Shell Met Python: Debugging a Zsh Escaping Nightmare in GPU Benchmarking

Subject Message (index 1285): ``` [assistant] Shell escaping issue with zsh. Let me write it as a file: [bash] ssh root@10.1.230.174 'cat > /tmp/p2p_bench.py << '\''PYEOF'\'' import torch import time

>

def measure_p2p(src, dst, size_mb=256, iters=50): numel = size_mb 1024 256 src_t = torch.randn(numel, device=f"cuda:{src}") dst_t = torch.empty(numel, device=f"cuda:{dst}") stream = torch.cuda.Stream(device=f"cuda:{src}") with torch.cuda.stream(stream): for _ in range(5): dst_t.copy_(src_t, non_blocking=True) stream.synchronize() s_evt = torch.cuda.Event(enable_timing=True) e_evt = torch.cuda.Event(enable_timing=True) s_evt.record(stream) with torch.cuda.stream(stream): for _ in range(iters): dst_t.copy_(src_t, non_blocking=True) e_evt.record(stream) e_evt.synchronize() elapsed_ms = s_evt.elapsed_time(e_evt) bw = size_mb * iters / (elapsed_ms / 1000) / 1024 del src_t, dst_t torch.cuda.empty_cache() return bw, elapsed_ms / iters

>

def measure_h2d(gpu, size_mb=256, iters=50): numel = size_mb 1024 256 h = torch.randn(numel, pin_memory=True) d = torch.empty(numel, device=f"cuda:{gpu}") stream = torch.cuda.Stream(device=f"cuda:{gpu}") with torch.cuda.stream(stream): for _ in range(3): d.copy_(h, non_blocking=True) stream.synchronize() s_evt = torch.cuda.Event(enable_timing=True) e_evt = torch.cuda.Event(enable_timing=True) s_evt.record(stream) with torch.cuda.stream(stream): for _ in range(iters): d.copy_(h, non_blocking=True) e_evt.record(stream) e_evt.synchronize() elapsed_ms = s_evt.elapsed_time(e_evt) bw = size_mb * iters / (elapsed_ms / 1000) / 1024 del h, d torch.cuda.empty_cache() return bw, elapsed_ms / iters

>

def measure_d2h(gpu, size_mb=256, iters=50): numel = size_mb 1024 256 d = torch.randn(numel, device=f"cuda:{gpu}") h = torch.empty(numel, pin_memory=True) stream = torch.cuda.Stream(device=f"cuda:{gpu}") with torch.cuda.stream(stream): for _ in range(3): h.copy_(d, non_blocking=True) stream.synchronize() s_evt = torch.cuda.Event(enable_timing=True) e_evt = torch.cuda.Event(enable_timing=True) s_evt.record(stream) with torch.cuda.stream(stream): for _ in range(iters): h.copy_(d, non_blocking=True) e_evt.record(stream) e_evt.synchronize() elapsed_ms = s_evt.elapsed_time(e_evt) bw = size_mb * iters / (elapsed_ms / 1000) / 1024 del h, d torch.cuda.empty_cache() return bw, elapsed_ms / iters

>

print("GPU P2P Bandwidth Test (256MB transfers, CUDA events timing)") print("After system tuning: MaxReadReq=4096, NUMA balancing off, C2 off") print("=" 65) print(f"{'Pair':<14} {'BW (GB/s)':>10} {'Lat (ms)':>10} {'Type':>12}") print("-" 65)

>

for s, d in [(0,1), (1,2), (2,3), (0,3)]: bw, lat = measure_p2p(s, d) print(f"GPU{s}->GPU{d} {bw:>10.1f} {lat:>10.2f} {'SAME-NUMA':>12}")

>

for s, d in [(0,4), (0,7), (3,4), (3,7)]: bw, lat = measure_p2p(s, d) print(f"GPU{s}->GPU{d} {bw:>10.1f} {lat:>10.2f} {'CROSS-NUMA':>12}")

>

for s, d in [(4,5), (5,6), (6,7)]: bw, lat = measure_p2p(s, d) print(f"GPU{s}->GPU{d} {bw:>10.1f} {lat:>10.2f} {'SAME-NUMA':>12}")

>

print() print("Host<->GPU transfers:") print("-" * 65) for gpu in [0, 4]: bw, lat = measure_h2d(gpu) print(f"Host->GPU{gpu} {bw:>10.1f} {lat:>10.2f} {'H2D':>12}") bw, lat = measure_d2h(gpu) print(f"GPU{gpu}->Host {bw:>10.1f} {lat:>10.2f} {'D2H':>12}") PYEOF ' zsh:1: bad pattern: (GB/s):>10} {Lat ```

Introduction

In the middle of an intensive performance-tuning session for a GLM-5-NVFP4 large language model running on eight RTX PRO 6000 Blackwell GPUs, a seemingly trivial task—running a GPU peer-to-peer (P2P) bandwidth benchmark—turned into a surprising debugging exercise at the intersection of shell parsing, Python string formatting, and remote command execution. The message at index 1285 captures this moment: the assistant, having just applied a series of critical system-level performance fixes on a Proxmox host, attempts to verify their effectiveness by running a CUDA P2P benchmark on a remote LXC container, only to be foiled by zsh's aggressive interpretation of parentheses in Python f-string format specifiers.

This article examines that single message in depth: why it was written, the reasoning behind the approach taken, the assumptions made, the mistakes encountered, and the broader context that makes this small episode a revealing window into the realities of high-performance ML infrastructure work.

Context: The Performance Tuning Campaign

To understand why this message exists, one must appreciate the broader effort underway. The session was part of an intensive optimization campaign for the GLM-5-NVFP4 model, a large Mixture-of-Experts (MoE) transformer that demands extraordinary computational resources. The team had already worked through numerous rounds of tuning: upgrading kernels, adjusting CUDA configurations, experimenting with expert parallelism, and benchmarking various inference server parameters. Despite achieving throughputs of several thousand tokens per second at high concurrency, single-stream performance remained stubbornly low—around 10.36 tok/s against a theoretical maximum of 309 tok/s, representing a staggering 3.4% efficiency gap.

To close this gap, a comprehensive 10-agent parallel system audit had been launched in the preceding messages ([msg 1273]). This audit uncovered a laundry list of misconfigurations: a suboptimal CPU scaling driver (acpi-cpufreq instead of amd_pstate), an outdated kernel, enabled NUMA balancing, deep CPU C-states, and critically, a PCIe MaxReadReq stuck at 512 bytes instead of the optimal 4096. The assistant had documented these findings in system-improve.md ([msg 1276]) and then proceeded to apply all runtime fixes that didn't require a reboot ([msg 1279]).

The fixes included: increasing PCIe MaxReadReq from 512 to 4096 bytes on all eight GPUs and their root ports, disabling NUMA balancing, disabling CPU C2 deep sleep states, turning off the NMI watchdog, disabling sched_autogroup, reducing swappiness and dirty page ratios, increasing network buffer sizes, and lowering the perf_event_paranoid setting. Nine out of ten fixes succeeded; the nvidia_peermem module failed to load, which was expected and harmless for this PCIe-only setup.

The Immediate Motivation: Verifying the Fixes

With the tuning applied, the next logical step was verification. The assistant needed to confirm that the changes hadn't broken anything and, ideally, that they had improved performance. The most direct way to test this was to run a GPU P2P bandwidth benchmark—a micro-benchmark that measures how fast data can be copied between GPU memory across the PCIe fabric. This benchmark would reveal whether the MaxReadReq increase had any effect on GPU-to-GPU transfer speeds, and whether NUMA balancing and C-state changes had impacted latency.

Message 1284 represents the first attempt: the assistant crafted an inline Python script using python3 -c &#34;...&#34; passed through SSH, complete with CUDA stream management, event timing, and formatted output using Python f-strings. The script was sophisticated—it measured P2P bandwidth for same-NUMA and cross-NUMA GPU pairs, host-to-GPU (H2D) transfers, and GPU-to-host (D2H) transfers, all using proper CUDA event timing for accuracy. But when executed, it failed with a cryptic error:

zsh:1: bad pattern: (GB/s):>10} | {Latency

The Subject Message: Diagnosis and First Workaround Attempt

Message 1285 is the assistant's response to this failure. The assistant correctly identifies the root cause: "Shell escaping issue with zsh." The parentheses in the Python f-string format specifier {bw:&gt;10.1f} are being interpreted by zsh as globbing patterns before the Python interpreter ever sees them. Zsh, unlike bash, treats parentheses in certain contexts as pattern-matching syntax, and when it encounters (GB/s) within the command string, it attempts to expand it as a file glob—finding no matching files, it reports a "bad pattern" error.

The assistant's chosen workaround is to avoid inline Python execution entirely and instead write the script to a file on the remote host using a heredoc:

ssh root@10.1.230.174 'cat > /tmp/p2p_bench.py << '\''PYEOF'\''
...
PYEOF
'

This approach has sound reasoning behind it. By writing the Python code to a file first and then executing it separately, the assistant hopes to bypass the shell escaping issues entirely. The heredoc delimiter PYEOF is quoted (using the clever &#39;\&#39;&#39;PYEOF&#39;\&#39;&#39; sequence to escape single quotes within a single-quoted SSH command), which should prevent variable expansion and command substitution within the heredoc body. The Python code itself is preserved verbatim from the previous attempt, complete with f-strings using the f&#34;...&#34; syntax with format specifiers like {bw:&gt;10.1f}.

However, this attempt also fails with the identical error:

zsh:1: bad pattern: (GB/s):>10} {Lat

Why the Heredoc Approach Also Failed

The failure reveals an important subtlety about how SSH and shell command parsing interact. When the assistant runs ssh root@10.1.230.174 &#39;cat &gt; /tmp/p2p_bench.py &lt;&lt; &#39;\&#39;&#39;PYEOF&#39;\&#39;&#39; ... PYEOF&#39;, the entire argument to ssh (including the heredoc) is passed as a single command string to the remote shell. But before the remote shell even receives it, the local shell—in this case, zsh on the Proxmox host—parses the command. The heredoc syntax &lt;&lt; &#39;PYEOF&#39; is shell syntax, not a standalone construct, and zsh processes the entire command string, including the Python code containing parentheses, before dispatching it to SSH.

The assistant's assumption was that quoting the heredoc delimiter would prevent all shell interpretation of the heredoc body. This is correct in principle—when a heredoc delimiter is quoted (e.g., &lt;&lt; &#39;EOF&#39;), the shell does not perform variable expansion, command substitution, or arithmetic expansion on the body. However, zsh's globbing behavior with parentheses is a different matter. Zsh's "bad pattern" error occurs during word expansion and pattern matching phases, which happen before or during the parsing of the heredoc structure itself. The parentheses in (GB/s) trigger zsh's extended globbing machinery, and the error is raised during the initial parsing pass, before the heredoc is even recognized as such.

In other words, the assistant correctly diagnosed the problem (zsh interpreting parentheses) but underestimated its depth. The issue isn't merely about escaping within the Python string—it's about the fundamental way zsh processes command arguments before dispatching them. Any approach that passes the Python code through zsh's command-line parser, whether inline with -c or via a heredoc argument to SSH, will encounter the same problem if parentheses remain unescaped.

The Broader Assumptions at Play

This message reveals several assumptions, some correct and some not:

Correct assumption: The assistant correctly assumed that the P2P benchmark was the right tool to verify the system tuning. Measuring GPU-to-GPU bandwidth directly tests the PCIe subsystem, and the CUDA event timing approach provides accurate, low-level measurements independent of the inference server.

Correct assumption: The assistant correctly identified that zsh was the culprit, not Python or CUDA. The error message "bad pattern" is a zsh-specific error, and the mention of (GB/s) in the error output points directly to the f-string format specifier as the trigger.

Incorrect assumption: The assistant assumed that moving from inline Python execution (python3 -c) to a heredoc-based file write would bypass the shell escaping issue. This assumption overlooked the fact that the heredoc is still passed as part of the SSH command string and must survive zsh's parsing on the local side before reaching the remote shell.

Incorrect assumption: The assistant assumed that quoting the heredoc delimiter (&#39;PYEOF&#39;) would protect the entire body from shell interpretation. While this protects against variable expansion and command substitution, it does not protect against zsh's glob/pattern matching phase, which operates on a different level of the parsing pipeline.

The Knowledge Required to Understand This Message

To fully grasp what's happening in message 1285, one needs knowledge spanning several domains:

Shell scripting and POSIX vs. zsh differences: Understanding that zsh has extended globbing features that differ from bash, and that parentheses have special meaning in zsh patterns. The bad pattern error is a zsh hallmark.

SSH command argument handling: Knowing that ssh host &#39;command&#39; passes the quoted string as a single argument to the remote shell, but the local shell still parses the entire command line before SSH receives it.

Heredoc mechanics: Understanding that heredocs are shell syntax, not standalone constructs, and that &lt;&lt; &#39;EOF&#39; prevents expansion but doesn't prevent the shell from parsing the content for syntax errors.

Python f-string formatting: Recognizing that f&#34;{value:&gt;10.1f}&#34; uses the format specifier syntax where &gt; means right-align, 10 is width, and .1f is float precision. The parentheses in format specifiers like (GB/s) are not special to Python—they're just literal characters in the format string—but they trigger zsh's pattern matching.

CUDA P2P benchmarking methodology: Understanding why the benchmark uses 256MB transfers, CUDA streams, event timing, warmup iterations, and non-blocking copies. These are standard practices for accurate GPU bandwidth measurement.

PCIe tuning concepts: Knowing what MaxReadReq is, why 4096 bytes is better than 512, and how it affects different types of PCIe transactions (reads vs. writes).

The Output Knowledge Created

Although the message itself ends in failure, it creates valuable knowledge:

  1. A documented failure mode: The interaction between zsh's parenthesis handling and Python f-string format specifiers is now captured. Anyone encountering zsh: bad pattern with Python code containing (something) can immediately recognize the issue.
  2. A diagnostic trace: The error output confirms that the problem is specifically with the format string (GB/s):&gt;10} | {Lat, which narrows the search space for debugging.
  3. A constraint on solution space: The failure of the heredoc approach tells us that the escaping issue must be solved at the zsh command-line level, not by changing how Python receives the code.
  4. The benchmark script itself: The Python code written in this message is a complete, production-quality P2P bandwidth benchmark. Even though it wasn't executed successfully here, the script's structure—CUDA stream management, event timing, warmup, multi-pair measurement—represents a reusable tool.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the transition from message 1284 to 1285, follows a clear problem-solving arc:

Step 1 — Symptom recognition: The error zsh:1: bad pattern: (GB/s):&gt;10} | {Latency is immediately recognized as a shell issue, not a Python issue. The assistant doesn't waste time questioning the Python code's validity.

Step 2 — Root cause diagnosis: The assistant correctly identifies "Shell escaping issue with zsh." The parentheses in the f-string format specifier are the likely trigger.

Step 3 — Workaround design: The assistant chooses to write the script to a file rather than inline, reasoning that a heredoc will avoid the escaping problems. This is a reasonable first attempt at a workaround, even if it ultimately fails.

Step 4 — Implementation: The assistant carefully constructs the SSH command with proper quoting of the heredoc delimiter, using the &#39;\&#39;&#39; sequence to embed a single quote within a single-quoted string—a classic shell quoting trick.

Step 5 — Failure observation: The same error recurs, confirming that the workaround was insufficient.

Step 6 — Adaptation (visible in the next message, msg 1286): The assistant pivots to a different approach: using scp /dev/stdin with a heredoc, and critically, rewriting the Python code to avoid f-strings entirely, using % formatting instead. This eliminates the parentheses from the format specifiers and solves the problem.

This thinking process exemplifies a pragmatic engineering approach: diagnose quickly, attempt the simplest fix, observe the result, and iterate. The assistant doesn't overanalyze the zsh parsing internals—it simply recognizes the constraint and works around it by changing the Python code's formatting style.

Conclusion

Message 1285 captures a small but instructive moment in a much larger performance optimization effort. It's a reminder that in complex ML infrastructure work, the most frustrating bottlenecks are often not the GPU kernels or the model architecture, but the mundane plumbing of shell escaping, remote execution, and tool interoperability. The assistant's attempt to verify system tuning via a P2P benchmark was thwarted not by CUDA or PCIe, but by a zsh globbing feature interacting with Python string formatting.

The message also illustrates a key characteristic of effective troubleshooting: the ability to rapidly identify the layer at which a problem occurs. The assistant correctly pinned the issue to the shell layer within seconds, avoiding a wild goose chase into Python or CUDA debugging. While the first workaround (heredoc) didn't succeed, it provided useful diagnostic information that informed the next iteration (using % formatting and scp), which did work.

In the end, the P2P benchmark ran successfully ([msg 1287]), confirming that the system tuning was functional: same-NUMA bandwidth at ~50 GB/s, cross-NUMA at ~37 GB/s, and H2D/D2H at ~53 GB/s. The MaxReadReq change didn't dramatically improve P2P bandwidth (as the assistant correctly noted, P2P copies are GPU-initiated writes governed by MaxPayload, not MaxReadReq), but the verification was done, and the team could move on to the next optimization round.

This episode serves as a case study in the hidden complexity of remote GPU benchmarking, the quirks of zsh, and the value of systematic debugging—even for what should be a simple "run a script" task.