When Shell Met Python: A Cautionary Tale of SSH, Zsh, and f-Strings

Introduction

In the high-stakes world of machine learning infrastructure optimization, even the most carefully planned diagnostic procedure can be derailed by a seemingly trivial technical detail. Message 1284 of this opencode session captures exactly such a moment: an assistant, having just completed a comprehensive system audit and applied nine critical runtime fixes to a Proxmox host running eight NVIDIA RTX PRO 6000 Blackwell GPUs, attempts to verify the changes by running a GPU peer-to-peer (P2P) bandwidth micro-benchmark. The command fails not because of hardware issues, kernel problems, or CUDA errors, but because of a mundane shell escaping problem: zsh:1: bad pattern.

This article examines this single message in depth, exploring the reasoning behind it, the assumptions that led to the failure, the technical knowledge required to understand what went wrong, and the broader lessons it offers about the fragility of command-line tooling in complex distributed systems.

The Message in Full

The subject message reads as follows:

[assistant] Now let me run P2P micro-benchmarks to verify everything is working correctly after the MaxReadReq change:
[bash] ssh root@10.1.230.174 'source /root/ml-env/bin/activate && python3 -c "
import torch
import time

def measure_p2p(src, dst, size_mb=256, iters=50):
    \"\"\"Measure P2P copy bandwidth with explicit stream sync.\"\"\"
    numel = size_mb * 1024 * 256  # 256 floats per KB
    src_t = torch.randn(numel, device=f\"cuda:{src}\")
    dst_t = torch.empty(numel, device=f\"cuda:{dst}\")
    
    stream = torch.cuda.Stream(device=f\"cuda:{src}\")
    
    # warmup
    with torch.cuda.stream(stream):
        for _ in range(5):
            dst_t.copy_(src_t, non_blocking=True)
    stream.synchronize()
    
    # timed
    start_evt = torch.cuda.Event(enable_timing=True)
    end_evt = torch.cuda.Event(enable_timing=True)
    
    start_evt.record(stream)
    with torch.cuda.stream(stream):
        for _ in range(iters):
            dst_t.copy_(src_t, non_blocking=True)
    end_evt.record(stream)
    end_evt.synchronize()
    
    elapsed_ms = start_evt.elapsed_time(end_evt)
    bw = size_mb * iters / (elapsed_ms / 1000) / 1024  # GB/s
    
    del src_t, dst_t
    torch.cuda.empty_cache()
    return bw, elapsed_ms / iters

print(\"GPU P2P Bandwidth Test (256MB transfers, CUDA events timing)\")
print(\"=\" * 65)
print(f\"{'Pair':>12} | {'BW (GB/s)':>10} | {'Latency (ms)':>12} | {'Type':>10}\")
print(\"-\" * 65)

# Same-NUMA pairs
for s, d in [(0,1), (1,2), (2,3), (0,3)]:
    bw, lat = measure_p2p(s, d)
    print(f\"GPU{s}->GPU{d}    | {bw:>10.1f} | {lat:>12.2f} | {'SAME-NUMA':>10}\")

# Cross-NUMA pairs
for s, d in [(0,4), (0,7), (3,4), (3,7)]:
    bw, lat = measure_p2p(s, d)
    print(f\"GPU{s}->GPU{d}    | {bw:>10.1f} | {lat:>12.2f} | {'CROSS-NUMA':>10}\")

# Same-NUMA node 1
for s, d in [(4,5), (5,6), (6,7)]:
    bw, lat = measure_p2p(s, d)
    print(f\"GPU{s}->GPU{d}    | {bw:>10.1f} | {lat:>12.2f} | {'SAME-NUMA':>10}\")

print()
print(\"Host-to-GPU and GPU-to-Host:\")
print(\"-\" * 65)
# H2D and D2H
for gpu in [0, 4]:
    # H2D
    size_mb = 256
    numel = size_mb * 1024 * 256
    h = torch.randn(numel, pin_memory=True)
    d = torch.empty(numel, device=f\"cuda:{gpu}\")
    stream = torch.cuda.Stream(device=f\"cuda:{gpu}\")
    # warmup
    with torch.cuda.stream(stream):
        for _ in range(3):
            d.copy_(h, non_blocking=True)
    stream.synchronize()
    s_evt = torch.cuda.Event(enable_timing=True)
    e_evt = torch.cuda.Event(enable_timing=True)
    s_evt.record(stream)
    with torch.cuda.stream(stream):
        for _ in range(50):
            d.copy_(h, non_blocking=True)
    e_evt.record(stream)
    e_evt.synchronize()
    elapsed_ms = s_evt.elapsed_time(e_evt)
    bw = size_mb * 50 / (elapsed_ms / 1000) / 1024
    print(f\"Host->GPU{gpu}   | {bw:>10.1f} | {elapsed_ms/50:>12.2f} | {'H2D':>10}\")
    del h, d
    
    # D2H
    d = torch.randn(numel, device=f\"cuda:{gpu}\")
    h = torch.empty(numel, pin_memory=True)
    stream = torch.cuda.Stream(device=f\"cuda:{gpu}\")
    with torch.cuda.stream(stream):
        for _ in range(3):
            h.copy_(d, non_blocking=True)
    stream.synchronize()
    s_evt = torch.cuda.Event(enable_timing=True)
    e_evt = torch.cuda.Event(enable_timing=True)
    s_evt.record(stream)
    with torch.cuda.stream(stream):
        for _ in range(50):
            h.copy_(d, non_blocking=True)
    e_evt.record(stream)
    e_evt.synchronize()
    elapsed_ms = s_evt.elapsed_time(e_evt)
    bw = size_mb * 50 / (elapsed_ms / 1000) / 1024
    print(f\"GPU{gpu}->Host   | {bw:>10.1f} | {elapsed_ms/50:>12.2f} | {'D2H':>10}\")
    del h, d
    torch.cuda.empty_cache()
" 2>&1'
zsh:1: bad pattern

The error output is terse: zsh:1: bad pattern. The Python script never executed. The entire diagnostic effort collapsed at the first hurdle — not because of any flaw in the measurement logic, but because the shell that received the command could not parse it.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the chain of events that led to it. The session had been engaged in an intensive, multi-hour optimization effort for the GLM-5-NVFP4 model running on SGLang across eight Blackwell GPUs. The assistant had computed a theoretical maximum single-stream performance of 309 tok/s, but actual performance was a mere 10.36 tok/s — a staggering 3.4% efficiency gap that demanded investigation.

In response, the assistant launched a parallel system audit via 10 agents ([msg 1266] through [msg 1271]), each investigating a different aspect of the system: PCIe configuration, power management, NCCL bandwidth, memory/NUMA allocation, sglang runtime configuration, and NVLink status. The consolidated findings ([msg 1273]) revealed multiple misconfigurations: a suboptimal CPU governor (acpi-cpufreq instead of amd_pstate), an outdated kernel, enabled NUMA balancing, deep CPU C-states, and a PCIe MaxReadReq stuck at 512 bytes instead of the optimal 4096.

The user then directed the assistant to "write down in system-improve.md, apply the changes runtime only for now, check that p2p works with some manual micro bench, then rerun inference benchmark" ([msg 1274]). The assistant followed this instruction precisely: it wrote the documentation ([msg 1276]), applied all nine runtime fixes ([msg 1279]), verified them ([msg 1280]), and then — at message 1284 — attempted to "check that p2p works" by running a micro-benchmark.

The message is thus the third step in a four-step plan: document, apply, verify, benchmark. It represents the verification phase, where the assistant seeks to confirm that the MaxReadReq change and other kernel tweaks have actually improved GPU-to-GPU communication before proceeding to the full inference benchmark suite.## The Reasoning Behind the Benchmark Design

The Python script embedded in the SSH command is not arbitrary — it reflects careful thought about what needs to be measured and how to measure it accurately. The assistant is running a P2P bandwidth test using 256 MB transfers with CUDA event timing, which is a well-established methodology for GPU communication benchmarking.

Several design decisions are visible in the code:

Explicit stream synchronization: The script creates a dedicated CUDA stream and uses torch.cuda.Event objects for precise timing. This avoids the "lazy synchronization" problem that had plagued earlier measurements — in [msg 1268], the assistant discovered that earlier P2P tests had produced artifacts like "~20 TB/s" because of missing stream synchronization. The current script explicitly calls end_evt.synchronize() to ensure all operations complete before reading timestamps.

Warmup iterations: The script runs 5 warmup iterations before the timed section, which is critical for GPU benchmarking. The first few kernel launches may include driver initialization overhead, CUDA context establishment, and memory allocation that would distort measurements. By warming up, the timed section measures only steady-state performance.

Multiple pair types: The test covers same-NUMA pairs (GPUs 0→1, 1→2, 2→3, 4→5, 5→6, 6→7), cross-NUMA pairs (0→4, 0→7, 3→4, 3→7), and host-device transfers (H2D and D2H). This comprehensive coverage reflects an understanding that NUMA topology affects bandwidth — same-NUMA transfers should be faster because they stay within the same CPU socket's PCIe hierarchy, while cross-NUMA transfers must traverse the inter-socket interconnect.

256 MB transfer size: This is large enough to saturate the PCIe link and measure sustained bandwidth rather than latency-bound small transfers. With PCIe Gen5 x16 offering ~64 GB/s theoretical bandwidth, a 256 MB transfer takes several milliseconds, giving the CUDA event timer sufficient resolution.

The script also includes proper cleanup with del src_t, dst_t and torch.cuda.empty_cache(), preventing GPU memory fragmentation across iterations. This attention to detail suggests the assistant has experience with GPU benchmarking pitfalls.

The Critical Failure: Shell Escaping

Despite the careful benchmark design, the command fails with zsh:1: bad pattern. The error originates from the shell that processes the command — in this case, Zsh on the remote machine (the LXC container at 10.1.230.174). The problem lies in how the Python f-string expressions interact with Zsh's pattern matching.

Consider this line from the script:

print(f\"{'Pair':>12} | {'BW (GB/s)':>10} | {'Latency (ms)':>12} | {'Type':>10}\")

When this Python code is embedded inside a double-quoted SSH command string, the curly braces { and } are interpreted by the shell before Python ever sees them. In Zsh, curly braces are used for brace expansion and parameter expansion. The expression {'Pair':>12} looks like a glob pattern or brace expansion to Zsh, and when it doesn't match valid syntax, Zsh reports a "bad pattern" error.

The assistant attempted to escape the Python string by using \" (escaped double quotes) throughout, but this only protects the double-quote characters from the outer shell quoting — it does nothing to protect the curly braces. The fundamental issue is that the entire Python script is passed as a single-quoted argument to SSH, but the outer quoting is actually double quotes (the command starts with ssh root@... 'source ... && python3 -c "). The Python code is inside double quotes, which means Zsh will expand {...} expressions.

The irony is that the assistant had successfully run similar Python scripts via SSH in earlier messages ([msg 1268], [msg 1270]), but those scripts did not contain f-strings with curly brace format specifiers. The introduction of f-string formatting — specifically the {...:>12} alignment syntax — introduced curly braces that clash with shell syntax.

Assumptions Made by the Assistant

The message reveals several assumptions, most of which turned out to be incorrect:

Assumption 1: The quoting would work as intended. The assistant assumed that wrapping the command in single quotes after ssh would protect the entire Python script from shell interpretation. However, the command structure is ssh host 'source ... && python3 -c "..."', where the outer quotes are single quotes but the Python code itself is inside double quotes within those single quotes. The double quotes are what cause the problem — Zsh still processes the content inside double quotes for certain expansions.

Assumption 2: The remote shell is Bash. The error message zsh:1: bad pattern reveals that the remote system uses Zsh as the default shell, not Bash. This is significant because Zsh has different quoting and expansion rules than Bash. In Bash, the same command might have worked (or at least produced a different error), because Bash's brace expansion has different syntax requirements. The assistant had been running commands on this container throughout the session without issue, but those earlier commands didn't contain curly braces in the Python code.

Assumption 3: The MaxReadReq change was the only variable that mattered. The assistant's focus on verifying the P2P bandwidth after the MaxReadReq change is logical — increasing MaxReadReq from 512 to 4096 bytes should improve PCIe read efficiency by allowing larger read requests per transaction. However, the verification step was blocked before it could begin, so the actual impact of the change remains unmeasured in this message.

Assumption 4: Inline Python via SSH is a reliable pattern. The assistant had used this pattern many times before in the session — embedding Python scripts inside SSH commands for quick diagnostics. Each time, the script was simple enough to avoid shell escaping issues. This time, the complexity of f-strings with format specifiers crossed the threshold where shell escaping becomes problematic.## Input Knowledge Required to Understand This Message

A reader needs substantial technical background to fully grasp what this message is doing and why it fails:

GPU topology and NUMA awareness: The script tests same-NUMA and cross-NUMA pairs, which requires understanding that the eight GPUs are split across two CPU sockets (NUMA nodes). GPUs 0-3 are on NUMA node 0, GPUs 4-7 on NUMA node 1. Cross-NUMA transfers go through the inter-socket interconnect (likely AMD Infinity Fabric), which has lower bandwidth than same-NUMA transfers that stay within a single PCIe root complex.

CUDA stream semantics: The use of torch.cuda.Stream and non-blocking copies reflects an understanding that CUDA operations are asynchronous by default. Without explicit stream synchronization, timing measurements can be meaningless — the end_evt.synchronize() call ensures all queued operations have completed before reading the elapsed time.

PCIe MaxReadReq and its impact: MaxReadReq (Maximum Read Request Size) controls the largest single read transaction a PCIe device can initiate. Increasing it from 512 to 4096 bytes allows the GPU to request more data per read transaction, reducing the number of round trips needed to transfer a given amount of data. For GPU P2P transfers, this can significantly improve bandwidth utilization.

Shell quoting and escaping: Understanding why the command fails requires knowledge of how SSH passes commands to remote shells, the difference between single and double quotes in shell syntax, and how Zsh handles curly braces differently from Bash.

Python f-string syntax: The f\"{'Pair':>12}\" syntax uses Python's formatted string literals (f-strings) with format specifiers. The :>12 means "right-align in a field of width 12." The nested single quotes inside the f-string are a common pattern for including literal strings in format expressions.

Output Knowledge Created by This Message

Although the benchmark failed to execute, the message still produces valuable knowledge:

A failed verification attempt is documented: The fact that the assistant tried and failed to run the P2P benchmark is recorded, which is useful for understanding the session's timeline and debugging approach. A human operator reading the logs would immediately see that the verification step was blocked and could intervene.

The shell environment is now known: The error zsh:1: bad pattern definitively reveals that the remote container uses Zsh as its default shell. This is information that was not explicitly known before — earlier commands succeeded because they didn't trigger Zsh's pattern matching. This knowledge can inform future command construction.

The benchmark script itself is preserved: Even though it didn't execute, the Python code represents a well-designed P2P measurement tool. It can be saved to a file and run directly, bypassing the shell escaping issue entirely. The script's structure — warmup, CUDA event timing, multiple pair types, proper cleanup — serves as a template for future diagnostic tools.

The MaxReadReq change is confirmed applied: While the P2P benchmark failed, the earlier verification ([msg 1280]) confirmed that the MaxReadReq was successfully changed to 4096 bytes on all GPUs. The verification step in this message was intended to measure the effect of that change, not to confirm the change itself.

The Thinking Process Visible in the Message

The assistant's reasoning is evident in the structure and content of the command:

Step 1: Recognize the need for verification. After applying nine runtime fixes, the assistant correctly identifies that changes should be verified before proceeding. The user explicitly requested "check that p2p works with some manual micro bench," so this step is not optional.

Step 2: Design an appropriate benchmark. The assistant chooses a P2P bandwidth test because the most impactful change was the MaxReadReq increase, which directly affects PCIe read performance. A 256 MB transfer size is selected to saturate the link and measure sustained bandwidth. CUDA event timing is chosen for precision.

Step 3: Cover all relevant topologies. The benchmark includes same-NUMA, cross-NUMA, and host-device transfers to provide a complete picture. This reflects an understanding that different communication paths have different bandwidth characteristics and that the MaxReadReq change might affect them differently.

Step 4: Execute via SSH. The assistant uses the same pattern that worked for earlier diagnostics — inline Python via SSH — without considering that the f-string syntax introduces new escaping challenges.

Step 5: Observe the failure. The error zsh:1: bad pattern is returned, and the message ends there. In the subsequent messages (not shown in this article), the assistant would need to recognize the quoting issue and adapt its approach.

The failure mode is instructive: the assistant's thinking was correct about what to measure and how to measure it, but incorrect about how to deliver the measurement code to the target system. This is a classic case of the right answer to the wrong problem.

Mistakes and Incorrect Assumptions

Several specific mistakes contributed to the failure:

The quoting strategy was fundamentally flawed. The command uses this structure:

ssh host 'source ... && python3 -c "..."

The outer single quotes protect most of the command, but the Python code is inside double quotes within those single quotes. In shell parsing, double-quoted sections still undergo parameter expansion, command substitution, and — in Zsh — brace expansion. The curly braces in the f-strings are thus interpreted by the shell.

The assistant did not account for Zsh as the remote shell. Earlier commands in the session had used simpler Python scripts without f-string formatting, so the Zsh shell was never triggered to reveal itself. The assistant implicitly assumed Bash, which is the default shell on most Linux systems. However, the LXC container had been configured with Zsh as the default shell for the root user, and this configuration detail was not accounted for.

No escaping of curly braces was attempted. Even within double quotes, curly braces can be escaped with backslashes in most shells. The assistant could have written \{'Pair':>12\} to prevent shell expansion, but did not. Alternatively, the entire Python script could have been written to a file on the remote system and executed from there, bypassing shell escaping entirely.

The benchmark was unnecessarily complex for inline execution. The f-string format specifiers (:>12, :>10.1f, etc.) are convenient for aligned output but introduce shell-hostile syntax. A simpler approach — using % formatting or str.format() instead of f-strings — would have avoided the curly brace issue entirely while producing identical output.

Broader Lessons

This message illustrates a fundamental tension in infrastructure automation: the convenience of inline code execution versus the reliability of file-based workflows. Inline Python via SSH is fast and requires no cleanup — you type the command, it runs, and you get output. But as scripts grow more complex, the escaping overhead grows non-linearly. What starts as a simple one-liner becomes a quoting nightmare when f-strings, nested quotes, or special characters are introduced.

The alternative — writing the script to a file on the remote system, then executing it — is more robust but requires additional steps: creating the file, ensuring it has proper permissions, cleaning up afterward. In a fast-paced debugging session where the assistant is iterating rapidly, the inline approach feels more natural. But this message demonstrates the cost of that convenience.

There is also a lesson about testing assumptions about the remote environment. The assistant had successfully run dozens of SSH commands in this session without issue, leading to the implicit assumption that "this works." But each command is different, and the introduction of new syntax elements (f-string curly braces) created a failure mode that previous commands had not exercised. A more defensive approach would be to test the quoting with a simple echo command first, or to check the remote shell type before constructing complex commands.

Conclusion

Message 1284 is a snapshot of a precise moment in a complex optimization workflow: the transition from applying fixes to verifying their impact. The assistant's benchmark design is sound, its understanding of GPU communication topology is thorough, and its methodology for measuring P2P bandwidth is rigorous. Yet all of this is rendered moot by a shell escaping issue — a reminder that in distributed systems, the delivery mechanism is as important as the payload.

The message also reveals the assistant's thinking process in microcosm: a logical progression from "changes applied" to "verify changes" to "measure impact," executed with appropriate technical depth at each step. The failure is not one of reasoning but of tooling — a mismatch between the complexity of the Python code and the limitations of inline SSH execution.

For anyone working with GPU clusters, this message offers a valuable caution: when your diagnostic scripts grow beyond simple one-liners, write them to files. The few extra seconds spent creating a script file are a small price to pay for avoiding the silent failure of a shell that cannot parse your carefully crafted benchmark.