When Shell Quoting Defeats AI: The SSH Heredoc Trap

In the midst of a complex machine learning deployment spanning multiple continents, GPU architectures, and inference engines, the most stubborn adversary turned out to be something far more mundane: shell quoting. Message [msg 7498] in this opencode session captures a brief but illuminating moment where the assistant—after an extended battle with SGLang server configuration, GPU memory allocation, and MTP speculative decoding—suddenly finds itself defeated not by CUDA OOM errors or incompatible attention backends, but by the humble Unix heredoc.

The Context: A Deployment Under Siege

To understand why this small message matters, we must first appreciate the battlefield. The assistant had been working for dozens of rounds to deploy Qwen3.6-27B with Multi-Token Prediction (MTP) speculative decoding on a 7× B200 NVL node. The goal was to generate 902,087 training completions for a DFlash drafter model—a massive data generation pipeline that required high-throughput inference.

The preceding messages ([msg 7479] through [msg 7496]) reveal a cascading series of failures. The assistant tried launching SGLang with --speculative-algorithm EAGLE and --mamba-scheduler-strategy extra_buffer, only to discover that the extra_buffer strategy doubled the Mamba state cache and caused out-of-memory errors. It tried capping the Mamba cache size with --max-mamba-cache-size 24 and reducing --mamba-full-memory-ratio 0.4. It tried different --mem-fraction-static values. It tried writing wrapper scripts. It tried debugging why nohup was silently failing over SSH.

Each attempt revealed a new layer of complexity. The no_buffer strategy was incompatible with speculative decoding's radix cache requirement. The extra_buffer strategy consumed too much GPU memory. The SSH session seemed to swallow commands without executing them. And through it all, the assistant was reasoning about memory budgets, Mamba state cache sizes per slot (roughly 143 MB), and the trade-offs between tensor parallelism across GPUs versus data parallelism.

The Message: A Moment of Meta-Realization

Message [msg 7498] is the turning point where the assistant stops debugging the ML stack and starts debugging its own communication channel. The full text reads:

## Agent Reasoning The heredoc didn't work properly with the SSH quoting. The script still has the old content (without extra_buffer and max-mamba-cache-size). The escaping is getting lost. Let me try using scp instead to upload the script. Heredoc quoting is getting mangled over SSH. Let me write locally and scp: [write] /data/dflash/scripts/launch_mtp.sh Wrote file successfully.

The trigger for this realization came in the previous message ([msg 7497]), where the assistant ran cat /workspace/dflash/scripts/launch_mtp.sh over SSH and discovered the script still contained the old content—the version without --mamba-scheduler-strategy extra_buffer, --max-mamba-cache-size 24, and --mamba-full-memory-ratio 0.4. Despite the assistant's careful attempt to write a new heredoc with these flags in [msg 7496], the file on the remote server remained unchanged.

Why the Heredoc Failed

The assistant's reasoning identifies the root cause correctly: "The escaping is getting lost." But the actual problem is more subtle and instructive. Looking at [msg 7496], the assistant constructed a complex SSH command:

ssh -p 19248 root@154.59.156.20 "pkill -9 -f sglang 2>/dev/null; sleep 2; cat > /workspace/dflash/scripts/launch_mtp.sh << 'EOF'
#!/bin/bash
...
EOF
chmod +x /workspace/dflash/scripts/launch_mtp.sh && echo ok" 2>&1

This is a nested quoting nightmare. The entire SSH command is wrapped in double quotes ("..."), which means the shell on the local machine interprets $1, $2, and any other shell variables before sending the command to the remote host. The assistant tried to escape this with \$1 and \$2, but the heredoc delimiter 'EOF'—which should prevent variable expansion in a local heredoc—behaves differently when embedded inside a double-quoted SSH command string.

The fundamental issue is that SSH passes the entire string as a single argument to the remote shell's -c option. When the remote shell receives cat > file << 'EOF' ... EOF, the quoting of 'EOF' is already stripped by the local shell's double-quote processing. The remote shell then sees << EOF (without quotes), which means it does perform variable expansion on the heredoc body. The \$1 that was intended to protect against local expansion ends up being passed literally as \$1 to the remote shell, which then tries to expand $1 as a positional parameter—which is empty in a non-script context.

This is a classic distributed systems debugging problem: the failure manifests as "the file has the wrong content," but the actual fault is in the serialization/deserialization of commands across a network boundary. The assistant's tools (bash execution) abstract away the SSH connection, but the quoting semantics leak through.

The Pivot: Writing Locally, Shipping Remotely

The assistant's solution is elegant and pragmatic: "Let me write locally and scp." Instead of trying to construct the perfect quoting incantation, it uses the write tool to create the file locally at /data/dflash/scripts/launch_mtp.sh, and presumably follows up with an scp command to transfer it to the remote server (the next message in the conversation would show this).

This is a textbook example of changing the communication protocol rather than fighting with the existing one. The assistant recognized that the SSH heredoc channel was lossy—it couldn't reliably transmit the script content with all the necessary flags. By writing the file locally and using a binary-safe transfer (scp), the assistant sidesteps the quoting problem entirely. The write tool creates the file with exact byte-level fidelity, and scp transfers it without any shell interpretation.

Assumptions and Their Consequences

Several assumptions led to this debugging detour:

1. The assistant assumed heredocs over SSH would work identically to local heredocs. This is a reasonable assumption—many engineers use heredocs over SSH daily—but the interaction between double-quoted SSH commands and heredoc quoting is a known pitfall. The assistant's initial approach of wrapping the entire command in double quotes ("...") made the heredoc vulnerable to the local shell's expansion rules.
2. The assistant assumed the script was being written correctly. The first sign of trouble appeared in [msg 7490] when the log file wasn't created, and again in [msg 7491] when ls reported "No such file or directory." But the assistant attributed this to "nohup silently failing" or "the SSH connection silently failing to run commands properly" ([msg 7492]), rather than suspecting the script content itself was wrong.
3. The assistant assumed the escaping \$1 would protect against both local and remote expansion. The \$1 prevents the local shell from expanding $1, but when the remote shell receives the literal string \$1, it treats the backslash as an escape character and expands $1 to an empty string. The intended behavior was to pass the literal string $1 to the script so that Bash would expand it at runtime.
4. The assistant assumed that no output from the SSH command meant success. In [msg 7496], the SSH command produced "(no output)", which the assistant interpreted as the command running silently. In reality, the heredoc likely failed to write the intended content, and the && echo ok at the end never executed because the preceding commands failed silently.

Input Knowledge Required

To fully understand this message, one needs:

• SSH quoting semantics: Understanding how the local shell processes double-quoted strings before sending them to the remote host, and how the remote shell interprets the received command.
• Heredoc behavior: Knowing that << 'EOF' (quoted delimiter) prevents variable expansion, while << EOF (unquoted) allows it, and how this interacts with nested quoting.
• The previous debugging context: The assistant had been iterating on SGLang launch parameters for several rounds, trying different combinations of --mamba-scheduler-strategy, --max-mamba-cache-size, and --mem-fraction-static to resolve OOM errors.
• The deployment architecture: The assistant is working with a remote server (154.59.156.20) running Ubuntu, with SGLang installed in a Python venv, and the model stored at /workspace/dflash/models/Qwen3.6-27B.

Output Knowledge Created

This message creates several important outputs:

1. A locally-written script file at /data/dflash/scripts/launch_mtp.sh with the correct content (including extra_buffer, max-mamba-cache-size 24, and mamba-full-memory-ratio 0.4). This file can be transferred via scp with exact fidelity.
2. A debugging heuristic: When SSH heredocs produce unexpected results, the problem is likely in the quoting chain. The assistant now has a pattern for recognizing and resolving this class of failure.
3. A protocol decision: The assistant commits to using local file writes + scp for future script deployments, avoiding SSH heredocs for complex multi-line scripts with shell variables.

The Thinking Process

The assistant's reasoning in this message reveals a sophisticated debugging process. It doesn't just notice the symptom ("the script has the old content") and apply a fix. It traces the causal chain:

1. The script was supposed to be updated with new flags
2. The SSH command used a heredoc to write the script
3. The heredoc was embedded in a double-quoted SSH string
4. The escaping got mangled in the process
5. Therefore, the file on disk never received the new content This is a classic "meta-debugging" pattern: the assistant stops debugging the application (SGLang server parameters) and starts debugging the debugging infrastructure (the SSH command construction). The recognition that "the escaping is getting lost" is the key insight—it correctly identifies that the problem is in the communication channel, not in the content being communicated. The decision to use scp is particularly insightful because it changes the fundamental nature of the transfer. Instead of serializing the script content through a shell command string (which must be parsed and re-parsed by multiple shells), scp treats the file as a binary blob and transfers it verbatim. This eliminates the quoting problem entirely.

A Broader Lesson

This message, for all its brevity, encapsulates a universal lesson in distributed systems debugging: when a command works locally but fails remotely, suspect the serialization layer. The SSH heredoc problem is a specific instance of a general class of bugs where data is corrupted during transmission between systems. The assistant's response—change the protocol, not the data—is the correct engineering instinct.

In the context of the larger session, this message represents a brief pause in the ML deployment firefight. The assistant had been battling GPU memory constraints, incompatible scheduler strategies, and silent process failures. Now it was battling shell quoting. The ability to recognize when to stop pushing against a broken abstraction and instead find a different path is a hallmark of experienced systems engineering—whether human or artificial.

The message also demonstrates something remarkable about the assistant's cognitive architecture: it can hold multiple debugging threads simultaneously. While reasoning about Mamba cache sizes and memory fractions, it also monitors the integrity of its own tool invocations. When the script content check ([msg 7497]) revealed the corruption, the assistant immediately pivoted from the ML debugging track to the infrastructure debugging track, resolved the quoting issue, and prepared to continue. This kind of meta-cognitive monitoring—watching not just the problem but the problem-solving process itself—is essential for autonomous systems operating in complex environments.