When Heredocs Fail Through SSH: A Case Study in Remote Script Deployment

Introduction

In the middle of an intensive optimization session for the GLM-5-NVFP4 large language model deployment on 8× RTX PRO 6000 Blackwell GPUs, the assistant encountered a frustrating but instructive problem: a heredoc-based shell script creation command was silently failing when executed through SSH. Message [msg 944] captures the moment of diagnosis and the pivot to a workaround. While the message itself is brief—a single bash command that echoes a script line by line onto a remote machine—it encapsulates a deeper story about the fragility of remote command execution, the importance of verifying assumptions, and the iterative nature of performance tuning at scale.

The Optimization Context

To understand why this message matters, one must first appreciate the broader optimization campaign underway. The assistant had been systematically working to improve the inference throughput of the GLM-5-NVFP4 model, a massive Mixture-of-Experts (MoE) architecture with 256 experts and 8 active experts per token. Earlier in the session, the assistant had confirmed that the model was compute-bound rather than communication-bound by benchmarking TP4+PP2 against TP8, finding TP8 to be 2× faster. This ruled out allreduce latency as the primary bottleneck and focused attention on the FP4 GEMM kernel efficiency on NVIDIA's SM120 architecture (Blackwell).

A deep investigation into the FP4 GEMM kernels had revealed that the GPUs were drawing only ~235W out of their 600W TDP during inference, and the CUTLASS kernels plateaued at ~1,300 TFLOPS (70% of dense peak) only for very large matrices. During actual decode, per-expert batch sizes of ~16–64 tokens achieved merely 0.8–55 TFLOPS (0.02–3% of peak). The 99KB shared memory limit on SM120 prevented using larger CUTLASS tile configurations, and cuBLASLt FP4 was found to be no faster than FlashInfer's CUTLASS path.

The breakthrough had come from systematic server parameter tuning. By raising --max-running-requests to 2048 and setting --num-continuous-decode-steps 8, the assistant achieved a 28% throughput improvement at 2048 concurrency, reaching 2,095 output tok/s and 4,151 total tok/s. This was documented in message [msg 937], which showed the performance comparison table.

The Next Iteration: Pushing Further

With the successful 28% improvement under their belt, the assistant naturally wanted to push further. The hypothesis was straightforward: if num-continuous-decode-steps 8 helped by batching more decode steps together before re-scheduling, perhaps num-continuous-decode-steps 16 would help even more. The idea was that more continuous decode steps would keep the batch "hot" for longer, reducing scheduling overhead and increasing the effective per-expert batch size.

In message [msg 940], the assistant attempted to restart the server with the new parameter. The command used a heredoc to create the launch script:

ssh root@10.1.230.174 'pkill -9 -f sglang; sleep 3

cat > /root/run_tp8_cds16.sh << "ENDSCRIPT"
#!/bin/bash
source /root/ml-env/bin/activate
...
ENDSCRIPT
chmod +x /root/run_tp8_cds16.sh
nohup /root/run_tp8_cds16.sh > /root/sglang-server.log 2>&1 &
echo "PID=$!"'

This is a common pattern for remote script deployment: use a heredoc to write a multi-line script, then execute it. But something went wrong.

Diagnosing the Failure

The failure wasn't immediately obvious. In message [msg 941], after a 100-second sleep, the assistant checked the server status and found NOT_READY. The log showed only the old server's shutdown message. In message [msg 942], the assistant checked for running processes and found none—the server hadn't started at all. Then in message [msg 943], the critical clue emerged:

ssh root@10.1.230.174 'cat /root/run_tp8_cds16.sh | head -5'
cat: /root/run_tp8_cds16.sh: No such file or directory

The script file simply didn't exist. The heredoc had failed silently, and the subsequent chmod +x and nohup commands either failed or operated on a non-existent file. This is message [msg 944]'s immediate predecessor—the moment the assistant realized the script hadn't been created.

Message 944: The Workaround

Message [msg 944] is the assistant's response to this diagnosis. The message reads:

The heredoc is failing through SSH. Let me use a different approach — echo line by line:

>

``bash ssh root@10.1.230.174 &#39;echo &#34;#!/bin/bash&#34; &gt; /root/run_tp8_cds16.sh echo &#34;source /root/ml-env/bin/activate&#34; &gt;&gt; /root/run_tp8_cds16.sh echo &#34;export PYTHONUNBUFFERED=1 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=5 NCCL_MIN_NCHANNELS=8 OMP_NUM_THREADS=8 SAFETENSORS_FAST_GPU=1 CUDA_HOME=/usr/local/cuda-12.8&#34; &gt;&gt; /root/run_tp8_cds16.sh echo &#34;python3 -u -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 --reasoning-parser glm45 --tool-call-parser glm47 --trust-remote-code --tp-size 8 --mem-fraction-static 0.92 --max-running-requests 2048 --kv-cache-dtype auto --quantization modelopt_fp4 --attention-backend flashinfer --fp8-gemm-backend cutlass --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutlass --disable-cuda-graph --disable-radix-cache --num-continuous-decode-steps 16 --host 0.0.0.0 --port 8000&#34; &gt;&gt; /root/run_tp8_cds16.sh chmod +x /root/run_tp8_cds16.sh cat /root/run_tp8_cds16.sh&#39; ``

>

The output confirms success:

>

`` #!/bin/bash source /root/ml-env/bin/activate export PYTHONUNBUFFERED=1 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=5 NCCL_MIN_NCHANNELS=8 OMP_NUM_THREADS=8 SAFETENSORS_FAST_GPU=1 CUDA_HOME=/usr/local/cuda-12.8 python3 -u -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 --reasoning-parser glm45 --tool-call-parser glm47 --trust-remote-code --tp-size 8 --mem-fraction-static 0.92 --max-running-requests 2048 --kv-cache-dtype auto --quantization modelopt_fp4 --attention-backend flashinfer --fp8-gemm-backend cutlass --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutlass --disable-cuda-graph --disable-radix-cache --num-continuous-decode-steps 16 --host 0.0.0.0 --port 8000 ``

The approach is elegantly simple: instead of sending a multi-line heredoc as a single SSH command, each line is sent as a separate echo command that appends to the file. The &gt; operator creates the file with the first line, and subsequent &gt;&gt; operators append. Finally, chmod +x makes it executable, and cat verifies the content.

Why Heredocs Fail Through SSH

The root cause of the heredoc failure is worth examining, as it's a subtle but common pitfall. When a heredoc is embedded inside an SSH command string, several things can go wrong:

  1. Quoting and escaping: The heredoc delimiter and its contents are inside a single-quoted SSH command string. The shell on the local machine processes the quoting before sending the command to the remote machine. If the heredoc delimiter appears inside single quotes, the shell may interpret it differently than intended.
  2. Newline handling: SSH passes the command string to the remote shell. Multi-line commands inside single quotes should work in theory, but the interaction between the local shell's parsing, the SSH protocol's command transmission, and the remote shell's interpretation can introduce subtle issues. Some SSH implementations or configurations may not handle multi-line command strings reliably.
  3. Timing and process management: In the original command (msg [msg 940]), the heredoc was followed by chmod +x and nohup ... &amp;. If the heredoc failed silently, the subsequent commands would either fail or run on a non-existent file, but the errors might not propagate back through SSH clearly.
  4. Shell differences: The remote machine might be using a different shell (e.g., dash instead of bash) that handles heredocs differently, especially when the command is passed as a string argument to SSH. The echo-based approach avoids all these issues. Each echo command is a simple, single-line command that SSH can execute reliably. There's no multi-line construct, no heredoc delimiter, and no quoting complexity. Each command is independent and atomic.

Assumptions and Their Consequences

This message reveals several assumptions the assistant had been operating under:

Assumption 1: Heredocs work reliably through SSH. This is generally true for simple cases, but the assistant learned the hard way that they can fail silently. The failure wasn't detected until the server failed to start, costing several minutes of debugging time.

Assumption 2: The script file was created successfully. In messages <msg id=941-942>, the assistant checked server status and process lists, but didn't immediately verify that the script file existed. The assumption was that the heredoc had worked, so the debugging effort focused on why the server wasn't starting rather than whether the script existed.

Assumption 3: The old log file was being overwritten. When checking the server log in message [msg 941], the assistant saw old log content and assumed the new server was writing to a different location or hadn't started yet. In reality, the old log was still there because the new server never started to overwrite it.

These assumptions cost time, but the assistant's debugging process was methodical: check server health → check process list → check log → check script file existence. Each step narrowed the problem space until the root cause was identified.

The Script's Purpose and Parameters

The script being deployed configures the SGLang server with a specific set of parameters optimized for the GLM-5-NVFP4 model. Let's examine what each parameter does:

The Thinking Process

The assistant's reasoning in this message demonstrates a systematic debugging approach:

  1. Observe the symptom: The server didn't start after the heredoc-based script creation.
  2. Formulate hypotheses: Possible causes include server crash, startup failure, or script not being created.
  3. Test hypotheses systematically: Check health endpoint → check process list → check log → check script file.
  4. Identify root cause: The script file doesn't exist—the heredoc failed.
  5. Implement workaround: Switch from heredoc to echo-based file creation.
  6. Verify the fix: The cat command at the end of the SSH command confirms the file was created correctly. This is classic debugging methodology: isolate variables, test the simplest explanations first, and verify fixes before moving on.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. A verified workaround for SSH heredoc failures: The echo-based approach is a reliable alternative that can be used whenever heredocs fail through SSH.
  2. A deployable server configuration: The script file /root/run_tp8_cds16.sh is now ready to execute on the remote machine, configured with num-continuous-decode-steps 16.
  3. A debugging pattern: The sequence of checks (health → process → log → file) serves as a template for diagnosing similar remote execution failures.
  4. Documentation of a subtle SSH behavior: The failure mode of heredocs in SSH commands is now documented implicitly, serving as a reference for future troubleshooting.

Broader Implications

While this message might seem like a minor operational hiccup, it illustrates several important principles for working with remote systems:

Always verify your assumptions. The assistant assumed the heredoc had worked because it's a common pattern that usually works. But "usually" isn't "always," and the time spent debugging could have been saved by adding a simple file existence check after the heredoc command.

Prefer simple, atomic operations over complex constructs. The echo-based approach is more verbose but more reliable. Each command does one thing and does it simply. This is a good principle for remote command execution, where complexity increases the chance of subtle failures.

Build debugging into your workflow. The assistant's systematic checking—health endpoint, process list, log file, script file—is a good model. Each check eliminates a class of possible causes and narrows the search space.

Document your workarounds. The assistant explicitly states the problem ("The heredoc is failing through SSH") and the solution ("Let me use a different approach — echo line by line"). This documentation helps anyone reading the conversation understand why this approach was chosen.

Conclusion

Message [msg 944] captures a small but meaningful moment in a larger optimization journey. The assistant was pushing the boundaries of what the GLM-5-NVFP4 model could achieve on 8× RTX PRO 6000 Blackwell GPUs, having already achieved a 28% throughput improvement. The attempt to push further with num-continuous-decode-steps 16 was blocked by a technical obstacle—heredoc failure through SSH—but the obstacle was quickly diagnosed and overcome.

The message demonstrates that even in the midst of high-level performance optimization work, low-level operational details matter. A reliable deployment mechanism is a prerequisite for iterative experimentation, and the assistant's ability to quickly diagnose and work around the heredoc failure kept the optimization campaign moving forward. The echo-based workaround may be less elegant than a heredoc, but it works—and in engineering, working reliably is often more important than working elegantly.