"Copy Didn't Stick": SSH Process Lifecycles and Infrastructure Debugging in ML Deployment

Introduction

In the sprawling, multi-day effort to deploy and train a speculative decoding drafter for Qwen3.6-27B across a cluster of NVIDIA B200 GPUs, one message stands out for its deceptive simplicity. At message index 7609, the assistant states: "Copy didn't stick (SSH session ended, background process died). Let me do it properly." This single sentence, followed by a corrected setsid-based copy command and a polling loop, encapsulates a profound lesson about the gap between local intuition and remote execution semantics. What appears to be a mundane file copy operation is, in reality, a microcosm of the entire infrastructure challenge: assumptions about process persistence, the fragility of SSH session management, the need for robust signaling across network boundaries, and the iterative debugging rhythm that defines high-stakes ML engineering.

This article examines message 7609 in depth—its context, its reasoning, its technical decisions, and the broader patterns it reveals about deploying large language models in heterogeneous cloud environments.

Context: The Mission and the Infrastructure

To understand why a simple cp command became a critical juncture, we must first understand the mission. The team was in the process of generating 902,087 completions from Qwen3.6-27B, a 27-billion-parameter MoE model, to serve as training data for a DFlash speculative decoding drafter. This generation run required deploying SGLang inference servers across a 7× B200 NVL node—a machine with seven B200 GPUs (183 GB each), 2.2 TB of RAM, and a 923 GB /dev/shm RAM disk.

The model (~54 GB) had been downloaded to /workspace, a network-mounted filesystem that the user described as "essentially S3." When the assistant initially launched seven SGLang instances loading directly from /workspace, the results were painful: each shard took ~28 seconds to load over the network, translating to roughly 7 minutes per instance just to load the model weights. With seven instances competing for the same network mount, the total loading time would have been crippling.

The obvious solution was to copy the model to /dev/shm, the 923 GB RAM disk. This would transform loading from a network-bound operation (limited by filesystem latency and bandwidth) into a memory-speed operation (limited only by PCIe and memory bandwidth). The copy itself would be a one-time cost, after which all seven instances could load from local RAM.

The First Attempt and Its Failure

The assistant's first attempt to copy the model to /dev/shm used a straightforward approach:

cp -r /workspace/models/Qwen3.6-27B /dev/shm/Qwen3.6-27B &

This command was issued over SSH. The & backgrounds the process within the shell, and the SSH session returns immediately. However, when the SSH client exits—which happens as soon as the command string completes—the remote shell sends a SIGHUP to its child processes. The backgrounded cp process, still a member of the shell's process group, receives the signal and terminates.

This is a classic Unix pitfall. The assistant's initial reasoning assumed that backgrounding with & would be sufficient to keep the process alive after the SSH session ended. It was not. The copy silently died, leaving /dev/shm empty and the assistant none the wiser until the next round of checks revealed zero files in the target directory.

The user's earlier observation—"The /workspace is essentially S3, we don't want venv there probably"—had already highlighted the theme of network filesystem slowness. But the copy failure introduced a second, equally important theme: SSH session lifecycle management.

The Corrected Approach: setsid and Robust Polling

Message 7609 is the assistant's response to discovering this failure. The reasoning is explicit and concise: "Copy didn't stick (SSH session ended, background process died)." The assistant correctly diagnosed the root cause and formulated a two-part fix.

Part 1: Process Detachment with setsid

The corrected command uses setsid to launch the copy in a new session:

ssh root@[REDACTED] -p 36472 "setsid sh -c 'cp -r /workspace/models/Qwen3.6-27B /dev/shm/Qwen3.6-27B && echo DONE > /dev/shm/copy_done' &"

setsid creates a new session for the process, completely detaching it from the SSH shell's process group. When the SSH session ends, SIGHUP is sent to the shell's process group—but the setsid-launched process belongs to a different session entirely and is immune. This is the standard Unix mechanism for creating "daemon" processes that survive terminal disconnection.

The command also chains a completion signal: && echo DONE > /dev/shm/copy_done. This creates a marker file on the remote machine that signals successful completion. This is a deliberate design choice: rather than relying on process exit codes (which are lost when the SSH session disconnects), the assistant creates an explicit, persistent artifact that can be checked from subsequent SSH calls.

Part 2: Polling with Progress Tracking

The assistant then implements a polling loop from the local machine:

for i in $(seq 1 30); do
  sleep 10
  STATUS=$(ssh root@[REDACTED] -p 36472 "du -sh /dev/shm/Qwen3.6-27B/ 2>/dev/null || echo 'not yet'; cat /dev/shm/copy_done 2>/dev/null" 2>&1)
  echo "[$((i*10))s] $STATUS"
  if echo "$STATUS" | grep -q "DONE"; then break; fi
done

This loop polls every 10 seconds for up to 5 minutes. Each iteration does two things:

  1. Runs du -sh on the target directory to show progress (the size grows as files are copied)
  2. Checks for the copy_done marker file The || echo 'not yet' fallback handles the case where the directory doesn't exist yet (copy hasn't started writing). The 2>/dev/null redirects suppress errors from non-existent files. The output confirms success on the very first poll (10 seconds):
[10s] 52G	/dev/shm/Qwen3.6-27B/
DONE

The 52 GB model copied to the RAM disk in under 10 seconds—a transfer rate of over 5 GB/s, consistent with local memory-to-memory copy performance (the source was on a network filesystem, but the destination was RAM, and the source may have been cached in the page cache from the earlier download).

Assumptions, Mistakes, and Lessons Learned

Assumptions made by the assistant:

Input Knowledge Required

To understand this message, one needs:

Output Knowledge Created

This message produces several valuable outputs:

  1. The model is now in /dev/shm: A 52 GB copy of Qwen3.6-27B resides in RAM, ready for fast loading by all 7 SGLang instances
  2. A proven pattern for SSH-persistent operations: The setsid + marker file + polling pattern is established and can be reused for any future long-running remote operation
  3. Confirmation of infrastructure capabilities: The 10-second copy time validates that the RAM disk and network filesystem can sustain high transfer rates
  4. A debugging narrative: The sequence of failures (first copy died, second copy with nohup also died, third with setsid succeeded) documents the infrastructure's behavior under different process management strategies

The Broader Pattern: Infrastructure as Debugging

This message is not just about a file copy. It exemplifies a pattern that recurs throughout the entire coding session: infrastructure debugging as a first-class activity. The team repeatedly encountered issues that were not about model architecture or training algorithms, but about the gritty reality of distributed systems: SSH sessions terminating, network filesystems being slow, CUDA compilation exhausting memory, process management across machine boundaries.

Each of these infrastructure challenges required the same skills as debugging a model: forming hypotheses, testing them, observing results, and iterating. The assistant's reasoning in message 7609 follows this pattern precisely:

  1. Observe failure: The copy didn't persist
  2. Diagnose root cause: SSH session ended, killing the background process
  3. Formulate fix: Use setsid to detach from the session
  4. Add robustness: Include a marker file for verification and a polling loop for progress tracking
  5. Verify: Confirm the copy completed successfully This is the same scientific method applied to systems engineering. The message is a case study in how ML deployment in heterogeneous cloud environments demands not just ML expertise, but deep systems knowledge.

Conclusion

Message 7609 is a masterclass in concise infrastructure debugging. In a few lines of bash, the assistant diagnosed a subtle process lifecycle issue, implemented a robust fix using setsid and marker files, verified success through polling, and moved the entire pipeline forward. The 52 GB model now resides in /dev/shm, ready to serve 7 SGLang instances that will generate 902,087 completions—the foundation of the DFlash drafter training pipeline.

The message's true significance lies not in the copy itself, but in what it reveals about the nature of ML infrastructure work. The most sophisticated model architectures and training algorithms are worthless if the underlying systems cannot reliably move data between storage tiers, keep processes alive across network boundaries, and signal completion across machine boundaries. Every infrastructure fix, no matter how small, is a brick in the foundation that enables the actual ML work to happen.

"Copy didn't stick" is a story about SSH. But it's also a story about the invisible labor that makes large-scale ML possible.