The Launch That Finally Worked: Orchestrating Multi-Node Inference Across Two DGX Sparks

In the sprawling narrative of deploying large language models across exotic hardware, most messages blend into a blur of bash commands, error messages, and incremental fixes. But occasionally, a message arrives that is deceptively simple on its surface yet carries the weight of everything that came before it. Message <msg id=6624> is such a moment.

The assistant types:

[assistant] [bash] # Launch worker first, then head
ssh aurora@10.1.230.180 'ssh aurora@192.168.200.13 "nohup /home/aurora/spark-launch-qwen35.sh worker > /home/aurora/sglang-worker.log 2>&1 &" && echo "Worker started"
sleep 3
nohup /home/aurora/spark-launch-qwen35.sh head > /home/aurora/sglang-head.log 2>&1 &
echo "Head started"' 2>&1
Worker started
Head started

Two SSH commands, a three-second sleep, and the laconic output "Worker started / Head started." On its own, this is barely worth a footnote. But to understand this message is to understand the entire multi-hour saga of deploying the Qwen3.5-122B-A10B-FP8 model—a 119-billion-parameter FP8 reasoning model—across two NVIDIA DGX Spark nodes connected by InfiniBand. This message represents the third attempt at a multi-node launch, and it is the one that finally works.

The Weight of Context: Why This Message Exists

To grasp why this message was written, one must understand the failure modes that preceded it. The assistant had been working methodically through a series of obstacles that would each have been showstoppers on their own.

The first obstacle was resource contention. The DGX Spark is a compact Blackwell-based system with 120GB of unified memory shared between CPU and GPU. The Qwen3.5-122B model in FP8 weighs approximately 119GB, leaving almost no headroom. On the second Spark node, an embeddings service was consuming nearly 12GB of GPU memory. The assistant's first multi-node attempt (at <msg id=6608-6609>) crashed with a CUDA out-of-memory error during distributed initialization—before the model even began loading. The assistant had to remotely stop the embeddings service, kill lingering Python processes, and verify that only the display server remained on the GPU.

The second obstacle was networking. The DGX Sparks are connected via a 192.168.200.x InfiniBand subnet, but the Gloo distributed communication backend used by PyTorch's torch.distributed defaulted to 127.0.0.1 for its TCP transport. The worker node tried to connect to localhost on the head node and timed out. The head sat waiting at "Init torch distributed begin" while the worker failed silently. The assistant diagnosed this in <msg id=6621> by examining the worker logs and recognizing the 127.0.0.1:17795 connection attempt, then edited the launch script to set GLOO_SOCKET_IFNAME and TP_SOCKET_IFNAME to the correct network interface. The updated script was copied to both nodes in <msg id=6623>.

Message <msg id=6624> is the third launch—the one that incorporates both the freed GPU memory and the corrected networking configuration. It is the synthesis of everything learned from the two previous failures.

The Reasoning Behind the Ordering

The assistant's decision to launch the worker first, wait three seconds, and then launch the head is not arbitrary. It reflects a deep understanding of how PyTorch distributed initialization works. The --dist-init-addr parameter points both nodes to the head's IP address as the rendezvous point. If the head starts first and immediately begins waiting for a peer that hasn't joined yet, it will block. If the worker starts first and finds no head to connect to, it may also block or fail. By launching the worker slightly before the head, the assistant ensures that when the head begins its distributed initialization, the worker is already listening and ready to respond. The three-second sleep is a pragmatic compromise—long enough for the worker's Docker container to start and begin its initialization, short enough to avoid excessive delay.

This ordering also reflects a lesson learned from earlier attempts. In the first launch (at <msg id=6602-6603>), the assistant launched the worker and head in rapid succession without the networking fixes, and the failure was ambiguous—it was unclear whether the OOM or the networking issue was primary. By fixing both problems independently before this launch, the assistant could attribute any remaining failure to a specific cause.

Assumptions and Their Validity

This message rests on several assumptions, most of which proved correct. The assistant assumed that the GLOO_SOCKET_IFNAME environment variable would force Gloo to use the correct InfiniBand interface rather than defaulting to localhost. This was validated in the subsequent message <msg id=6625>, where the logs showed "CustomAllreduce is disabled because this process group spans across nodes"—confirming that the two nodes had successfully discovered each other.

The assistant assumed that the freed GPU memory on the second Spark (after killing the embeddings service) would be sufficient for the model shard. Each node runs with tensor parallelism (TP=2), meaning each node holds approximately half the model weights—roughly 63GB per shard. With 120GB of unified memory and the embeddings service gone, this was tight but feasible. The assistant also implicitly assumed that the model loading would proceed without hitting the 95% memory threshold that triggers Ray's OOM killer—an assumption that would later require additional tuning.

One assumption that deserves scrutiny is that the --dist-init-addr parameter alone was sufficient for correct multi-node rendezvous. In earlier attempts, the parameter was set correctly to 192.168.200.12:20000, yet the worker still tried 127.0.0.1. The assistant correctly diagnosed that Gloo's TCP transport was overriding the init address, and that GLOO_SOCKET_IFNAME was the missing piece. This diagnosis required knowledge that PyTorch's distributed backend has two layers of addressing: the rendezvous endpoint (specified by --dist-init-addr) and the transport-level socket binding (controlled by GLOO_SOCKET_IFNAME). The former tells nodes where to find each other; the latter tells each node which network interface to bind to. Both must be consistent for multi-node communication to work.

Input Knowledge Required

To understand this message, a reader needs considerable background knowledge. One must understand the concept of tensor parallelism (TP) and why a 119GB model requires two nodes—each Blackwell GPU has 120GB of unified memory, but the model plus KV cache and CUDA overhead exceeds a single node's capacity. One must understand how PyTorch distributed initialization works, including the role of the Gloo backend and the distinction between rendezvous addressing and socket binding. One must understand the DGX Spark's unified memory architecture and why GPU memory and system memory are the same pool, making memory pressure a cross-cutting concern. One must also understand the SSH jump host pattern used here—the assistant connects to aurora@10.1.230.180 (a jump host or the head node's external IP) and then tunnels to aurora@192.168.200.13 (the second Spark's InfiniBand IP).

Output Knowledge Created

This message creates several concrete outputs. First, it produces running Docker containers on both nodes (sglang-qwen35-head and sglang-qwen35-worker) that begin the process of loading the Qwen3.5-122B model. Second, it produces log files (sglang-head.log and sglang-worker.log) that will be examined in subsequent messages to verify progress. Third, it creates a distributed process group that spans two physical nodes, enabling the model to be loaded across both GPUs.

But the most important output is negative: the absence of failure. Previous attempts produced OOM errors or connection timeouts. This attempt produces neither. The assistant's next message (<msg id=6625>) confirms the nodes connected successfully, and subsequent messages track the model loading through to completion, culminating in a working inference endpoint serving ~27 tok/s.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the structure of the command itself. The comment # Launch worker first, then head reveals a deliberate ordering decision. The sleep 3 between launches reveals an understanding of timing dependencies. The use of nohup and backgrounding (&) reveals the need for these processes to outlive the SSH session. The redirection of both stdout and stderr to log files reveals an awareness that debugging will require post-mortem log analysis.

The assistant also demonstrates a pattern of progressive debugging that is characteristic of expert systems engineering. Each failure produced a hypothesis, a fix, and a retry. The OOM failure led to freeing GPU memory. The connection timeout led to setting socket interface names. Message <msg id=6624> is the third iteration of this loop, and it succeeds because the assistant treated each failure as a learning opportunity rather than a dead end.

Conclusion

Message <msg id=6624> is a study in the power of context. A reader who sees only these two SSH commands might dismiss it as trivial. But a reader who understands the two failed launches that preceded it, the networking debugging that informed it, and the successful model serving that followed it recognizes it as a pivotal moment. It is the launch that finally worked—the moment when all the pieces clicked into place and the distributed inference engine began to breathe.