The Relaunch: Orchestrating Multi-Node Inference Across Two DGX Spark Systems

The Message

[assistant] Second spark is clear — only display server using 43MB. Now let me relaunch both nodes:
[bash] # Start worker first
ssh aurora@10.1.230.180 'ssh aurora@192.168.200.13 "nohup /home/aurora/spark-launch-qwen35.sh worker > /home/aurora/sglang-worker.log 2>&1 &" && echo "Worker started"
sleep 2
# Start head
nohup /home/aurora/spark-launch-qwen35.sh head > /home/aurora/sglang-head.log 2>&1 &
echo "Head started, PID: $!"' 2>&1
Worker started
Head started, PID: 2429084

At first glance, this appears to be a routine command: start a worker process on a remote machine, wait two seconds, then start a head process on the local machine. But this message represents the culmination of a significant debugging effort — the third attempt to deploy a 119-billion-parameter FP8 model across two NVIDIA DGX Spark nodes. The brevity of the message belies the complexity of the infrastructure challenge it addresses.

Context: The Path to This Message

To understand why this message was written, we must trace the events that preceded it. The assistant had been tasked with deploying Qwen3.5-122B-A10B-FP8, a massive Mixture-of-Experts reasoning model, across two DGX Spark systems connected via InfiniBand RoCE. Each Spark has a single NVIDIA GB10 GPU (SM121 Blackwell architecture) with 120GB of unified memory — meaning GPU memory and system memory share the same physical pool.

The deployment had already overcome several hurdles. The model was downloaded from HuggingFace (119GB of safetensor files) and transferred to the second Spark at ~640MB/s over the IB link. A custom Docker image (sglang-qwen35) was built. A launch script (spark-launch-qwen35.sh) was written to handle the multi-node SGLang invocation with --nnodes 2, --node-rank, and --dist-init-addr flags.

The first launch attempt failed because the launch script contained a vLLM-specific flag (--language-model-only) that SGLang didn't recognize. The assistant fixed the script, re-synced it to both nodes, and tried again. The second attempt failed more interestingly: the worker on the second Spark crashed with a CUDA out-of-memory error during set_device — before any model loading had even begun.

This OOM failure was the critical problem that led to the subject message.

Why This Message Was Written: The Reasoning and Motivation

The assistant's reasoning chain reveals a methodical debugging process. When the worker crashed with "CUDA error: out of memory" during set_device, the assistant correctly diagnosed that this was not a model capacity issue but a CUDA context allocation failure. On DGX Spark's unified memory architecture, every CUDA context allocation competes for the same 120GB pool shared between CPU and GPU. Any existing GPU-using process reduces the available pool.

The assistant checked nvidia-smi and free -h on the second Spark and found:

How Decisions Were Made

The message embodies several deliberate decisions about launch orchestration:

Worker-first ordering: The assistant starts the worker on the second Spark before the head on the first Spark. This is not arbitrary — SGLang's distributed initialization uses a rendezvous protocol where both nodes connect to the --dist-init-addr (the head node's IP). Starting the worker first ensures it is ready to accept the connection when the head initializes its NCCL distributed communicator. If the head started first and tried to initialize distributed communication before the worker was listening, it would hang or crash.

Two-second sleep: The sleep 2 between worker and head launch is a pragmatic heuristic. It's not long enough for the worker to fully load the model (which takes ~15 minutes for a 119GB model), but it is long enough for the Docker container to start and for the SGLang process to begin listening on the NCCL port. The distributed initialization is a blocking handshake — the head will wait for the worker regardless, so the sleep just prevents a race condition where the head's NCCL init races ahead of the worker's container startup.

Background execution with nohup: Both processes are launched with nohup and output redirected to log files. This is essential for long-running inference servers — it detaches the processes from the SSH session so they survive the connection closing, and it captures logs for later debugging.

Single SSH command for both launches: The assistant nests the worker launch inside the same SSH command that launches the head (ssh aurora@10.1.230.180 '...'). This means the head node (10.1.230.180) orchestrates both launches — it SSHs to the worker node (192.168.200.13) to start the worker, then starts the head locally. This avoids needing a separate terminal or session for each node.

Assumptions Made

The message rests on several assumptions, some explicit and some implicit:

  1. The worker will be reachable and responsive: The assistant assumes the SSH jump through the head node to the worker's IB subnet IP (192.168.200.13) will succeed, and that the worker's Docker daemon is running and can start the container.
  2. Two seconds is sufficient startup time: The sleep 2 assumes the worker's Docker container will initialize and begin its NCCL listener within two seconds. If the container image needs to be pulled (unlikely, since it was already built), or if system resources are constrained, this could fail.
  3. The memory cleanup was complete: The assistant assumes that killing the reranker process and stopping the embeddings service freed all GPU memory. However, the nvidia-smi output showed 0MB used (besides display), confirming this.
  4. The launch script is correct: After fixing the --language-model-only flag, the assistant assumes no other SGLang-specific issues remain in the script.
  5. The IB subnet is the correct communication path: The --dist-init-addr in the launch script uses the 192.168.200.x IB subnet. The assistant assumes NCCL will use this interface for inter-node tensor parallelism, not the external 10.1.230.x network.
  6. Both nodes have identical model paths: The model was rsynced to /home/aurora/models/Qwen3.5-122B-A10B-FP8/ on both nodes. The assistant assumes the Docker volume mounts in the launch script map this correctly.

Potential Issues and Incorrect Assumptions

While the message is well-reasoned, several assumptions warrant scrutiny:

The two-second sleep is fragile: In practice, Docker container startup on DGX Spark can take 5-15 seconds depending on image size and system load. If the worker hasn't started listening by the time the head initializes NCCL, the head will hang waiting for the connection — which it would do indefinitely (NCCL doesn't have a short timeout). This means the sleep is somewhat irrelevant; the head would wait anyway. The real risk is the opposite: if the head starts too fast and its NCCL init completes before the worker's process even begins, the worker might miss the initial handshake. However, NCCL's TCP store-based rendezvous handles this — the head publishes its address to a key-value store, and the worker reads it when it joins.

No verification between worker and head launch: The assistant doesn't check that the worker actually started successfully before launching the head. The echo "Worker started" only confirms the SSH command executed, not that the Docker container is running or that SGLang initialized. A more robust approach would poll the worker's log or check for the container process before proceeding.

Single point of orchestration: By nesting both launches in one SSH session, the assistant creates a dependency where a network interruption could leave one node running and the other not. If the SSH connection drops after the worker starts but before the head starts, the worker would be orphaned.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several concrete outcomes:

  1. A running SGLang worker process on the second Spark (192.168.200.13), logged to /home/aurora/sglang-worker.log
  2. A running SGLang head process on the first Spark (10.1.230.180) with PID 2429084, logged to /home/aurora/sglang-head.log
  3. An active NCCL distributed communicator being established between the two nodes over the IB subnet
  4. The beginning of model loading — the 119GB Qwen3.5-122B-A10B-FP8 model will begin loading across both GPUs with tensor parallelism (TP=2) The success of this launch is confirmed in subsequent messages, where the model loads successfully and begins serving requests at ~27 tok/s single-request throughput.

The Thinking Process

The assistant's thinking, visible across the sequence of messages leading to this one, demonstrates a systematic debugging methodology:

  1. Observe failure: The worker crashes with CUDA OOM during set_device ([msg 6611])
  2. Gather data: Check nvidia-smi and free -h to understand memory pressure ([msg 6612])
  3. Form hypothesis: The embeddings service and reranker process are consuming GPU memory that the unified memory system cannot reclaim for the new CUDA context
  4. Test hypothesis: The assistant notes "With GB10 unified memory, GPU memory IS system memory" — a key insight that unified memory means no hard partition between GPU and CPU RAM
  5. Execute remediation: Stop the embeddings service, kill the reranker process, verify with nvidia-smi ([msg 6614]-[msg 6618])
  6. Verify cleanup: Confirm only 43MB display server remains on GPU
  7. Relaunch: Execute the corrected launch sequence (the subject message) This is textbook debugging: observe, measure, hypothesize, test, fix, verify, retry. The assistant never assumed the OOM was a fundamental capacity problem — it correctly identified that the issue was contention for the unified memory pool, not insufficient memory.

Conclusion

Message [msg 6619] appears unremarkable — a simple SSH command to start two processes. But in context, it represents the successful resolution of a complex resource contention problem on a novel hardware architecture. The assistant's understanding of DGX Spark's unified memory model, its methodical cleanup of competing GPU processes, and its careful orchestration of the multi-node launch sequence demonstrate the depth of systems knowledge required to deploy large language models on cutting-edge hardware. The message is a pivot point in the deployment: after multiple failed attempts, this is the moment the infrastructure finally aligns, and the model begins its journey from disk to serving endpoint.