The Relaunch That Changed Everything: Pivoting from SGLang to vLLM on Dual DGX Spark

ssh aurora@10.1.230.180 'ssh aurora@192.168.200.13 "nohup /home/aurora/spark-launch-qwen35.sh worker > /home/aurora/sglang-worker.log 2>&1 &" && sleep 3 && nohup /home/aurora/spark-launch-qwen35.sh head > /home/aurora/sglang-head.log 2>&1 & echo "Both launched"' 2>&1
Both launched

At first glance, message [msg 6635] appears to be a routine launch command — a simple SSH invocation that starts two background processes across two DGX Spark nodes. The bash tool timed out after 15 seconds, but the output "Both launched" suggests success. Yet this seemingly unremarkable message marks a critical inflection point in a grueling multi-hour debugging session. It was the last attempt to deploy the Qwen3.5-122B-A10B-FP8 model using SGLang's native multi-node tensor parallelism before the assistant abandoned the approach entirely and pivoted to vLLM. Understanding why this message was written — and what its timeout revealed — requires reconstructing the chain of failures that led to it.

The Context: A Multi-Node Nightmare

The assistant had been working for hours to deploy a 119-billion-parameter FP8 model across two NVIDIA DGX Spark systems (ARM64, SM121 Blackwell, 120GB unified memory each) connected via InfiniBand RoCE. The journey had already been brutal: memory exhaustion from a competing embeddings container, networking issues where Ray auto-detected the wrong IP address, NCCL initialization hangs, and a Gloo transport that defaulted to 127.0.0.1 instead of the InfiniBand subnet. Each failure had been diagnosed and patched in the launch script — environment variables for GLOO_SOCKET_IFNAME, NCCL_SOCKET_IFNAME, TP_SOCKET_IFNAME, explicit --dist-init-addr, and --disable-custom-all-reduce had all been added.

By the time we reach [msg 6635], the assistant had just finished editing the launch script to add NCCL_DEBUG=INFO (see [msg 6633]) and distributed the updated script to both nodes ([msg 6634]). This was the first run with NCCL debug logging enabled — a diagnostic shot in the dark after the previous attempt ([msg 6624]) had produced the cryptic CustomAllreduce is disabled message and then hung silently for minutes.

Why This Message Was Written

The immediate motivation was straightforward: test whether the NCCL debug flags would reveal the cause of the multi-node deadlock. But the deeper reasoning was more strategic. The assistant had exhausted most of the obvious knobs. The process group was forming correctly — both nodes logged CustomAllreduce is disabled because this process group spans across nodes — but then nothing happened. No weight loading, no memory allocation, no progress. The NCCL communicator channels were being created (the logs showed Channel 00/0 : 0[0] -> 1[0] [send] via NET/IBext_v11/0) but the initialization never completed.

The assistant had three hypotheses at this point:

  1. A second NCCL communicator group was deadlocking — SGLang creates multiple process groups during initialization (one for tensor parallelism, one for custom allreduce testing, possibly one for pipelining). If any of these hung, the entire startup would stall.
  2. The disable_custom_all_reduce=False default was causing trouble — even though the custom allreduce was correctly disabled across nodes, the NCCL communicator for the allreduce test might still be initializing and hanging.
  3. The ARM architecture was simply slow — with unified memory and a 119GB model, perhaps NCCL initialization genuinely took 10+ minutes and the assistant wasn't waiting long enough. Message [msg 6635] was designed to test hypothesis 3 while gathering data for hypotheses 1 and 2. By launching with NCCL_DEBUG=INFO, the assistant would be able to see exactly where the hang occurred. And by launching fresh containers (the old ones had been destroyed in [msg 6632]), the state would be clean.

The Decisions Made

Several decisions are embedded in this single command:

Launch order: worker first, then head. The assistant waited 3 seconds after launching the worker before starting the head. This is a deliberate choice — the head node hosts the TCP rendezvous store (port 20000), and the worker needs to connect to it. By starting the worker first and giving it a few seconds, the assistant ensured the worker's NCCL bootstrap would find the head's store already listening. This ordering had been refined through earlier failures where the head started first and the worker couldn't connect.

Nested SSH through a jump host. The command uses ssh aurora@10.1.230.180 as a jump host, then from there SSHs to 192.168.200.13 (the second Spark's InfiniBand IP). This architecture reflects the network topology: the assistant's machine (the Proxmox host with RTX PRO 6000 GPUs) can only reach the first DGX Spark directly. The second Spark is only accessible via the InfiniBand interconnect from the first Spark. This nested SSH pattern had been established earlier in the session and was working reliably.

Backgrounding with nohup and log redirection. Both processes use nohup and redirect stdout/stderr to log files. This is standard practice for long-running inference servers, but it also served a diagnostic purpose: the assistant could inspect the logs after the fact without needing to attach to running containers.

The 15-second timeout was a feature, not a bug. The bash tool's timeout of 15000ms meant the command would return after 15 seconds regardless of whether the SGLang processes had fully initialized. This was intentional — the assistant knew model loading would take many minutes (the model is 119GB, and on ARM with unified memory, loading is slow). The timeout allowed the launch to proceed without blocking the conversation, and the assistant would check progress separately in the next message.

Assumptions Made

The assistant made several assumptions, some of which proved incorrect:

That NCCL debug logging would reveal the hang. This was the core assumption behind the message. The assistant believed that with NCCL_DEBUG=INFO, the NCCL initialization would produce enough log output to pinpoint where the deadlock occurred. In practice, as we see in the subsequent messages ([msg 6636] through [msg 6646]), NCCL logged the channel setup successfully but then fell silent — the hang was in a phase that NCCL's debug logging didn't cover, or it was in PyTorch distributed code above NCCL.

That the launch script was correct. The assistant had just edited the script to add NCCL debug flags and distribute it to both nodes. The assumption was that the script was syntactically valid and would execute without errors. This was correct — the containers started and the processes ran — but the script didn't fix the underlying deadlock.

That the previous hang was reproducible. The assistant assumed that killing the containers and relaunching would reproduce the same hang, allowing diagnosis with debug logging. This was correct — the same hang occurred again.

That the InfiniBand interconnect was functioning correctly. The NCCL logs later showed NET/IBext_v11 being used successfully for channel setup, confirming this assumption was valid. The physical network was fine; the problem was in the software stack.

Mistakes and Incorrect Assumptions

The most significant mistake was the continued investment in SGLang's multi-node support. The assistant had already spent hours debugging NCCL initialization hangs across the two DGX Spark nodes. Each attempt produced the same deadlock at the same point. The assistant was operating under the assumption that SGLang could work multi-node on DGX Spark — that the hang was a configuration issue that could be fixed with the right flags.

In reality, as the assistant would discover in the next several messages ([msg 6647] onward), SGLang's multi-node NCCL initialization on DGX Spark was fundamentally broken for this model. The assistant eventually pivoted to vLLM (using the hellohal2064/vllm-qwen3.5-gb10 image discovered via web search in [msg 6649]), which worked immediately with Ray-based multi-node deployment.

Another subtle mistake was the assumption that the timeout was harmless. While the command executed successfully (the processes were launched), the 15-second timeout truncated the SSH session. If the SSH session had been killed mid-command (rather than after the command completed), the background processes might have been orphaned or killed by the SSH session's death. In this case, the output "Both launched" confirms the command completed before the timeout, so this wasn't an issue — but it was a risk.

Input Knowledge Required

To understand this message, one needs:

Network topology knowledge. The IP addresses reveal the architecture: 10.1.230.180 is a jump host (the first DGX Spark's external IP), while 192.168.200.13 is the second Spark's InfiniBand subnet address. The nested SSH pattern implies the assistant's machine cannot directly reach the second Spark.

DGX Spark hardware knowledge. The GB10 system-on-module has 120GB unified memory shared between CPU and GPU. This means GPU memory pressure directly impacts system memory, and vice versa. The assistant had already dealt with this by killing the embeddings container on the second Spark ([msg 6614]) to free memory.

SGLang and NCCL architecture. Understanding why CustomAllreduce is disabled is a normal message (it just means the process group spans multiple nodes) but why the subsequent hang is abnormal requires knowledge of SGLang's initialization sequence: process group creation, NCCL communicator initialization, custom allreduce testing, weight loading, KV cache allocation.

The history of failures. This message is meaningless without knowing about the previous hangs at [msg 6625] (where both nodes logged CustomAllreduce is disabled and then stalled), the OOM issues at [msg 6611], and the networking fixes at [msg 6622].

Output Knowledge Created

This message produced several valuable outputs:

Confirmation that the launch script executed correctly. The "Both launched" output confirmed the SSH commands ran without syntax errors, the worker started on the second Spark, and the head started on the first Spark.

A fresh set of NCCL debug logs. The subsequent messages ([msg 6636] through [msg 6646]) would show the NCCL debug output, revealing that NCCL channels were being created over InfiniBand but the process hung after channel setup. This ruled out network connectivity issues and pointed to a higher-level deadlock.

The final piece of evidence for the pivot. After this attempt produced the same deadlock, the assistant checked the worker log ([msg 6646]) and found identical behavior. This was the moment the assistant concluded SGLang multi-node was not going to work and began exploring alternatives — first checking the existing vLLM image ([msg 6648]), then discovering the Qwen3.5-specific vLLM image via web search ([msg 6649]), and finally deploying successfully with vLLM Ray.

The Thinking Process

The assistant's reasoning is visible in the sequence of messages surrounding [msg 6635]. After the previous launch ([msg 6624]) hung for 90+ seconds, the assistant checked the logs ([msg 6625]) and saw both nodes stuck at CustomAllreduce is disabled. The assistant then spent several messages investigating: checking container status ([msg 6628]), grepping for initialization markers ([msg 6629]), checking NCCL port availability ([msg 6631]), and ultimately deciding to add NCCL debug logging ([msg 6632]-[msg 6633]).

The decision tree was:

  1. The process group formed (both nodes see each other)
  2. NCCL channels were created (the NET/IBext_v11 messages)
  3. But nothing happened after that
  4. Possible causes: slow ARM initialization (wait longer), NCCL communicator deadlock (add debug logging), or a second process group hang (try --disable-custom-all-reduce) Message [msg 6635] was the diagnostic launch — the assistant was gathering data to distinguish between these possibilities. The timeout was pragmatic: the assistant knew the model load would take minutes, so it launched and planned to check progress separately. What's particularly interesting is what the assistant didn't do. It didn't check if the SGLang version supported multi-node on DGX Spark. It didn't search for known issues with SGLang multi-node on ARM64. It didn't consider vLLM as an alternative until after this attempt failed. The assistant was operating in a mode of incremental debugging — tweaking one flag at a time, relaunching, checking logs, tweaking again. This is effective for configuration issues but less so for fundamental incompatibilities.

Conclusion

Message [msg 6635] is a study in the limits of incremental debugging. The assistant had invested hours in making SGLang work across two DGX Spark nodes, systematically addressing each failure: OOM, networking, NCCL configuration. But the core problem — that SGLang's multi-node NCCL initialization deadlocked on ARM64 with this model — was not a configuration issue. It was a fundamental incompatibility that no amount of flag tweaking could fix.

The pivot to vLLM that followed ([msg 6648]-[msg 6650]) was the correct decision, and it worked immediately. The lesson is one that every infrastructure engineer learns eventually: sometimes the best debugging technique is knowing when to switch tools. The 15-second timeout on this message wasn't just a tool limit — it was the signal that the current approach had reached its dead end.