The Pivot Point: From SGLang to vLLM for Multi-Node Qwen3.5 Deployment on DGX Spark

Introduction

In any complex infrastructure deployment, there comes a moment when a carefully pursued strategy must be abandoned in favor of a fundamentally different approach. Message [msg 6657] in this opencode session captures precisely such a turning point. After spending hours wrestling with SGLang's multi-node NCCL initialization deadlock across two NVIDIA DGX Spark systems, the assistant has just discovered a promising alternative — a community-built Docker image (hellohal2064/vllm-qwen3.5-gb10) specifically engineered for running Qwen3.5 on these ARM-based Blackwell GPU machines. Message 6657 is the brief but consequential moment where the assistant transitions from reconnaissance (verifying the image exists and has the right model support) to execution (figuring out how to actually launch it in multi-node mode). It is a message of strategic reorientation, and understanding it requires tracing the thread of failures, discoveries, and assumptions that led to this point.

The Context: A Long Road of Deadlocks

The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter Mixture-of-Experts reasoning model — across two DGX Spark nodes connected via InfiniBand RoCE. The DGX Spark is NVIDIA's compact ARM-based desktop AI supercomputer, featuring a GB10 Blackwell GPU with 120GB of unified memory. Running a 119B model requires splitting it across both nodes using tensor parallelism (TP=2).

The assistant's first instinct was to use SGLang, a framework it had successfully deployed in earlier segments of this conversation (see [msg 6629] through [msg 6647]). SGLang had been the workhorse for previous model deployments on the Proxmox RTX PRO 6000 setup. However, SGLang's multi-node NCCL initialization consistently hung. The logs showed NCCL channels being created — Channel 00/0 : 1[0] -> 0[0] [receive] via NET/IBext_v11/0 — but the process never progressed beyond CustomAllreduce is disabled because this process group spans across nodes. After seven minutes of waiting, multiple restarts with different flags (--disable-custom-all-reduce, adjusted NCCL ports), and careful examination of NCCL debug output, the assistant concluded that SGLang's distributed initialization was fundamentally broken for this configuration.

This failure prompted a strategic pivot. The assistant searched for alternatives and discovered hellohal2064/vllm-qwen3.5-gb10 — a Docker image built by a community member specifically for running vLLM with Qwen3.5 on DGX Spark. The image contained vLLM 0.17.1rc1 with explicit Qwen3.5 model support (qwen3_5.py, qwen3_5_mtp.py), custom config parsers, and GB10-specific patches. Crucially, vLLM had a proven track record of multi-node operation on these Spark systems — the existing GLM deployment used it. The image was pulled and transferred to the second node via docker save | ssh | docker load in message [msg 6656].

The Message Itself

Message 6657 reads as follows:

Good. Now let me adapt the existing launch-cluster.sh infrastructure for our model. Looking at how the existing cluster script works, it uses Docker with node IPs for NCCL. Let me try vLLM's multi-node approach directly — vLLM uses Ray for multi-node:

>

``bash ssh aurora@10.1.230.180 'docker run --rm --entrypoint bash hellohal2064/vllm-qwen3.5-gb10 -c "vllm serve --help 2>/dev/null | grep -iE \"tensor.parallel|dist|nnodes|ray\" | head -10"' 2>&1 | tail -10 ``

At first glance, this appears to be a simple reconnaissance command — check what multi-node flags vLLM's CLI offers. But the reasoning text reveals a deeper deliberation. The assistant is weighing two distinct approaches to launching the multi-node deployment:

  1. Adapt the existing launch-cluster.sh infrastructure — a shell script from the spark-vllm-docker project that handles Ray cluster formation, Docker networking, and NCCL configuration. This is the "known working" path, leveraging proven infrastructure.
  2. Use vLLM's native multi-node approach directly — vLLM uses Ray under the hood for distributed serving, and the assistant is curious whether it can simply pass --tensor-parallel-size 2 with some node discovery flags and let vLLM handle the Ray orchestration internally. The phrase "Let me try vLLM's multi-node approach directly" signals the assistant's inclination toward the simpler, more direct path. Why write and debug a custom launch script if vLLM can handle multi-node natively? This is a classic engineering trade-off: invest time in understanding and adapting existing infrastructure versus exploring whether the framework itself provides a cleaner solution.

The Reasoning Process

The assistant's thinking reveals several layers of analysis:

First, there is an acknowledgment of what the existing infrastructure does: "it uses Docker with node IPs for NCCL." This tells us the assistant has already studied launch-cluster.sh and understands its mechanism — it manually sets up NCCL communication by specifying IP addresses, bypassing automatic node discovery.

Second, there is an assumption about vLLM's architecture: "vLLM uses Ray for multi-node." This is correct — vLLM's distributed serving relies on Ray for cluster management, task scheduling, and inter-node communication. The assistant is reasoning that if vLLM already has Ray integration, the multi-node setup might be as simple as passing the right CLI flags.

Third, there is an implicit cost-benefit calculation. The launch-cluster.sh approach requires: understanding the script's internals, adapting it for a different model path and image, transferring it to both nodes, and debugging any issues. The "vLLM directly" approach requires: discovering the right flags, testing them, and potentially debugging Ray networking. The assistant chooses to investigate the latter first — a reasonable choice given that it promises a cleaner solution.

Fourth, the assistant is thinking ahead about the command structure. By grepping for tensor.parallel|dist|nnodes|ray, it's looking for flags like --tensor-parallel-size, --distributed-executor-backend, --nnodes, and Ray-related options. These are the building blocks of a multi-node vLLM command.

Assumptions and Their Consequences

This message rests on several assumptions, some of which proved incorrect:

Assumption 1: vLLM's CLI help will work without GPU access. The assistant runs the command with --entrypoint bash and --rm, meaning the container starts but has no GPU devices attached (Docker's --gpus all is absent). The assumption is that vllm serve --help is a lightweight operation that parses argument definitions without touching hardware. This assumption was wrong — as messages [msg 6658] and [msg 6659] reveal, vLLM 0.17's argument parser imports the full engine configuration, which triggers GPU-dependent code paths and crashes without CUDA devices.

Assumption 2: The existing launch-cluster.sh can be straightforwardly adapted. The assistant's reasoning mentions "adapting the existing launch-cluster.sh infrastructure" as if it's a known quantity. While the script does exist on the system, its internals may have assumptions about model paths, Docker image names, and network topology that don't match the new deployment.

Assumption 3: vLLM's multi-node mode is well-documented and straightforward. The assistant is hoping for a simple flag-based multi-node setup. In practice, vLLM multi-node requires careful Ray cluster initialization, NCCL network interface configuration, and node IP specification — the same complexity the assistant was trying to avoid.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

Output Knowledge Created

This message, by itself, is a reconnaissance step. Its primary output is:

Broader Significance

Message 6657 exemplifies a pattern that recurs throughout infrastructure engineering: the moment of strategic reorientation. The assistant had invested significant effort in SGLang — building it from source, configuring it for Blackwell GPUs, debugging NCCL initialization. Abandoning that investment for a new framework (vLLM) and a community-built Docker image required both humility and pragmatism.

The message also illustrates the importance of reconnaissance before commitment. Rather than immediately writing a launch script, the assistant pauses to investigate vLLM's native capabilities. This reconnaissance saves time in the long run — even though the specific command fails, it quickly reveals that the direct approach won't work without GPU access, prompting a faster pivot to the tested strategy.

The two-path reasoning ("adapt existing infrastructure" vs. "use framework-native approach") is a classic engineering dilemma. The existing infrastructure path is safer but requires adaptation work. The native approach is cleaner but its feasibility is unknown. The assistant's decision to investigate the native approach first is rational: if it works, it saves significant effort; if it doesn't, the fallback path remains available.

Conclusion

Message 6657 is a brief but pivotal moment in a complex multi-node deployment. It captures the assistant's reasoning as it pivots from a failed SGLang strategy to a vLLM-based approach, weighs two implementation paths, and launches a reconnaissance command to inform the decision. The message's significance lies not in its immediate output (a bash command that will fail) but in the strategic reorientation it represents. It is the hinge point between hours of debugging a dead end and the eventual successful deployment of Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes — a deployment that would go on to achieve correct reasoning output and tool calling at ~27 tok/s single-request throughput.