The Pivot Point: From Validation to Deployment of Qwen3.5-122B on Dual DGX Spark

In any complex infrastructure deployment, there comes a moment when uncertainty crystallizes into certainty — when a hypothesis is confirmed and the path forward becomes clear. Message [msg 6666] captures precisely such a moment in the deployment of the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark nodes. It is a brief message, barely a few lines of reasoning followed by a single bash command, yet it represents the culmination of hours of debugging, the validation of a critical hypothesis, and a decisive strategic pivot that would determine the success of the entire deployment.

The Long Road to This Moment

To understand why [msg 6666] was written, one must appreciate the struggle that preceded it. The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter Mixture-of-Experts reasoning model quantized to FP8 — across two DGX Spark systems connected via InfiniBand RoCE. The DGX Spark is an unusual machine: it runs on an ARM Cortex-X925 CPU with 120GB of unified memory and a single NVIDIA GB10 Blackwell GPU (SM121 architecture). To serve a 119GB model, the assistant needed both nodes working together, splitting the model across two GPUs via tensor parallelism.

The initial approach used SGLang, but the multi-node NCCL initialization repeatedly deadlocked. The assistant spent messages [msg 6640] through [msg 6647] diagnosing SGLang's NCCL communicator creation, watching processes hang at channel setup without ever progressing to weight loading. After exhausting SGLang options, the assistant pivoted to vLLM — a proven multi-node inference engine on these systems — and discovered a community Docker image (hellohal2064/vllm-qwen3.5-gb10) specifically built for Qwen3.5 on the DGX Spark's ARM64 architecture with CUDA 13 support.

The single-node test in [msg 6660] through [msg 6665] was the critical validation experiment. It went through its own tribulations: the first attempt crashed immediately due to the --rm flag consuming the container before logs could be read ([msg 6661]), the second attempt revealed the image had its own entrypoint script that conflicted with manually-specified vllm serve arguments ([msg 6663]), and only the third attempt — mounting the model at /models/model and using environment variables VLLM_MAX_MODEL_LEN, VLLM_GPU_MEMORY_UTIL, etc. — succeeded in getting vLLM to start loading the model ([msg 6664]).

The Message Itself: Reasoning and Decision-Making

Message [msg 6666] opens with the assistant observing the state of the running test container:

Still loading (weight loading is slow on ARM with unified memory). This is going to OOM with TP=1 since 119GB model needs fit in 120GB along with OS/display. Let me stop this and go straight to the dual-node setup. The key learning: vLLM 0.17 with this image successfully resolves and starts loading Qwen3.5-122B-A10B-FP8.

This paragraph is dense with reasoning. The assistant makes several simultaneous observations and projections:

First, it confirms that the model is still loading — weight loading is slow on ARM with unified memory. This is a hardware-specific insight: the DGX Spark's ARM Cortex-X925 CPU, while powerful for an ARM processor, is not a server-class x86 chip. Combined with the 120GB unified memory architecture (where CPU and GPU share the same physical memory pool), weight loading involves large memory transfers that are inherently slower than on discrete GPU systems with dedicated high-bandwidth memory.

Second, the assistant projects an inevitable out-of-memory (OOM) condition. The model is 119GB in FP8. The DGX Spark has 120GB total unified memory. Even if vLLM could load the model entirely into GPU-accessible memory, the OS, display server (Xorg), and other system processes consume several gigabytes. The assistant had already seen evidence of this in [msg 6642] where nvidia-smi showed 27MiB consumed by Xorg. The margin is razor-thin or negative — the model simply cannot fit on a single node.

Third, and most importantly, the assistant extracts the critical learning: the model loads correctly. The vLLM 0.17 image successfully resolves the architecture to Qwen3_5MoeForConditionalGeneration, detects FP8 quantization, selects the appropriate backends (TRITON for MoE, FLASHINFER for attention), and begins weight loading without errors. This is the validation that the entire prior effort was building toward.

The Strategic Decision

The decision to stop the single-node test and proceed directly to the dual-node setup is a textbook example of efficient engineering judgment. The assistant could have waited for the OOM to occur naturally, confirming the prediction. It could have run additional diagnostics on the single-node configuration. Instead, it recognized that:

The single-node test had already served its purpose — it proved the model/image/architecture combination works.
The OOM outcome was deterministic and predictable — no new information would be gained by waiting for it.
The dual-node setup was the actual deployment target, and every minute spent waiting for an inevitable OOM was a minute not spent on the real work. This is the hallmark of an experienced systems engineer: knowing when a test has yielded its maximum information value and when to cut losses and move forward.

Assumptions and Their Validity

The message rests on several assumptions, most of which were well-founded:

The model will OOM on single-node TP=1. This was a near-certainty given the 119GB model size versus 120GB total memory. The assumption proved correct — the model could not fit on a single node, which is precisely why the dual-node deployment was necessary.

vLLM 0.17 with this image will work for dual-node. This was the critical assumption underlying the pivot. The assistant had proven single-node loading works, but multi-node vLLM introduces additional complexity: Ray cluster formation, NCCL initialization across nodes, and InfiniBand communication. The assistant was implicitly assuming that vLLM's multi-node support (which had been tested with other models on these same DGX Sparks, as evidenced by the existing launch-cluster.sh infrastructure) would work with this model and image. This assumption would be tested and validated in the subsequent messages.

The dual-node setup should use Ray. The assistant states "The existing setup uses Ray for multi-node." This was an accurate characterization of vLLM's architecture — vLLM uses Ray for multi-node orchestration, with each node running a Ray worker and communicating via NCCL over the InfiniBand interconnect.

Input Knowledge Required

To fully understand [msg 6666], the reader needs substantial context:

Hardware architecture knowledge: Understanding that the DGX Spark uses unified memory (CPU and GPU share the same 120GB pool via NVLink-C2C), that it runs on ARM architecture, and that weight loading is slower on ARM than on x86. Also understanding that the Blackwell GB10 GPU is an SM121 architecture chip requiring specific CUDA and kernel support.

Model characteristics: The Qwen3.5-122B-A10B-FP8 is a Mixture-of-Experts model with ~122B total parameters but only ~10B active per token (hence "A10B" in the name). The FP8 quantization reduces memory footprint from ~244GB (BF16) to ~119GB, making it feasible on the DGX Spark's 120GB unified memory — but only barely, and only with tensor parallelism across two nodes.

vLLM architecture: Understanding vLLM's V1 engine, its model registry, the role of the reasoning parser (--reasoning-parser qwen3), and how multi-node deployment works via Ray with NCCL for inter-node tensor parallelism.

The prior debugging history: The SGLang NCCL deadlock, the discovery of the hellohal2064/vllm-qwen3.5-gb10 image, the trial-and-error with Docker entrypoints and mount paths — all of this context is essential to understand why this message represents a breakthrough rather than a routine status update.

Output Knowledge Created

Message [msg 6666] creates and solidifies several pieces of knowledge:

Confirmed compatibility: vLLM 0.17.1rc1 with the hellohal2064/vllm-qwen3.5-gb10 image can load and serve Qwen3.5-122B-A10B-FP8 on DGX Spark hardware. This was not known before — the assistant had to verify it through the single-node test.

Confirmed memory constraint: The model cannot run on a single DGX Spark node, confirming the necessity of the dual-node approach. This validated the entire deployment strategy.

Established deployment pattern: The image expects the model at /models/model and uses environment variables (VLLM_MAX_MODEL_LEN, VLLM_GPU_MEMORY_UTIL, VLLM_HOST, VLLM_PORT) for configuration rather than command-line arguments. This knowledge was essential for the subsequent dual-node launch script.

Documented the entrypoint behavior: The image has a custom entrypoint that applies a Qwen3 reasoning parser patch, sets thermal limits, and wraps the vllm serve command. This was discovered through the trial-and-error in [msg 6663] and [msg 6664].

The Thinking Process

The message reveals the assistant's thinking process with remarkable clarity. The opening line — "Still loading (weight loading is slow on ARM with unified memory)" — shows the assistant monitoring the test container's progress in real-time, interpreting the slow loading not as a problem but as an expected characteristic of the hardware. The immediate projection of the OOM condition demonstrates forward thinking: the assistant is not just observing what is happening, but predicting what will happen.

The phrase "Let me stop this and go straight to the dual-node setup" is the decisive pivot. It's a conscious decision to truncate the current experiment because its marginal information value has dropped to zero. The assistant then explicitly captures the key learning in bold, ensuring it's not lost in the transition.

The final line — "Now I need to use the launch-cluster.sh infrastructure for dual-node. The existing setup uses Ray for multi-node. Let me set it up" — shows the assistant already planning the next steps, drawing on prior knowledge of the existing infrastructure. The launch-cluster.sh script was part of the DGX Spark's pre-existing vLLM Docker setup, and the assistant is leveraging it rather than reinventing the wheel.

The Bash Command and Its Significance

The message concludes with a bash command that stops the test container:

ssh aurora@10.1.230.180 'docker rm -f vllm-qwen35-test 2>/dev/null && echo "Stopped test container"' 2>&1

This command is notable for what it reveals about the assistant's operational style. It uses docker rm -f (force remove) rather than a graceful stop, appropriate for a test container that may be in an intermediate state. The 2>/dev/null suppresses any error messages if the container was already gone. The && echo "Stopped test container" provides a clear confirmation message. The command terminated after exceeding a 15-second timeout, suggesting the docker rm -f itself may have hung — perhaps because the container was in the middle of GPU memory allocation and the GPU was slow to release resources. This timeout is itself informative: it hints at the difficulty of managing GPU state on these systems, a challenge that would persist throughout the deployment.

Broader Significance

In the larger narrative of the coding session, [msg 6666] is the fulcrum on which the entire deployment turns. Before this message, the assistant was in discovery mode — testing, failing, iterating, and searching for a viable path. After this message, the assistant shifts into execution mode — building the dual-node Ray cluster, launching the distributed vLLM server, monitoring the ~15-minute model load, and verifying correct inference output with reasoning parsing.

The subsequent messages bear this out. In [msg 6667] through [msg 6670], the assistant adapts the launch-cluster.sh script, creates a new launch script for the Qwen3.5 model, and starts the Ray cluster. By [msg 6674], the model is loaded and serving requests. The verification in [msg 6675] shows correct reasoning output — the model generates a reasoning trace in a separate field, confirming the reasoning parser is active and working as intended.

None of that success would have been possible without the validation and strategic pivot captured in [msg 6666]. It is a small message — barely 100 words of reasoning and a single command — but it represents the moment when a complex, multi-hour debugging effort finally yielded a clear path forward. In the art of systems engineering, recognizing when you have enough information to make a decision is as important as the technical debugging itself. Message [msg 6666] is a masterclass in that recognition.