The Pivot: From SGLang Deadlock to a Single-Node vLLM Test on DGX Spark

In the sprawling, multi-session journey of deploying large language models across heterogeneous GPU infrastructure, few moments are as decisive as the one captured in message 6660. After spending hours wrestling with a stubborn multi-node NCCL deadlock in SGLang, the assistant makes a clean break: it abandons the SGLang approach entirely and pivots to a community-built vLLM Docker image, launching a single-node test container for the Qwen3.5-122B-A10B-FP8 model on a DGX Spark. This message is a textbook example of pragmatic infrastructure decision-making under pressure—a moment where diagnosis gives way to action, and where the assistant's deep understanding of the deployment stack shines through.

The Context: A Multi-Node Nightmare

To understand why this message matters, one must appreciate the ordeal that precedes it. The assistant had been attempting to deploy the 122B-parameter Qwen3.5 MoE model (in FP8 precision) across two NVIDIA DGX Spark nodes—ARM-based systems with GB10 Blackwell GPUs and 120GB of unified memory each, connected via InfiniBand RoCE. The initial approach used SGLang with multi-node tensor parallelism, a configuration that had worked for other models on this hardware.

But SGLang's multi-node NCCL initialization kept hanging. The logs showed NCCL channels being created successfully (send/receive pairs via NET/IBext_v11), but the process never progressed past the CustomAllreduce is disabled message. Both nodes sat idle for minutes on end, GPU memory barely touched, while NCCL communicator groups deadlocked. The assistant tried multiple remedies: disabling custom all-reduce, adjusting NCCL debug settings, waiting extended periods for ARM-based initialization. Nothing worked. The forum threads for DGX Spark mentioned similar issues with SGLang's allreduce fusion causing crashes on multi-node setups.

This is the crucible in which message 6660 is forged.

Discovery of the Community Image

The assistant's pivot began with a web search that turned up hellohal2064/vllm-qwen3.5-gb10—a Docker image specifically built for running Qwen3.5 on DGX Spark GB10 hardware (ARM64, SM120/121 compute capability, CUDA 13). This was not an official NVIDIA image; it was a community contribution from a user named hellohal2064, updated just 20 days prior to the session.

The assistant methodically validated this image before committing to it. First, it pulled the image and attempted to inspect its contents, discovering that vLLM's import chain crashes without GPU access (a quirk of its pydantic-based argument parsing). Undeterred, the assistant used --entrypoint bash to bypass the auto-import and directly inspect the filesystem. The results were promising: the image contained qwen3_5.py, qwen3_5_mtp.py, and custom config classes (qwen3_5_moe.py) in vLLM's transformers_utils/configs/ directory. This meant vLLM 0.17.1rc1 had native Qwen3.5 support with its own config parser, not relying on an upstream transformers release for the model class. Transformers 4.57.3 was present but secondary.

The assistant then transferred the image to the second Spark node using docker save piped over SSH to docker load, achieving transfer at approximately 640MB/s over the InfiniBand link. The infrastructure for multi-node was prepared—but the assistant wisely chose not to jump straight into a multi-node launch.

The Decision in Message 6660

The subject message opens with a telling observation: "This vLLM version really needs GPU access to even parse its args." The assistant had just spent several commands trying to extract CLI help text from the image, only to be met with pydantic validation errors caused by the absence of GPU devices in the test environment. This is a frustrating constraint—it means the assistant cannot even preview what flags are available without running a full container with GPU access.

The reasoning then takes a pragmatic turn: "Let me use the existing launch-cluster.sh from spark-vllm-docker which already handles all the Ray + Docker complexity. I'll adapt it." This is a critical design decision. Rather than reinventing the multi-node launch infrastructure from scratch, the assistant plans to leverage proven tooling. The spark-vllm-docker repository (referenced as the existing setup on these machines) contains scripts that handle Ray cluster formation, NCCL environment variables, and Docker networking across the two Spark nodes.

But then the assistant does something even more telling: it does not immediately launch the multi-node cluster. Instead, it runs a single-node test first:

docker run --rm -d \
  --gpus all --ipc=host --network host \
  --privileged \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /home/aurora/models/Qwen3.5-122B-A10B-FP8:/models/Qwen3.5-122B-A10B-FP8:ro \
  --name vllm-qwen35-test \
  hellohal2064/vllm-qwen3.5-gb10 \
  vllm serve /models/Qwen3.5-122B-A10B-FP8 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --reasoning-parser qwen3

This is a textbook "validate before scale" strategy. Before attempting the complexity of multi-node Ray orchestration, the assistant wants to confirm that:

The image can load the model at all
The FP8 quantization format is compatible
The reasoning parser works correctly
The model generates coherent output

Breaking Down the Command

Every flag in this docker run command reflects deep knowledge of the deployment environment. --gpus all --ipc=host --network host --privileged provides full GPU access, shared memory for NCCL, host networking for Ray/NCCL communication, and the privileged mode needed for CUDA operations on the DGX Spark's ARM architecture. The --ulimit memlock=-1 --ulimit stack=67108864 flags prevent memory locking limits from interfering with NCCL's pinned memory allocations—a common source of cryptic failures in distributed GPU workloads.

The model path is mounted read-only (:ro) from /home/aurora/models/Qwen3.5-122B-A10B-FP8, a 119GB directory that was previously downloaded from HuggingFace and rsynced between nodes. The --tensor-parallel-size 1 flag explicitly limits this test to a single GPU, even though the Spark has one GPU (the unified memory architecture means TP=1 is the only option on a single node anyway). The --gpu-memory-utilization 0.85 reserves 15% of memory for CUDA kernels, KV cache fragmentation headroom, and other overheads—a conservative but prudent setting for a 122B parameter model in FP8.

The --reasoning-parser qwen3 flag is particularly noteworthy. Qwen3.5 models, like their Qwen3 predecessors, can output reasoning traces in a separate structured field before producing the final answer. The reasoning parser extracts this content from the raw token stream and formats it appropriately for the OpenAI-compatible API response. Without this flag, the model's output would appear garbled, with reasoning content mixed into the message content field.

Assumptions and Blind Spots

The assistant makes several assumptions in this message, some justified and some that would prove incorrect. It assumes that the community image is correctly built for the DGX Spark's ARM64 architecture and CUDA 13 stack—a reasonable assumption given that it was specifically advertised for this purpose. It assumes that the model's FP8 format (Qwen3.5-122B-A10B-FP8) is compatible with vLLM's FP8 quantization path, which is plausible since vLLM 0.17 has native FP8 support. It assumes that --reasoning-parser qwen3 works for Qwen3.5 models, even though the parser is named for Qwen3—a subtle versioning assumption.

But the most significant assumption is that the container would start successfully. Message 6661 (the immediate follow-up) reveals that the container crashed instantly: "Error response from daemon: No such container: vllm-qwen35-test". The container exited before the sleep 30 completed, leaving no logs to inspect. This failure would force the assistant to diagnose the crash—likely related to CUDA library incompatibilities, missing dependencies in the image, or an argument parsing error that only manifests with GPU access.

The Thinking Process Visible in the Message

The assistant's reasoning in this message is compressed but revealing. The opening sentence—"This vLLM version really needs GPU access to even parse its args"—shows a moment of frustration transformed into adaptive strategy. Rather than continuing to fight the tool's constraints, the assistant pivots to a more productive approach: just run the container and see what happens.

The phrase "Let me use the existing launch-cluster.sh from spark-vllm-docker which already handles all the Ray + Docker complexity" reveals the assistant's awareness of existing infrastructure. It knows the environment has pre-built tooling for multi-node deployment, and it intends to leverage that. But the actual command that follows is a simple docker run, not a cluster launch. This suggests the assistant is following a two-phase plan: first validate single-node viability, then adapt the cluster script for multi-node. The message only captures phase one.

The choice of parameters also reveals thinking: --max-model-len 4096 is relatively short for a 122B model (which could handle 32K+ context), but the assistant likely chose this to minimize memory pressure during the validation test. The --gpu-memory-utilization 0.85 similarly reflects a conservative posture—better to succeed at reduced capacity than fail at full utilization.

Knowledge Input and Output

To understand this message, a reader needs knowledge of: Docker container runtime flags (--gpus, --ipc, --ulimit), vLLM serving architecture and its CLI parameters, the Qwen3.5 model family and its reasoning output format, the DGX Spark hardware specifications (ARM64, unified memory, single GPU), and the multi-node NCCL/Ray infrastructure that exists on these machines.

The message creates new knowledge by: establishing that the hellohal2064/vllm-qwen3.5-gb10 image can be launched as a container with the model mounted, defining a specific set of parameters that are candidates for the production deployment, and documenting the assistant's intent to use the launch-cluster.sh infrastructure. It also implicitly creates a test result—the container's crash—which becomes input knowledge for the next diagnostic phase.

Conclusion

Message 6660 represents a critical inflection point in the deployment saga. After hours of deadlock with SGLang's multi-node NCCL, the assistant pivots decisively to a community vLLM image, validates its contents, transfers it to both nodes, and launches a single-node test. The container crashes immediately, but that failure is itself valuable data—it tells the assistant that something is wrong with the image, the model path, the arguments, or the environment. The message captures the moment of action, the transition from diagnosis to experimentation, and the pragmatic decision-making that defines expert infrastructure engineering. It is a small message with outsized consequences, setting the stage for the multi-node vLLM deployment that would eventually succeed after further debugging of Ray networking, OOM killers, and NCCL configuration.