The Moment of Arrival: Deploying Qwen3.5-122B-A10B-FP8 Across Two DGX Spark Nodes

Message excerpt: "Server is UP and serving! The model loaded successfully, CUDA graphs captured, and the API is responding on port 30000 with Qwen3.5-122B-A10B-FP8 across 2x DGX Spark with TP=2."

Introduction

Message [msg 6722] is a landmark in a long and technically demanding session. After dozens of rounds spent wrestling with incompatible container images, misconfigured networking, Ray's OOM killer, and the intricate dance of multi-node tensor parallelism across two NVIDIA DGX Spark systems, the assistant finally announces success. The message is brief — a bold declaration followed by a structured todo list showing completed milestones — but it represents the culmination of a multi-hour debugging marathon. This article examines that single message in depth: why it was written, what decisions it reflects, what knowledge it presupposes, and what it creates.

The Message Itself

The assistant writes:

Server is UP and serving! The model loaded successfully, CUDA graphs captured, and the API is responding on port 30000 with Qwen3.5-122B-A10B-FP8 across 2x DGX Spark with TP=2.

This is followed by a todowrite JSON block containing a structured list of tasks with completion statuses. The tasks include stopping the old GLM service, determining the deployment approach (using the hellohal2064/vllm-qwen3.5-gb10 image with Ray multi-node), downloading the model from HuggingFace, and copying both the Docker image and model to the second Spark node. The todo list is truncated in the conversation data, but the visible entries all show "completed" status.

Why This Message Was Written

The primary motivation for this message is announcement and checkpointing. The assistant has been engaged in a complex, multi-step deployment process spanning dozens of rounds. Each round involved issuing tool calls (bash commands, file edits, scp transfers), waiting for results, and reacting. The process was fraught with failures: SGLang's multi-node NCCL initialization hung indefinitely, Ray registered the head node with an unreachable external IP, the OOM killer terminated the process during CUDA graph capture, and the PyTorch distributed backend (c10d) couldn't connect across nodes.

After each failure, the assistant diagnosed the root cause, adjusted parameters, and retried. Message [msg 6722] is the first moment where all those adjustments converge into a working system. The assistant is not just reporting status — it is declaring a milestone to both the user and itself. The todowrite block serves as a persistent state tracker, allowing the assistant to maintain awareness of progress across rounds. This is especially important in a session where individual tool calls can time out after 60 seconds, and model loading takes 12–15 minutes, creating long gaps between observable progress.

The message also serves an orienting function. After a 900-second sleep followed by a curl check (see [msg 6721]), the assistant needed to confirm that the server was genuinely operational, not just partially loaded. The phrase "CUDA graphs captured" is a critical detail: CUDA graph capture is the final, memory-intensive step before the server becomes ready to accept requests. Earlier attempts had failed precisely at this stage when Ray's memory monitor killed the process at 95% usage. By explicitly mentioning that CUDA graphs were captured, the assistant signals that the entire initialization pipeline completed successfully.

How Decisions Were Made

This message does not itself contain decision-making — it is a status announcement. However, the decisions that led to this moment are embedded in the todo list and the context. The most consequential decision was pivoting from SGLang to vLLM. The assistant initially attempted to use lmsysorg/sglang:spark for multi-node deployment, but that image was too old to support Qwen3.5. The hellohal2064/vllm-qwen3.5-gb10 image (vLLM 0.17.1rc1, specifically built for Qwen3.5 on GB10) proved to be the correct choice.

A second critical decision was forcing Ray to use the 192.168.200.x subnet for all communication. The DGX Spark nodes have two network interfaces: an external IP (10.1.230.180) reachable from the outside, and an InfiniBand/RoCE subnet (192.168.200.x) for inter-node communication. Ray's auto-detection chose the external IP, which the worker node could not reach. The assistant fixed this by passing --node-ip-address to Ray and setting GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME to the correct RoCE interface (enp1s0f1np1).

A third decision was disabling Ray's OOM monitor with RAY_memory_monitor_refresh_ms=0. The 119B FP8 model consumes approximately 85% of the available GPU memory across two 120GB unified-memory DGX Sparks. CUDA graph capture requires additional temporary memory, pushing the system above Ray's default 95% threshold. Rather than reducing --gpu-memory-utilization further (which would limit KV cache capacity and hurt throughput), the assistant chose to disable the OOM killer entirely — a pragmatic tradeoff that trusts the GPU memory allocator to handle transient spikes.

Assumptions Made

This message makes several implicit assumptions. First, it assumes that the model is functionally correct — that the FP8 quantization, the MoE architecture, and the tensor-parallel split across two nodes produce valid outputs. This assumption is validated in the subsequent message ([msg 6723]) where a test query confirms the model correctly identifies itself as Qwen3.5 and generates coherent text.

Second, it assumes that the InfiniBand interconnect is reliable enough for production inference. The NCCL configuration uses NET/IBext_v11, which is the RoCE (RDMA over Converged Ethernet) transport. While this worked for model loading and the initial request, sustained throughput under load was not yet tested.

Third, it assumes that the Ray cluster will remain stable without the memory monitor. Disabling the OOM killer means that if a genuine memory leak occurs, the process will crash rather than being gracefully terminated. This is acceptable for a benchmarking session but would need revisiting for a production deployment.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption was that SGLang's multi-node NCCL initialization would work out of the box. The assistant spent considerable effort trying to make SGLang work before pivoting to vLLM. This was not a mistake per se — SGLang had been the preferred serving framework throughout the session — but it reflects an underestimation of the compatibility gap between the SGLang Spark image and the Qwen3.5 model architecture.

Another incorrect assumption was that Ray's auto-detected IP would be reachable from the worker node. The head node registered as 10.1.230.180, which is the external IP of the first DGX Spark. However, the second Spark only has connectivity to the first via the 192.168.200.x subnet. This caused the PyTorch distributed backend (c10d) to fail with "The client socket has failed to connect to any network address." The fix required multiple iterations: first setting VLLM_HOST_IP, then restructuring the launch script to use per-node IPs, and finally forcing Ray's --node-ip-address to the IB subnet.

A subtle mistake was the initial approach to VLLM_HOST_IP. The assistant set it to 192.168.200.12 for both nodes, but Ray was still using 10.1.230.180 for the head node's registration. This mismatch caused vLLM to look for a node:192.168.200.12 placement group that didn't exist. The fix required understanding that Ray's node registration and vLLM's placement group constraints must agree on the IP address.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains:

  1. Multi-node LLM serving architecture: Understanding that tensor parallelism (TP=2) splits model layers across two GPUs, requiring high-bandwidth interconnects and coordinated weight loading.
  2. Ray cluster fundamentals: Ray is a distributed computing framework that manages resources across nodes. The head node coordinates the cluster, while worker nodes provide compute. Placement groups allow requesting resources on specific nodes.
  3. NCCL and distributed communication: NCCL (NVIDIA Collective Communications Library) handles GPU-to-GPU communication. The NET/IBext_v11 transport indicates InfiniBand over RoCE. GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME control which network interface is used for distributed communication.
  4. CUDA graphs: A CUDA graph captures a sequence of GPU operations for optimized replay. In vLLM, CUDA graphs are used to accelerate the prefill and decode steps, but capturing them requires additional GPU memory.
  5. DGX Spark hardware: Each DGX Spark has a single NVIDIA GB10 GPU (SM121 Blackwell architecture) with 120GB of unified memory, connected via InfiniBand RoCE. The ARM Cortex-X925 CPU and the specific CUDA 13.2 environment are relevant for compatibility.
  6. Qwen3.5 model architecture: Qwen3.5-122B-A10B-FP8 is a Mixture-of-Experts model with 122B total parameters and 10B active parameters per token, quantized to FP8. The FP8 format reduces memory and bandwidth requirements compared to BF16.

Output Knowledge Created

This message creates several important outputs:

  1. A verified deployment configuration: The combination of hellohal2064/vllm-qwen3.5-gb10 image, Ray cluster with --node-ip-address on the IB subnet, disabled OOM monitor, and TP=2 across two nodes is now a proven recipe for running Qwen3.5-122B-A10B-FP8 on DGX Spark hardware.
  2. A launch script: The spark-vllm-qwen35.sh script (referenced in the todo list) encapsulates the entire deployment process, making it reproducible. This script handles Ray head/worker startup, NCCL configuration, and vLLM server launch.
  3. A baseline for benchmarking: With the server operational, the user can now request performance benchmarks ([msg 6725] asks "benchmark?"). The message establishes that the system is ready for load testing.
  4. Confirmation of multi-node viability: The successful deployment proves that two DGX Spark nodes can jointly serve a 122B-parameter model with tensor parallelism, despite the challenges of networking and memory management.

The Thinking Process

The thinking process visible in this message is one of systematic verification. The assistant does not simply declare the server "UP" — it provides three specific pieces of evidence: model loading succeeded, CUDA graphs were captured, and the API responds on port 30000. Each of these represents a stage where previous attempts had failed.

The todo list reveals the assistant's mental model of the deployment as a sequence of discrete, completable tasks. The todowrite mechanism is a form of externalized working memory, allowing the assistant to track progress across rounds without relying on conversation history alone. This is particularly valuable in a session with long timeouts and asynchronous operations.

The structure of the message — bold announcement followed by structured data — reflects a dual audience: the human user who needs a clear status update, and the assistant's own process management system. The todo list serves as both a record of accomplishment and a guide for what comes next.

Conclusion

Message [msg 6722] is a moment of arrival after a journey through networking hell, memory management puzzles, and compatibility chasms. It is deceptively simple — a few lines of text and a JSON block — but it encodes hours of debugging, half a dozen critical decisions, and a deep understanding of distributed LLM serving infrastructure. The message does not explain the journey; it only marks the destination. But for anyone who has followed the preceding rounds, the weight of that simple declaration — "Server is UP and serving!" — is immense. It is the sound of a complex system clicking into place, and the quiet before the next challenge begins.