The Moment of Launch: Orchestrating Multi-Node vLLM Across Two DGX Spark Systems

In the span of a single bash command, an AI assistant transitions from troubleshooting to execution. Message [msg 6670] captures this precise inflection point: the moment when all the preparatory work — the image discovery, the connectivity tests, the single-node validation, the flag corrections — crystallizes into a single decisive action. The assistant launches a multi-node vLLM cluster across two NVIDIA DGX Spark systems, deploying the Qwen3.5-122B-A10B-FP8 model with tensor parallelism across InfiniBand interconnect. This message is not merely a command invocation; it is a carefully weighted decision that reflects hours of accumulated debugging, architectural pivots, and infrastructure understanding.

The Context That Made This Message Necessary

To understand why this message was written, one must trace the path that led here. The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a massive 119-billion-parameter mixture-of-experts model in FP8 quantization — across two DGX Spark nodes (NVIDIA's ARM-based GB10 systems with 120GB unified memory each). The original approach used SGLang, but multi-node NCCL initialization consistently hung at channel setup, never progressing to weight loading ([msg 6646], [msg 6647]). After extensive debugging, the assistant pivoted to vLLM, discovering the hellohal2064/vllm-qwen3.5-gb10 Docker image — a community build specifically optimized for Qwen3.5 on DGX Spark's SM121 Blackwell architecture ([msg 6650]).

A single-node test confirmed the image could resolve and begin loading the model ([msg 6665]), but the real challenge was multi-node orchestration. The DGX Spark systems already had a launch-cluster.sh script from a previous spark-vllm-docker setup, designed to handle Ray-based multi-node deployment with NCCL over InfiniBand. The assistant's first attempt to use this script failed: the IMAGE_NAME environment variable override was silently ignored by the script, which defaulted to the vllm-node image instead of the new hellohal2064/vllm-qwen3.5-gb10 image ([msg 6668]). Only after discovering the -t flag for specifying the image tag did the configuration check confirm the correct image ([msg 6669]).

Message [msg 6670] is the direct consequence of that discovery. The brief preamble — "Now with -t flag the image is correct. Let me launch the cluster" — encapsulates the entire motivation. It is a statement of resolved precondition, followed by the action that precondition enables.

The Anatomy of the Command

The bash command itself is a dense bundle of decisions, each parameter reflecting a specific requirement or lesson learned from prior failures:

ssh aurora@10.1.230.180 'cd /home/aurora/spark-vllm-docker && nohup ./launch-cluster.sh \
  -t hellohal2064/vllm-qwen3.5-gb10 \
  -n 192.168.200.12,192.168.200.13 \
  --name vllm_qwen35 \
  --nccl-debug WARN \
  exec vllm serve /models/Qwen3.5-122B-A10B-FP8 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
  > /home/aurora/vllm-qwen35-cluster.log 2>&1 &'

The -t flag specifies the custom image, directly addressing the previous failure where the script defaulted to the wrong image. The -n parameter lists both nodes by their InfiniBand subnet IPs (192.168.200.x), a critical detail: earlier attempts with SGLang had struggled because Ray auto-detected the external IP (10.1.230.180), which was unreachable from the worker node. Forcing the IB subnet IPs was a lesson learned from those failures.

The --tensor-parallel-size 2 setting distributes the model across both GPUs (one per node), which is essential because the 119GB FP8 model cannot fit in a single DGX Spark's 120GB unified memory when accounting for OS overhead and CUDA context. The --gpu-memory-utilization 0.90 allows 90% of GPU memory to be used, a relatively aggressive setting that reflects confidence in the model's memory footprint.

The --max-model-len 32768 is an interesting choice. Earlier single-node tests used 4096 or 8192 ([msg 6660], [msg 6664]), but the multi-node launch jumps to 32768. This likely reflects the assistant's understanding that with two nodes and TP=2, the available KV cache memory doubles, enabling longer context windows. It also suggests the assistant is targeting the model's full capability rather than just a minimal viable deployment.

The reasoning and tool-calling flags — --reasoning-parser qwen3, --tool-call-parser qwen3_coder, --enable-auto-tool-choice — indicate that the assistant is configuring the model for its intended production use case: a reasoning-capable model that can invoke tools. This is not a simple text-generation deployment; it is setting up an agent-capable inference endpoint.

Assumptions Embedded in the Launch

Every deployment command carries assumptions, and this one is no exception. The assistant assumes that:

1. The launch-cluster.sh script will correctly handle Ray orchestration. This is a reasonable assumption given that the script was designed for this exact purpose and the configuration check passed, but it remains untested with the new image.
2. The InfiniBand interconnect will sustain tensor-parallel communication. The assistant relies on NCCL over NET/IBext_v11 (as seen in earlier NCCL debug output at [msg 6646]), assuming the RoCE (RDMA over Converged Ethernet) interfaces are properly configured for the high-bandwidth, low-latency communication that TP=2 requires.
3. The vLLM 0.17.1rc1 image will correctly handle the Qwen3.5 model's architecture. The single-node test showed the model resolving to Qwen3_5MoeForConditionalGeneration and beginning weight loading ([msg 6665]), but it never completed loading before being killed. The assistant assumes that the full loading and inference pipeline works correctly.
4. The model path is consistent across nodes. The command references /models/Qwen3.5-122B-A10B-FP8, which was mounted via Docker volume bind in the single-node test. The launch-cluster.sh script presumably handles volume mounting, but the assistant assumes both nodes have the model at the same path.
5. The timeout of 30 seconds is acceptable. The bash command itself timed out after 30 seconds (as indicated by the <bash_metadata> block), but the assistant used nohup and backgrounding (&) to ensure the cluster launch continues independently. The timeout is not a failure — it simply means the SSH session returned before the script completed, which is expected for a long-running orchestration task.

Potential Issues and Incorrect Assumptions

While the command is well-constructed, several assumptions could prove incorrect. The most significant risk is Ray's OOM killer. In earlier multi-node attempts with SGLang, the assistant encountered issues where Ray's memory monitor killed processes during CUDA graph capture ([msg 6650] context). The assistant does not set RAY_memory_monitor_refresh_ms=0 in this command, which could lead to the head node's Ray process being killed when memory pressure spikes during model loading.

Another concern is NCCL P2P compatibility. The assistant had previously encountered P2P DMA corruption issues under SEV-SNP IOMMU on the Proxmox-hosted RTX PRO 6000 systems ([segment 40]), requiring NCCL_P2P_DISABLE=1. The DGX Spark systems may have different IOMMU configurations, but the assistant does not explicitly set any NCCL P2P flags here, relying on the script's default NCCL configuration.

The --trust-remote-code flag is also notable. This flag allows execution of custom model code from HuggingFace, which is necessary for Qwen3.5's custom model implementation. However, it represents a security assumption — that the model's code is safe to execute. In a production deployment, this would typically be reviewed more carefully.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

• Multi-node distributed inference: Understanding why tensor parallelism across two nodes requires careful NCCL configuration, InfiniBand networking, and Ray orchestration.
• DGX Spark architecture: Knowing that these are ARM-based systems (Cortex-X925) with 120GB unified memory, SM121 Blackwell GPUs, and RoCE/InfiniBand interconnect.
• vLLM deployment patterns: Familiarity with vLLM's command-line flags, the distinction between --tensor-parallel-size (across GPUs within a node or across nodes via Ray) and pipeline parallelism, and the reasoning/tool-calling parsers.
• The Qwen3.5 model family: Understanding that Qwen3.5-122B-A10B-FP8 is a mixture-of-experts model with 122B total parameters, 10B active parameters per token, using FP8 quantization, and that it produces separate reasoning and content fields in its output.
• Ray's role in multi-node vLLM: Knowing that vLLM uses Ray as the distributed runtime for multi-node deployments, handling worker discovery, task scheduling, and fault tolerance.
• The previous debugging history: Understanding why the IB subnet IPs are used instead of the external IP, why the -t flag was necessary, and why SGLang was abandoned in favor of vLLM.

Output Knowledge Created

This message creates several forms of knowledge:

1. A deployable multi-node inference cluster: The immediate output is a running vLLM server across two DGX Spark nodes, serving the Qwen3.5-122B-A10B-FP8 model. This is the primary artifact.
2. A validated deployment recipe: The combination of hellohal2064/vllm-qwen3.5-gb10 image, launch-cluster.sh with -t flag, and the specific vLLM arguments constitutes a reproducible deployment pattern for Qwen3.5 on DGX Spark.
3. A benchmarkable configuration: The specific parameters (TP=2, max-model-len=32768, gpu-memory-utilization=0.90) define a configuration point that can be measured against others (e.g., TP=1, different max lengths) to understand scaling characteristics.
4. Evidence of vLLM's multi-node viability on DGX Spark: Given the earlier SGLang failures, this successful launch (if it succeeds) demonstrates that vLLM is the correct inference engine for multi-node deployments on this hardware.

The Thinking Process Visible in the Message

The preamble — "Now with -t flag the image is correct" — reveals the assistant's mental model. It is not simply executing a command; it is explicitly verifying a precondition that was previously incorrect. This is a hallmark of systematic debugging: identify the failure mode, fix the root cause, verify the fix, then proceed.

The structure of the command itself reflects deliberate decision-making. Each flag was chosen based on prior experience:

• --tensor-parallel-size 2 was chosen over pipeline parallelism because the model is too large for a single node's memory, and TP distributes both memory and computation evenly.
• --gpu-memory-utilization 0.90 was chosen over lower values (like 0.85 used in the single-node test) because the assistant learned from the single-node test that the model needs most of the available memory, and with TP=2, each node holds half the model.
• --max-model-len 32768 was increased from the earlier 4096/8192 values because the assistant understands that TP=2 doubles the KV cache capacity.
• --reasoning-parser qwen3 and --tool-call-parser qwen3_coder were included because the assistant knows the model supports these features and the deployment is intended for agentic use cases. The use of nohup and backgrounding with log redirection shows the assistant's understanding that this is a long-running operation that should not block the SSH session. The timeout metadata confirms this: the command was intentionally designed to outlive the SSH connection.

The Broader Significance

This message represents a critical transition in the deployment lifecycle. Up to this point, the assistant has been in a diagnostic and preparation phase — testing images, verifying connectivity, fixing flags. Message [msg 6670] is the commitment point, where the assistant commits to a specific configuration and launches it into production. The subsequent messages ([msg 6671] onward) will reveal whether the assumptions hold, whether the cluster forms correctly, and whether the model serves requests as expected.

In the broader narrative of the coding session, this message is the culmination of a multi-hour effort to deploy a cutting-edge reasoning model across a novel hardware platform. The DGX Spark with its ARM CPU and Blackwell GPU represents a new architecture, and getting inference to work reliably across two of them requires solving problems that are not well-documented. The assistant's systematic approach — test single-node first, verify image compatibility, fix orchestration flags, then launch multi-node — is a textbook example of distributed systems deployment methodology.

The message also illustrates the importance of understanding the tools you're working with. The launch-cluster.sh script's -t flag was not obvious from the first attempt (which used IMAGE_NAME environment variable), and discovering the correct invocation required reading the script's interface carefully. This attention to detail — noticing that the configuration check showed the wrong image, then finding the correct flag — is what separates a successful deployment from another hanging NCCL initialization.

Ultimately, message [msg 6670] is a testament to the value of persistence in infrastructure engineering. Each failure — the SGLang hangs, the wrong image, the single-node OOM — contributed knowledge that fed into this command. The assistant did not guess at parameters; it earned them through debugging.