The Dual DGX Spark Deployment: From Reconnaissance to Multi-Node Inference on Blackwell ARM

In the sprawling opencode session spanning dozens of segments and hundreds of messages, few transitions are as abrupt — or as consequential — as the pivot captured in message [msg 6553]. After an extended optimization campaign tuning speculative decoding parameters on a Proxmox-hosted RTX PRO 6000 Blackwell setup, the user issues a stark directive: "New mission: same model to deploy, different hardware — 2x GB10 dgx sparks." With those words, the conversation shifts from the familiar terrain of x86 servers with discrete VRAM to the uncharted waters of NVIDIA's DGX Spark — a compact ARM-based Blackwell system with unified memory, InfiniBand interconnects, and a host of novel constraints that would challenge even an experienced ML engineer.

This article traces the full arc of that deployment: the reconnaissance phase that mapped the hardware landscape, the strategic reasoning that guided the approach, the infrastructure archaeology that uncovered existing assets, and the eventual multi-node deployment that achieved ~27 tok/s single-request throughput with correct reasoning output. It is a case study in methodical infrastructure engineering — the art of understanding what exists before building something new.

The Mission: Deploying Qwen3.5-122B Across Two DGX Sparks

The user's mission brief in [msg 6553] is a model of concise technical communication. In a single paragraph, it conveys hardware specifications (2× GB10 DGX Spark, 120 GB unified memory each, SM121 Blackwell, ARM CPU, InfiniBand interconnect), access topology (head node at 10.1.230.180, second node via SSH JumpHost at 192.168.200.13), deployment strategy (prefer file transfer over internet gateway), research methodology (use Exa MCP to find recipes), and success criteria (a working multi-node TP inference deployment serving Qwen3.5-122B-A10B-FP8 from official HuggingFace).

The DGX Spark is NVIDIA's compact personal AI supercomputer, built on the GB10 system-on-module. It pairs a Grace ARM CPU (Cortex-X925/A725 complex) with a Blackwell GPU (compute capability SM121) in a unified memory architecture where the 120 GB pool is shared between CPU and GPU — fundamentally different from the discrete VRAM model of traditional GPU servers. The InfiniBand RoCE interconnect between the two nodes provides the high-bandwidth, low-latency link essential for tensor parallelism across machines.

The assistant's response in [msg 6554] reveals a methodical, reconnaissance-first approach. Rather than jumping into deployment, the assistant formulates a structured plan with a todo list: explore both systems, search for deployment recipes, download the model, set up networking, and deploy. The reasoning section captures the key constraints: "GB10 DGX Spark: ARM CPU, 120GB unified memory each, SM121 (Blackwell), InfiniBand interconnect between them. Total: 240GB unified memory across 2 nodes. Need TP=2 across both Sparks via IB."

The Reconnaissance Phase: Mapping the Battlefield

Messages [msg 6555] through [msg 6559] constitute a textbook reconnaissance phase — parallel exploration of both target machines combined with external searches for deployment recipes. The assistant dispatches SSH probes and web searches simultaneously, maximizing information density per round.

The initial exploration in [msg 6555] reveals two systems in dramatically different states. The head Spark (spark-1a32) is running Linux 6.14.0-1015-nvidia on aarch64 with Ubuntu 24.04, but critically, 104 GB of its 119 GB unified memory is already in use — an existing vLLM deployment is consuming nearly all available resources. The second Spark (spark-5be3), identical hardware, has only 18 GB used with 70 GB free. Both share CUDA 13.0 and driver 580.95.05, confirming a modern, consistent software stack.

The deep reconnaissance in [msg 6556] drills into GPU details, disk space, Python and CUDA versions, Docker configuration, running services, InfiniBand status, network topology, and installed pip packages. The assistant discovers three running Docker containers (vllm_node, vllm_embeddings, reranker) using a vllm-node image — a 26.2 GB Docker image purpose-built for the DGX Spark. The InfiniBand interface on the second Spark shows "State: Down, Physical state: Disabled" — a red flag for multi-node tensor parallelism that will need addressing.

Message [msg 6557] investigates the existing Docker-based vLLM setup and community knowledge. The assistant fetches NVIDIA's official SGLang guide for DGX Spark and a community vLLM setup repository, while simultaneously probing the HuggingFace cache and the spark-vllm-docker directory. This dual-track reconnaissance — local state plus external knowledge — is a hallmark of mature engineering practice.

The Synthesis Moment: Strategic Reasoning Takes Shape

Message [msg 6558] is the pivot point where exploration crystallizes into action. The assistant synthesizes everything learned into a coherent mental model and launches the first concrete verification commands. The reasoning block reveals sophisticated multi-level thinking:

Hardware comprehension: The assistant correctly identifies the unified memory constraint — "GB10 has unified memory where the 120GB is shared between CPU and GPU, not dedicated VRAM." This is a crucial insight that shapes the entire memory budget calculation.

Model architecture awareness: The assistant recognizes that Qwen3.5-122B-A10B is a hybrid GDN (Mixture-of-Experts) model. From prior experience on the kpro6 setup, the assistant knows that flashinfer — the default attention backend in both SGLang and vLLM — does not support hybrid GDN architectures. This learned knowledge, applied to a new context, prevents a potentially catastrophic deployment failure.

Memory budgeting: The assistant performs a rough calculation: 122B parameters in FP8 requires approximately 122 GB. With tensor parallelism across two nodes, each node holds roughly 61 GB of weights, leaving ~59 GB per node for KV cache and overhead. Given 120 GB unified memory per node with ~15 GB OS overhead, approximately 105 GB is available — tight but feasible.

Infrastructure assessment: The assistant notes the IB link state, the second Spark's lack of internet access, and the existing launch-cluster.sh script. The initial plan is to "leverage the existing multi-node infrastructure," download the model on the head node, and distribute it to the second Spark.

The Infrastructure Archaeology: Verifying the Software Stack

Message [msg 6559] is where the assistant's assumptions meet reality. The reasoning begins with a "But wait—" moment: the assistant realizes the existing vLLM Docker image may be too old to support Qwen3.5's hybrid GDN architecture. Verification commands are dispatched to check the actual versions.

The results are revealing. The vLLM version inside the Docker container is 0.14.0rc2.dev262+g7fe255889.cu130 — a development release candidate, significantly newer than the assistant's estimate of "v0.11." The transformers library is 5.0.0rc3 (bleeding-edge), and PyTorch is 2.9.1+cu130 (a nightly build matching the CUDA toolkit). The Docker image environment includes TORCH_CUDA_ARCH_LIST=12.1a, confirming it is a Spark-optimized build targeting SM121 Blackwell compute capability.

These findings narrow the decision space considerably. The software stack is modern enough to potentially support Qwen3.5, and the primary remaining question is whether vLLM 0.14.0rc2 specifically includes the Qwen3.5 model class. The assistant's cautious approach — verifying before committing — proves justified.

The Challenges That Followed

While the reconnaissance messages (6553–6559) establish the foundation, the chunk summary reveals the full scope of challenges that the deployment would encounter:

Freeing GPU memory: The head Spark's existing GLM-4.7-Flash container had to be stopped and its Docker containers removed to reclaim the ~98 GB of unified memory consumed by the old deployment. This required careful coordination to avoid disrupting other services.

Downloading and distributing the model: The 119 GB FP8 model was downloaded from HuggingFace on the head Spark (which had internet access) and then rsynced to the second Spark over the InfiniBand link at approximately 640 MB/s — a testament to the IB interconnect's bandwidth.

Pivoting from SGLang to vLLM: The assistant first attempted SGLang's official Spark image, but discovered it lacked Qwen3.5 support. SGLang's multi-node NCCL initialization then hung indefinitely, forcing a pivot to a community vLLM image (hellohal2064/vllm-qwen3.5-gb10, based on vLLM 0.17.1rc1) specifically built for Qwen3.5 on GB10 hardware.

Ray networking issues: Ray's auto-detection used the external IP (10.1.230.180) which was unreachable from the second Spark on the private IB subnet. The assistant had to force node IPs to the IB subnet (192.168.200.x) using --node-ip-address and set GLOO_SOCKET_IFNAME/NCCL_SOCKET_IFNAME to the correct RoCE interface.

Ray OOM killer: During CUDA graph capture on the head node, Ray's memory monitor (which kills processes exceeding 95% memory threshold) terminated the process. This required disabling the memory monitor with RAY_memory_monitor_refresh_ms=0 and reducing --gpu-memory-utilization.

NCCL over InfiniBand: The final deployment uses NCCL over NET/IBext_v11 for inter-node tensor parallelism (TP=2), achieving the necessary communication performance for distributed inference.

The Outcome: A Working Multi-Node Deployment

Despite these challenges, the deployment ultimately succeeded. The final configuration achieved approximately 27 tok/s single-request throughput with correct reasoning output and tool calling enabled. This is a significant achievement for a 122-billion-parameter MoE model running across two compact ARM-based workstations with unified memory, connected via InfiniBand.

The session concluded with a clean restart of the services after removing old containers, confirming the model loaded successfully and served queries with proper reasoning field extraction. The deployment demonstrated that the DGX Spark — despite its unconventional unified memory architecture and ARM CPU — is a viable platform for running large language models at the edge.

Lessons in Infrastructure Engineering

The DGX Spark deployment offers several enduring lessons for ML infrastructure engineering:

Reconnaissance is not optional. The assistant spent multiple rounds exploring both systems before making a single deployment decision. This investment paid dividends when the IB link was found down, the existing containers were consuming critical memory, and the vLLM version needed verification.

Assumptions must be tested. The assistant's "But wait—" moment in [msg 6559] — questioning whether the existing Docker image supported Qwen3.5's architecture — prevented a potentially catastrophic deployment failure. Every assumption about software compatibility, network configuration, and memory availability was verified before proceeding.

Community knowledge is essential. The assistant's use of Exa search to find deployment recipes, NVIDIA's official SGLang guide, and community vLLM repositories reflects an understanding that novel hardware platforms often have undocumented quirks that only early adopters have encountered.

Pivoting is a strength, not a failure. When SGLang's multi-node NCCL initialization hung, the assistant pivoted to vLLM without hesitation. When Ray's OOM killer terminated the process, the assistant found a workaround. The ability to recognize when a path is blocked and redirect effort is more valuable than stubborn persistence.

The unified memory model changes everything. The DGX Spark's shared CPU-GPU memory pool requires a fundamentally different approach to memory budgeting than discrete GPU servers. The assistant's recognition of this constraint shaped every subsequent decision about model sharding, KV cache allocation, and resource cleanup.

Conclusion

The DGX Spark deployment captured in messages [msg 6553] through [msg 6559] — and the challenges that followed — represents a masterclass in methodical infrastructure engineering. From the initial reconnaissance that mapped the hardware landscape, through the strategic reasoning that identified critical constraints, to the verification that tested assumptions before committing to action, the assistant demonstrated the discipline that separates robust deployments from fragile ones.

The final achievement — ~27 tok/s single-request throughput across two ARM-based Blackwell workstations with unified memory — is a testament to both the hardware's capability and the assistant's systematic approach. It proves that large language model inference is not limited to traditional GPU servers, and that with careful engineering, even exotic hardware configurations can serve production-quality inference.