The Dual DGX Spark Deployment: From Reconnaissance to Production Inference on Blackwell ARM

Introduction

In the sprawling opencode session spanning dozens of segments and hundreds of messages, few transitions are as abrupt — or as consequential — as the pivot captured in message [msg 6553]. After an extended optimization campaign tuning speculative decoding parameters on a Proxmox-hosted RTX PRO 6000 Blackwell setup, the user issues a stark directive: "New mission: same model to deploy, different hardware — 2x GB10 dgx sparks." With those words, the conversation shifts from the familiar terrain of x86 servers with discrete VRAM to the uncharted waters of NVIDIA's DGX Spark — a compact ARM-based Blackwell system with unified memory, InfiniBand interconnects, and a host of novel constraints that would challenge even an experienced ML engineer.

This article traces the full arc of that deployment: the reconnaissance phase that mapped the hardware landscape, the strategic reasoning that guided the approach, the infrastructure archaeology that uncovered existing assets, the multi-node deployment that achieved ~27 tok/s single-request throughput, the post-reboot recovery that tested operational resilience, and the nuanced diagnostic chain that separated correct behavior from apparent failure. It is a case study in methodical infrastructure engineering — the art of understanding what exists before building something new, and the discipline of maintaining it when everything changes.

The Mission: Deploying Qwen3.5-122B Across Two DGX Sparks

The user's mission brief in [msg 6553] is a model of concise technical communication. In a single paragraph, it conveys hardware specifications (2× GB10 DGX Spark, 120 GB unified memory each, SM121 Blackwell, ARM CPU, InfiniBand interconnect), access topology (head node at 10.1.230.180, second node via SSH JumpHost at 192.168.200.13), deployment strategy (prefer file transfer over internet gateway), research methodology (use Exa MCP to find recipes), and success criteria (a working multi-node TP inference deployment serving Qwen3.5-122B-A10B-FP8 from official HuggingFace).

The DGX Spark is NVIDIA's compact personal AI supercomputer, built on the GB10 system-on-module. It pairs a Grace ARM CPU (Cortex-X925/A725 complex) with a Blackwell GPU (compute capability SM121) in a unified memory architecture where the 120 GB pool is shared between CPU and GPU — fundamentally different from the discrete VRAM model of traditional GPU servers. The InfiniBand RoCE interconnect between the two nodes provides the high-bandwidth, low-latency link essential for tensor parallelism across machines.

The assistant's response in [msg 6554] reveals a methodical, reconnaissance-first approach. Rather than jumping into deployment, the assistant formulates a structured plan with a todo list: explore both systems, search for deployment recipes, download the model, set up networking, and deploy. The reasoning section captures the key constraints: "GB10 DGX Spark: ARM CPU, 120GB unified memory each, SM121 (Blackwell), InfiniBand interconnect between them. Total: 240GB unified memory across 2 nodes. Need TP=2 across both Sparks via IB."

The Reconnaissance Phase: Mapping the Battlefield

Messages [msg 6555] through [msg 6559] constitute a textbook reconnaissance phase — parallel exploration of both target machines combined with external searches for deployment recipes. The assistant dispatches SSH probes and web searches simultaneously, maximizing information density per round.

The initial exploration in [msg 6555] reveals two systems in dramatically different states. The head Spark (spark-1a32) is running Linux 6.14.0-1015-nvidia on aarch64 with Ubuntu 24.04, but critically, 104 GB of its 119 GB unified memory is already in use — an existing vLLM deployment is consuming nearly all available resources. The second Spark (spark-5be3), identical hardware, has only 18 GB used with 70 GB free. Both share CUDA 13.0 and driver 580.95.05, confirming a modern, consistent software stack.

The deep reconnaissance in [msg 6556] drills into GPU details, disk space, Python and CUDA versions, Docker configuration, running services, InfiniBand status, network topology, and installed pip packages. The assistant discovers three running Docker containers (vllm_node, vllm_embeddings, reranker) using a vllm-node image — a 26.2 GB Docker image purpose-built for the DGX Spark. The InfiniBand interface on the second Spark shows "State: Down, Physical state: Disabled" — a red flag for multi-node tensor parallelism that will need addressing.

Message [msg 6557] investigates the existing Docker-based vLLM setup and community knowledge. The assistant fetches NVIDIA's official SGLang guide for DGX Spark and a community vLLM setup repository, while simultaneously probing the HuggingFace cache and the spark-vllm-docker directory. This dual-track reconnaissance — local state plus external knowledge — is a hallmark of mature engineering practice.

The Synthesis Moment: Strategic Reasoning Takes Shape

Message [msg 6558] is the pivot point where exploration crystallizes into action. The assistant synthesizes everything learned into a coherent mental model and launches the first concrete verification commands. The reasoning block reveals sophisticated multi-level thinking:

Hardware comprehension: The assistant correctly identifies the unified memory constraint — "GB10 has unified memory where the 120GB is shared between CPU and GPU, not dedicated VRAM." This is a crucial insight that shapes the entire memory budget calculation.

Model architecture awareness: The assistant recognizes that Qwen3.5-122B-A10B is a hybrid GDN (Mixture-of-Experts) model. From prior experience on the kpro6 setup, the assistant knows that flashinfer — the default attention backend in both SGLang and vLLM — does not support hybrid GDN architectures. This learned knowledge, applied to a new context, prevents a potentially catastrophic deployment failure.

Memory budgeting: The assistant performs a rough calculation: 122B parameters in FP8 requires approximately 122 GB. With tensor parallelism across two nodes, each node holds roughly 61 GB of weights, leaving ~59 GB per node for KV cache and overhead. Given 120 GB unified memory per node with ~15 GB OS overhead, approximately 105 GB is available — tight but feasible.

Infrastructure assessment: The assistant notes the IB link state, the second Spark's lack of internet access, and the existing launch-cluster.sh script. The initial plan is to "leverage the existing multi-node infrastructure," download the model on the head node, and distribute it to the second Spark.

The Infrastructure Archaeology: Verifying the Software Stack

Message [msg 6559] is where the assistant's assumptions meet reality. The reasoning begins with a "But wait—" moment: the assistant realizes the existing vLLM Docker image may be too old to support Qwen3.5's hybrid GDN architecture. Verification commands are dispatched to check the actual versions.

The results are revealing. The vLLM version inside the Docker container is 0.14.0rc2.dev262+g7fe255889.cu130 — a development release candidate, significantly newer than the assistant's estimate of "v0.11." The transformers library is 5.0.0rc3 (bleeding-edge), and PyTorch is 2.9.1+cu130 (a nightly build matching the CUDA toolkit). The Docker image environment includes TORCH_CUDA_ARCH_LIST=12.1a, confirming it is a Spark-optimized build targeting SM121 Blackwell compute capability.

These findings narrow the decision space considerably. The software stack is modern enough to potentially support Qwen3.5, and the primary remaining question is whether vLLM 0.14.0rc2 specifically includes the Qwen3.5 model class. The assistant's cautious approach — verifying before committing — proves justified.

The Deployment Gauntlet: Six Challenges, Six Solutions

The deployment path from reconnaissance to a working server required solving six distinct infrastructure challenges, each of which could have derailed the entire effort.

Challenge 1: SGLang Incompatibility. The assistant first attempted SGLang's official Spark image (lmsysorg/sglang:spark), but discovered it was too old to support Qwen3.5. SGLang's multi-node NCCL initialization then hung indefinitely, forcing a pivot to vLLM via the community image hellohal2064/vllm-qwen3.5-gb10 (vLLM 0.17.1rc1, specifically built for Qwen3.5 on GB10 hardware).

Challenge 2: Freeing GPU Memory. The head Spark's existing GLM-4.7-Flash container had to be stopped and its Docker containers removed to reclaim the ~98 GB of unified memory consumed by the old deployment. This required careful coordination to avoid disrupting other services.

Challenge 3: Model Distribution. The 119 GB FP8 model was downloaded from HuggingFace on the head Spark (which had internet access) and then rsynced to the second Spark over the InfiniBand link at approximately 640 MB/s — a testament to the IB interconnect's bandwidth.

Challenge 4: Ray Networking. Ray's auto-detection used the head node's external IP (10.1.230.180), which was unreachable from the second Spark on the private IB subnet. The fix required forcing node IPs to the IB subnet (192.168.200.x) using --node-ip-address and setting GLOO_SOCKET_IFNAME/NCCL_SOCKET_IFNAME to the correct RoCE interface.

Challenge 5: Ray OOM Killer. During CUDA graph capture on the head node, Ray's memory monitor (which kills processes exceeding 95% memory threshold) terminated the vLLM process. This was particularly insidious because it happened after the model had loaded successfully. The fix required RAY_memory_monitor_refresh_ms=0 to disable the OOM killer and reduced --gpu-memory-utilization to leave headroom.

Challenge 6: NCCL Over InfiniBand. The final deployment uses NCCL over NET/IBext_v11 for inter-node tensor parallelism (TP=2), achieving the necessary communication performance for distributed inference. The NCCL socket interface had to be explicitly bound to the RoCE interface to ensure all distributed communication traversed the InfiniBand link rather than the external network.

The orchestration of the final successful launch is captured in [msg 6720], which shows a carefully constructed shell command that copies the updated launch script to both nodes, stops any existing containers, starts the Ray head on the first Spark, starts the Ray worker on the second, waits for cluster formation, and launches the vLLM serve process via nohup in the background. The command's structure reveals the assistant's deep understanding of the deployment lifecycle: cleanup before creation, sequential dependency management, and background execution for long-running processes.## The Moment of Arrival: A Multi-Node Model Comes to Life

The central achievement of this segment is the successful deployment of Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes connected via InfiniBand RoCE. Message [msg 6722] announced the milestone: "Server is UP and serving! The model loaded successfully, CUDA graphs captured, and the API is responding on port 30000 with Qwen3.5-122B-A10B-FP8 across 2x DGX Spark with TP=2."

This declaration was not made lightly. The path to this moment required solving all six infrastructure challenges described above, each of which could have derailed the entire deployment. The fact that the assistant navigated them in sequence — pivoting from SGLang to vLLM when the former's multi-node NCCL hung, discovering and disabling Ray's OOM killer after a silent process termination, forcing IP addresses onto the IB subnet when auto-detection failed — is a testament to systematic debugging discipline.

The verification query in [msg 6723] confirmed everything was working correctly. The assistant issued a simple curl request to the chat completions endpoint with a deliberately minimal prompt: "Hello! What model are you? Reply briefly." The response was definitive: "I am Qwen3.5, the latest large language model developed by Tongyi Lab. How can I assist you today?"

This single curl command represents the culmination of an extraordinarily complex deployment effort. Every token in that response traveled across a physical InfiniBand link between two machines, through NCCL's collective communication primitives, through tensor parallelism sharding, through CUDA graph-optimized kernels on Blackwell GPUs. The simplicity of the output belies the complexity of the system that produced it.

Benchmarking: The Memory-Bound Reality Check

The user's response to the successful deployment was characteristically concise: "benchmark?" This single-word query marked the transition from deployment to evaluation, from infrastructure to performance. The assistant responded by writing a benchmark script, copying it to the Spark head node, and executing it with concurrency levels of 1, 2, 4, 8, and 16 concurrent requests.

The benchmark results revealed the true character of the DGX Spark hardware:

| Concurrency | Aggregate tok/s | Per-request tok/s | Avg TTFT | |-------------|-----------------|-------------------|----------| | 1 | 26 | 27 | 0.48s | | 2 | 41 | 21 | 0.86s | | 4 | 71 | 19 | 1.48s | | 8 | 94 | 12 | 2.29s | | 16 | 144 | 10 | 3.20s |

At single-request throughput of ~27 tok/s, the DGX Spark setup is clearly not competitive with discrete GPU configurations. The assistant's analysis correctly identified the root cause: unified memory bandwidth. The DGX Spark's LPDDR5x memory, shared between CPU and GPU, has significantly lower bandwidth than the GDDR7 memory found in discrete GPUs like the RTX PRO 6000 Blackwell. During autoregressive decoding, the dominant cost is loading model weights from memory to compute units for each token. With unified memory, this bandwidth is shared with the CPU and system, creating a bottleneck that manifests clearly in the benchmark data.

The assistant provided three points of comparison that illuminated the results:

1. DGX Spark (this benchmark): 27 tok/s single-request — characterized as "reasonable" for a 122B MoE model in FP8 on this hardware.
2. RTX PRO 6000 setup: 277 tok/s with MTP speculation — roughly 10x faster, attributed to the fundamentally different memory architecture (discrete GDDR7 vs unified LPDDR5x).
3. Community benchmark: ~75 tok/s for GPT-OSS-120B with MXFP4 — providing external validation that the Spark's performance is in the expected range. The assistant's analysis demonstrated sophisticated systems thinking: it didn't just dump numbers but connected the hardware characteristic (LPDDR5x bandwidth) to the observed behavior (memory-bandwidth-bound decode), providing a diagnosis that would inform future infrastructure decisions.

The Relaunch Request: When Infrastructure State Surprises You

The user's next command was even more concise: "relaunch the models" ([msg 6731]). Two words that, within the context of the conversation, carried the full weight of the multi-system topology — both the DGX Spark pair and the RTX PRO 6000 LXC container.

The assistant's response was a masterclass in operational reasoning. It first articulated its understanding: "On the Sparks: Qwen3.5-122B-A10B-FP8 via vLLM. On the RTX PRO 6000 (LXC container): Qwen3.5-122B-A10B BF16 via SGLang with MTP." Then it issued parallel health checks to both systems — a deliberate design choice enabled by the tool-calling architecture.

The results were alarming. The RTX PRO 6000 host at 10.1.230.174 returned "No route to host" — a network-level failure indicating the machine was completely unreachable. The Spark endpoint's health check returned ambiguous results.

The assistant then pivoted to infrastructure-level diagnostics, SSHing into the head Spark node and running docker ps to see what containers were actually running. The output revealed a critical finding: the Qwen3.5 containers were gone. Instead, the old GLM-based services (vllm_node, reranker, vllm_embeddings) were running, having been started approximately 24 minutes prior. This was the same GLM-4.7-Flash setup that the assistant had stopped earlier in the session to free GPU memory for the Qwen3.5 deployment. Something — likely an auto-restart mechanism, a systemd service, or a Docker restart policy — had resurrected the old containers and replaced the Qwen3.5 deployment.

This discovery fundamentally changed the assistant's understanding of the system state. The API endpoint checks that returned empty results were now explained: the old GLM containers were running on a different port or with different API paths than the Qwen3.5 deployment. The assistant wasn't dealing with a crashed service — it was dealing with a replaced service.

The Clarification: A Single Sentence Changes Everything

The missing piece arrived in [msg 6736]. The user wrote six words: "Just on the sparks, those were rebooted."

This brief message accomplishes several things simultaneously. It narrows the scope of work exclusively to the DGX Spark nodes, implicitly telling the assistant to stop worrying about the unreachable RTX PRO 6000 setup. It provides the causal explanation for the otherwise inexplicable state of the Sparks — a reboot explains why the old GLM services (configured with systemctl enable) auto-started while the manually launched Qwen3.5 containers disappeared. And it implicitly defines the next work plan: the Qwen3.5 deployment needs to be rebuilt from scratch.

The user's message corrected three assumptions the assistant was operating under: that the Sparks still had the Qwen3.5 deployment intact, that the "relaunch" request applied to both systems, and that the RTX PRO 6000 might come back online. In six words, the user eliminated all ambiguity and set the assistant on a clear path forward.

The Cleanup: Clearing the Ground

With the scope clarified, the assistant's first task was to dismantle the old GLM infrastructure that had auto-started after the reboot. The command proceeded in deliberate stages. First, it stopped four systemd services (vllm-cluster.service, vllm-proxy.service, vllm-embeddings.service, vllm-reranker.service) — the correct order of operations, ensuring systemd doesn't try to restart containers about to be deleted. Second, it force-removed the corresponding Docker containers (vllm_node, reranker, vllm_embeddings). Third, it waited three seconds and verified with nvidia-smi that GPU memory had been freed. Fourth, it SSHed into the worker node and performed an even more aggressive cleanup — removing all running containers — and verified that GPU memory was clear there too.

The output confirmed success: on the head node, GPU memory showed only 18 MiB consumed by Xorg and 6 MiB by gnome-shell — essentially zero GPU utilization by inference workloads. The worker node similarly showed only 18 MiB for Xorg. The old services were dead, the containers were gone, and the GPUs were pristine.

This cleanup phase reveals several profound truths about AI infrastructure management. The lifecycle of AI services is often more complex than their deployment — the assistant spent dozens of messages deploying Qwen3.5, but the teardown of the old GLM stack happens in a single command. System state persistence across reboots is a design decision, not a given — the old GLM stack survived because it was configured with systemctl enable, while the new Qwen3.5 stack did not because it was launched ad-hoc. And the assistant demonstrates a sophisticated understanding of operational sequencing, reflecting a mental model of dependency chains and state verification characteristic of experienced infrastructure engineers.

The Rebuild: Re-establishing the Ray Cluster

With the GPUs cleared, the assistant moved to the construction phase. The command chained four operations in a single SSH invocation to the head Spark. First, it ran the Ray head startup script inside the vllm-qwen35 Docker container, binding to 192.168.200.12 — the InfiniBand subnet IP, not the external 10.1.230.180 address. This was a hard-won fix from earlier debugging: Ray's default IP detection had picked the external interface, making the worker node unreachable. Second, it SSHed from the head node to the second Spark over the IB network and started the Ray worker, connecting back to the head at 192.168.200.12:6379. Third, it slept for eight seconds to allow Ray to fully initialize and register the worker's resources. Fourth, it queried ray status from inside the head container, filtering for active nodes and GPU resources.

The output confirmed success: two Docker container IDs printed (one per node), both Ray processes started cleanly, and the status line 0.0/2.0 GPU indicated that the cluster sees two GPUs across two nodes, with zero currently allocated to any Ray task. This is exactly the expected pre-allocation state — the GPUs are registered as available resources but not yet claimed by the vLLM engine.

This moment represents a critical inflection point. The assistant has navigated a multi-node distributed system through a reboot — one of the hardest tests of infrastructure robustness. The fact that the Ray cluster forms cleanly, with the correct networking configuration, on the first attempt after reboot, validates every fix applied in the preceding messages. The 0.0/2.0 GPU output is a quiet triumph: the cluster is ready, the GPUs are waiting, the infrastructure is sound.

The Launch: Committing to the Wait

With the Ray cluster healthy, the assistant issued the serve command. The command was deceptively brief: a single SSH invocation that launches the vLLM server via nohup in the background, with output redirected to a log file. The assistant's only comment: "Serve launched, waiting for model load (~15 min)..."

This message represents the culmination of all prior preparation — network debugging, OOM workarounds, Ray cluster formation, GPU cleanup — converging into a single act of trust. The assistant cannot verify success in real time; loading a 119 GB FP8 model across two nodes with tensor parallelism takes 10-15 minutes. The nohup and backgrounding ensure the process survives SSH session termination. The log redirection preserves output for later inspection.

The 15-minute estimate was not arbitrary. The assistant had earlier observed the model loading at approximately 18 seconds per shard across 39 shards, calculating a ~12-minute baseline for weight loading alone. Adding CUDA graph capture and server initialization, the 15-minute estimate was a reasoned upper bound — a textbook example of experiential knowledge informing operational decisions.

The choice of the shell script path was also significant. This script had been iteratively refined over the course of the session to include critical environment variables: RAY_memory_monitor_refresh_ms=0 to disable the OOM killer that had previously terminated the process during CUDA graph capture, --gpu-memory-utilization 0.85 to leave headroom, and proper NCCL socket interface bindings. By the time of this message, the script represented accumulated debugging wisdom.

The Watch: Patient Monitoring

The assistant did not simply wait passively. It implemented a structured monitoring loop that polled the vLLM server's /v1/models endpoint every 60 seconds. The loop used a for iteration with seq 1 20 to cap the total wait at approximately 20 minutes — a safety net ensuring termination even if the server never starts. The health check endpoint was carefully chosen: the OpenAI-compatible model list endpoint only returns a valid response when the entire pipeline — Ray cluster, model loading, CUDA graph capture, and API server initialization — has completed successfully.

The output told a story of patience: ten "loading" responses spanning from 15:05 to 15:15, until finally at 15:16:00 the model ID /models/Qwen3.5-122B-A10B-FP8 appeared. The model had loaded. The server was accepting requests.

This monitoring loop exemplifies the kind of operational thinking that distinguishes robust deployments from fragile ones. The assistant recognizes that waiting is not passive but active — feedback loops are essential for managing uncertainty. The simplest tools — bash, curl, Python — are often the most reliable for monitoring complex distributed systems. The inclusion of timestamps in each status line provides critical temporal information, allowing the operator to see exactly how long each phase took and detect anomalies.

The Diagnostic Chain: Decoding None

With the server confirmed live, the assistant performed a sanity check. The result was a single word: None. The model appeared to return no content at all.

This moment is where the assistant's deep understanding of model architecture separates it from a less experienced operator. A None response from a freshly deployed model could mean many things: the model is generating empty responses, the parsing is wrong, the endpoint is returning malformed JSON, or the model architecture has a bug. An inexperienced operator might immediately assume the deployment is broken and begin tearing down the setup.

But the assistant did none of these things. Instead, it performed a systematic three-step investigation.

Step 1: Quick check with max_tokens=50, extracting only the content field. Result: None. This is the trigger — the assistant now knows something is unusual.

Step 2: Full JSON inspection of the same request using python3 -m json.tool. This reveals that 50 completion tokens were generated despite content being None. The model is producing output — it's just going somewhere other than the content field.

Step 3: Targeted extraction of both reasoning and content fields with a larger token budget (200). The result confirms the hypothesis: the reasoning field contains the beginning of a thinking process, while content is still None. The model consumed all 200 tokens on its internal reasoning and never reached the output phase.

The assistant's conclusion, delivered in [msg 6744], is definitive: "Content is None because 200 tokens wasn't enough to finish the thinking block. Working correctly — reasoning parser is active, model is generating."

This diagnostic chain reveals a critical architectural detail about Qwen3.5 and modern reasoning models. These models separate their internal reasoning process from their final visible output, placing the reasoning in a dedicated reasoning field in the API response. The content field only contains the final answer after the reasoning block is complete. If the model runs out of tokens before finishing its reasoning, content stays None. This is not an error — it is correct behavior.

The assistant's ability to recognize this pattern stems from its understanding of the full deployment pipeline: the model architecture (Qwen3.5), the serving framework (vLLM 0.17.1rc1), the chat template configuration (qwen3_coder with reasoning parser), and the API response format. This holistic knowledge allows it to interpret raw API output not as a series of disconnected fields but as a coherent system where each field has a specific meaning.

The State Synchronization: Recording the Baseline

The final message in this sequence, [msg 6745], is empty on its surface — no user text at all. But in the context of the opencode session protocol, this is a powerful coordination mechanism: a request for state synchronization.

The assistant's response produced a comprehensive structured document covering goals, constraints, progress, blocked items, key decisions, next steps, critical context, and a complete file inventory across both machine clusters. This document serves as a permanent record of the project state at this point in time — a baseline that can be consulted in future rounds without re-reading the entire conversation history.

The timing is significant. The assistant had just finished relaunching the Spark deployment after a reboot, completing a full lifecycle from infrastructure failure to verified success. Recording this state now creates a reference point for all future work. The document includes hardware specifications (SM120 vs SM121, GDDR7 vs LPDDR5x), software versions (CUDA 13.0.1 vs 13.2, PyTorch 2.12.0.dev vs transformers 4.57.3), known limitations (P2P blocked, SGLang multi-node hangs on Spark), and the key decisions that explain why certain choices were made.

Lessons in Infrastructure Engineering

The DGX Spark deployment offers several enduring lessons for ML infrastructure engineering:

Reconnaissance is not optional. The assistant spent multiple rounds exploring both systems before making a single deployment decision. This investment paid dividends when the IB link was found down, the existing containers were consuming critical memory, and the vLLM version needed verification.

Assumptions must be tested. The assistant's "But wait—" moment — questioning whether the existing Docker image supported Qwen3.5's architecture — prevented a potentially catastrophic deployment failure. Every assumption about software compatibility, network configuration, and memory availability was verified before proceeding.

Community knowledge is essential. The assistant's use of Exa search to find deployment recipes, NVIDIA's official SGLang guide, and community vLLM repositories reflects an understanding that novel hardware platforms often have undocumented quirks that only early adopters have encountered.

Pivoting is a strength, not a failure. When SGLang's multi-node NCCL initialization hung, the assistant pivoted to vLLM without hesitation. When Ray's OOM killer terminated the process, the assistant found a workaround. The ability to recognize when a path is blocked and redirect effort is more valuable than stubborn persistence.

The unified memory model changes everything. The DGX Spark's shared CPU-GPU memory pool requires a fundamentally different approach to memory budgeting than discrete GPU servers. The assistant's recognition of this constraint shaped every subsequent decision about model sharding, KV cache allocation, and resource cleanup.

Systematic triage is essential. When multiple systems fail simultaneously, the operator must quickly assess what is reachable, what is not, and where effort should be concentrated. The assistant's decision to abandon the unreachable RTX PRO 6000 setup and focus on the Sparks was a pragmatic choice that maximized progress.

Cleanup is as important as deployment. The old GLM services that auto-started after the reboot had to be methodically dismantled before the new deployment could proceed. The assistant's structured cleanup — stop services, remove containers, verify memory, repeat on worker — is a template for reliable state management across multi-node systems.

Monitoring transforms waiting into observation. Rather than setting a single timer and hoping for the best, the assistant implemented a polling loop that provided real-time feedback, graceful failure handling, and automatic early termination. This is the difference between passive waiting and active observation.

Model-specific knowledge is infrastructure knowledge. The content: None response could have triggered an unnecessary debugging cycle. Instead, the assistant recognized it as correct behavior for a reasoning model under tight token constraints. Understanding the output format of the models you serve is not optional — it is a core operational requirement.

State documentation creates continuity. The comprehensive state dump at the end of the sequence ensures that the session can resume without losing context, even after a long pause or a context window limit. This is infrastructure management at scale: not just running services, but maintaining the knowledge of what is running and why.

Conclusion

The DGX Spark deployment captured in this segment represents a masterclass in methodical infrastructure engineering. From the initial reconnaissance that mapped the hardware landscape, through the strategic reasoning that identified critical constraints, to the verification that tested assumptions before committing to action, the assistant demonstrated the discipline that separates robust deployments from fragile ones.

The final achievement — ~27 tok/s single-request throughput across two ARM-based Blackwell workstations with unified memory — is a testament to both the hardware's capability and the assistant's systematic approach. It proves that large language model inference is not limited to traditional GPU servers, and that with careful engineering, even exotic hardware configurations can serve production-quality inference.

But perhaps the most important lesson is the simplest: in distributed AI serving, the work is never finished. Every successful deployment is just the beginning of a continuous cycle of verification, monitoring, and maintenance — a cycle that the assistant navigated with skill, transparency, and systematic reasoning throughout this session. The infrastructure that went dark was brought back to light — not through magic, but through the careful application of engineering discipline at every step.