From Ashes to Inference: The Full Lifecycle of Multi-Node Model Deployment on DGX Spark

Introduction

In the world of distributed AI infrastructure, the most instructive stories are rarely about clean, linear successes. They are about the messy middle — the moments when carefully assembled systems collapse, when assumptions prove wrong, when a single word like None can either signal catastrophe or confirm that everything is working exactly as designed. This article synthesizes a remarkable sequence of twelve messages from a single opencode session, tracing the complete lifecycle of deploying the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark nodes after an unexpected reboot. What emerges is a masterclass in operational discipline: systematic diagnosis, pragmatic triage, methodical cleanup, precise re-deployment, patient monitoring, and nuanced interpretation of model output.

The narrative arc spans from infrastructure failure to verified success, passing through every phase that defines professional AI infrastructure management. Along the way, it reveals how an experienced operator thinks about distributed systems, how reasoning models differ from their non-reasoning counterparts, and how the most critical skill in AI operations is not fixing things when they break — but knowing how to tell when they are actually working.

The Collapse: When Infrastructure Goes Dark

The story begins with a deceptively simple user request at message 6731: "relaunch the models." This single directive triggers a diagnostic cascade that exposes the fragility of distributed inference infrastructure. The assistant had just completed an arduous multi-hour deployment of Qwen3.5-122B-A10B-FP8 — a 119-billion-parameter mixture-of-experts model quantized to FP8 — across two DGX Spark nodes connected via InfiniBand RoCE, achieving approximately 27 tok/s single-request throughput. Benchmarks had been run, the model was correctly identifying itself, and everything was working.

Then the user asked for a relaunch.

The assistant's first action was to check both active deployments simultaneously: the DGX Spark cluster and a separate Proxmox-hosted LXC container with RTX PRO 6000 GPUs running a BF16 version of the same model. The results were immediate and troubling. The RTX PRO 6000 container at 10.1.230.174 was unreachable — ssh: connect to host ... port 22: No route to host. The assistant escalated to the Proxmox hypervisor at 10.1.2.6, attempting to use pct exec to bypass the network layer entirely, but that host was also unreachable. Meanwhile, the DGX Sparks — which should have been running the freshly deployed Qwen3.5 containers — were instead running old GLM-4.7-Flash services (vllm_node, reranker, vllm_embeddings) that had somehow returned.

As [1] documents in "When the Infrastructure Goes Dark," this moment reveals the assistant's sophisticated diagnostic reasoning. The assistant recognized the old GLM services not as a random occurrence but as a predictable pattern — services that were previously configured with systemctl enable had auto-started, while the manually launched Qwen3.5 containers had vanished. The assistant's troubleshooting followed a clear hierarchical escalation: check the service directly (curl the API), check the container status (Docker ps), check the host directly (SSH to the LXC IP), check the hypervisor (SSH to Proxmox host + pct exec). Each step narrowed the failure domain.

The Pivot: Choosing What to Save

Faced with two unreachable systems, the assistant made a critical strategic decision. As [2] captures in "The Pivot Point," message 6735 distills the situation into a single, decisive sentence: "Both the Proxmox host and the LXC are unreachable — likely a different network or the host is down. Let me focus on what I can reach: the Sparks."

This is a textbook example of triage in distributed systems engineering. The assistant does not waste time lamenting the unreachable RTX PRO 6000 setup — which, at 277 tok/s, was roughly ten times faster than the Spark deployment. It does not attempt alternative network paths, ask the user to investigate connectivity, or wait for the hosts to come back online. Instead, it works with what it has. The empty [todowrite] {} that follows is a powerful visual signal: the previous task structure is invalidated, and a new plan must be built from scratch.

But a critical piece of information was still missing: why were the Sparks running old services instead of the freshly deployed Qwen3.5? The assistant's reasoning at this point could only hypothesize — perhaps a configuration error, a manual intervention, or an automated restart mechanism.

The Clarification: A Single Sentence Changes Everything

The missing piece arrived in message 6736, as analyzed in [3] "The Power of a Single Clarifying Sentence." The user wrote six words: "Just on the sparks, those were rebooted."

This brief message accomplishes several things simultaneously. It narrows the scope of work exclusively to the DGX Spark nodes, implicitly telling the assistant to stop worrying about the unreachable RTX PRO 6000 setup. It provides the causal explanation for the otherwise inexplicable state of the Sparks — a reboot explains why the old GLM services (configured with systemctl enable) auto-started while the manually launched Qwen3.5 containers disappeared. And it implicitly defines the next work plan: the Qwen3.5 deployment needs to be rebuilt from scratch.

The user's message corrected three assumptions the assistant was operating under: that the Sparks still had the Qwen3.5 deployment intact, that the "relaunch" request applied to both systems, and that the RTX PRO 6000 might come back online. In six words, the user eliminated all ambiguity and set the assistant on a clear path forward.

The Cleanup: Clearing the Ground

With the scope clarified, the assistant's first task was to dismantle the old GLM infrastructure that had auto-started after the reboot. Message 6737, analyzed in [4] "The Cleanup Before Creation," is a single bash command that executes a carefully structured, multi-phase cleanup strategy across both Spark nodes.

The command proceeds in deliberate stages. First, it stops four systemd services (vllm-cluster.service, vllm-proxy.service, vllm-embeddings.service, vllm-reranker.service) — the correct order of operations, ensuring systemd doesn't try to restart containers about to be deleted. Second, it force-removes the corresponding Docker containers (vllm_node, reranker, vllm_embeddings). Third, it waits three seconds and verifies with nvidia-smi that GPU memory has been freed. Fourth, it SSHes into the worker node and performs an even more aggressive cleanup — removing all running containers — and verifies that GPU memory is clear there too.

The output confirms success: on the head node, GPU memory shows only 18 MiB consumed by Xorg and 6 MiB by gnome-shell — essentially zero GPU utilization by inference workloads. The worker node similarly shows only 18 MiB for Xorg. The old services are dead, the containers are gone, and the GPUs are pristine.

As [4] notes, this message reveals several profound truths about AI infrastructure management. The lifecycle of AI services is often more complex than their deployment — the assistant spent dozens of messages deploying Qwen3.5, but the teardown of the old GLM stack happens in a single command. System state persistence across reboots is a design decision, not a given — the old GLM stack survived because it was configured with systemctl enable, while the new Qwen3.5 stack did not because it was launched ad-hoc. And the assistant demonstrates a sophisticated understanding of operational sequencing, reflecting a mental model of dependency chains and state verification characteristic of experienced infrastructure engineers.

The Rebuild: Re-establishing the Ray Cluster

With the GPUs cleared, the assistant moved to the construction phase. Message 6738, documented in [5] "The Orchestrator's Pivot," re-establishes the multi-node Ray cluster that underpins the distributed inference deployment.

The command chains four operations in a single SSH invocation to the head Spark. First, it runs /home/aurora/spark-vllm-qwen35.sh head to start the Ray head node inside the vllm-qwen35 Docker container, binding to 192.168.200.12 — the InfiniBand subnet IP, not the external 10.1.230.180 address. This was a hard-won fix from earlier debugging: Ray's default IP detection had picked the external interface, making the worker node unreachable. Second, it SSHes from the head node to the second Spark over the IB network and starts the Ray worker, connecting back to the head at 192.168.200.12:6379. Third, it sleeps for eight seconds to allow Ray to fully initialize and register the worker's resources. Fourth, it queries ray status from inside the head container, filtering for active nodes and GPU resources.

The output confirms success: two Docker container IDs printed (one per node), both Ray processes started cleanly, and the status line 0.0/2.0 GPU indicates that the cluster sees two GPUs across two nodes, with zero currently allocated to any Ray task. This is exactly the expected pre-allocation state — the GPUs are registered as available resources but not yet claimed by the vLLM engine.

As [5] observes, this message represents a critical inflection point. The assistant has navigated a multi-node distributed system through a reboot — one of the hardest tests of infrastructure robustness. The fact that the Ray cluster forms cleanly, with the correct networking configuration, on the first attempt after reboot, validates every fix applied in the preceding messages. The 0.0/2.0 GPU output is a quiet triumph: the cluster is ready, the GPUs are waiting, the infrastructure is sound.

The Launch: Committing to the Wait

With the Ray cluster healthy, the assistant issued the serve command in message 6739, analyzed in [6] "The Moment of Trust." The command is deceptively brief: a single SSH invocation that launches the vLLM server via nohup in the background, with output redirected to a log file. The assistant's only comment: "Serve launched, waiting for model load (~15 min)..."

This message represents the culmination of all prior preparation — network debugging, OOM workarounds, Ray cluster formation, GPU cleanup — converging into a single act of trust. The assistant cannot verify success in real time; loading a 119 GB FP8 model across two nodes with tensor parallelism takes 10-15 minutes. The nohup and backgrounding ensure the process survives SSH session termination. The log redirection preserves output for later inspection.

The 15-minute estimate is not arbitrary. As [6] details, the assistant had earlier observed the model loading at approximately 18 seconds per shard across 39 shards, calculating a ~12-minute baseline for weight loading alone. Adding CUDA graph capture and server initialization, the 15-minute estimate was a reasoned upper bound — a textbook example of experiential knowledge informing operational decisions.

The choice of the shell script path (/home/aurora/spark-vllm-qwen35.sh serve) is also significant. This script had been iteratively refined over the course of the session to include critical environment variables: RAY_memory_monitor_refresh_ms=0 to disable the OOM killer that had previously terminated the process during CUDA graph capture, --gpu-memory-utilization 0.85 to leave headroom, and proper NCCL socket interface bindings. By the time of this message, the script represented accumulated debugging wisdom.

The Watch: Patient Monitoring

The assistant did not simply wait passively. Message 6740, documented in [7] "The Patient Watch," implements a structured monitoring loop that polls the vLLM server's /v1/models endpoint every 60 seconds. The loop uses a for iteration with seq 1 20 to cap the total wait at approximately 20 minutes — a safety net ensuring termination even if the server never starts. The health check endpoint is carefully chosen: the OpenAI-compatible model list endpoint only returns a valid response when the entire pipeline — Ray cluster, model loading, CUDA graph capture, and API server initialization — has completed successfully.

The output tells a story of patience: ten "loading" responses spanning from 15:05 to 15:15, until finally at 15:16:00 the model ID /models/Qwen3.5-122B-A10B-FP8 appears. The model has loaded. The server is accepting requests.

As [7] notes, this monitoring loop exemplifies the kind of operational thinking that distinguishes robust deployments from fragile ones. The assistant recognizes that waiting is not passive but active — feedback loops are essential for managing uncertainty. The simplest tools — bash, curl, Python — are often the most reliable for monitoring complex distributed systems. The inclusion of timestamps in each status line provides critical temporal information, allowing the operator to see exactly how long each phase took and detect anomalies.

The Diagnostic Chain: Decoding None

With the server confirmed live, the assistant performed a sanity check in message 6741. The result was a single word: None. The model appeared to return no content at all.

This moment, documented across [8], [9], [10], and [11], is where the assistant's deep understanding of model architecture separates it from a less experienced operator. A None response from a freshly deployed model could mean many things: the model is generating empty responses, the parsing is wrong, the endpoint is returning malformed JSON, or the model architecture has a bug. An inexperienced operator might immediately assume the deployment is broken and begin tearing down the setup.

But the assistant did none of these things. Instead, it performed a systematic three-step investigation.

Step 1 (message 6741): Quick check with max_tokens=50, extracting only the content field. Result: None. This is the trigger — the assistant now knows something is unusual.

Step 2 (message 6742): Full JSON inspection of the same request using python3 -m json.tool. This reveals that 50 completion tokens were generated despite content being None. The model is producing output — it's just going somewhere other than the content field.

Step 3 (message 6743): Targeted extraction of both reasoning and content fields with a larger token budget (200). The result confirms the hypothesis: the reasoning field contains the beginning of a thinking process ("Thinking Process:\n\n1. Analyze the Request:..."), while content is still None. The model consumed all 200 tokens on its internal reasoning and never reached the output phase.

The assistant's conclusion, delivered in message 6744, is definitive: "Content is None because 200 tokens wasn't enough to finish the thinking block. Working correctly — reasoning parser is active, model is generating."

This diagnostic chain reveals a critical architectural detail about Qwen3.5 and modern reasoning models. These models separate their internal reasoning process from their final visible output, placing the reasoning in a dedicated reasoning field in the API response. The content field only contains the final answer after the reasoning block is complete. If the model runs out of tokens before finishing its reasoning, content stays None. This is not an error — it is correct behavior.

As [11] articulates, the assistant's ability to recognize this pattern stems from its understanding of the full deployment pipeline: the model architecture (Qwen3.5), the serving framework (vLLM 0.17.1rc1), the chat template configuration (qwen3_coder with reasoning parser), and the API response format. This holistic knowledge allows it to interpret raw API output not as a series of disconnected fields but as a coherent system where each field has a specific meaning.

The State Synchronization: Recording the Baseline

The final message in this sequence, message 6745, is analyzed in [12] "The Empty Signal." On its surface, it is nothing — empty <conversation_data></conversation_data> tags. But in the context of the opencode session protocol, this is a powerful coordination mechanism: a request for state synchronization.

The assistant's response produces a comprehensive structured document covering goals, constraints, progress, blocked items, key decisions, next steps, critical context, and a complete file inventory across both machine clusters. This document serves as a permanent record of the project state at this point in time — a baseline that can be consulted in future rounds without re-reading the entire conversation history.

The timing is significant. The assistant had just finished relaunching the Spark deployment after a reboot, completing a full lifecycle from infrastructure failure to verified success. Recording this state now creates a reference point for all future work. The document includes hardware specifications (SM120 vs SM121, GDDR7 vs LPDDR5x), software versions (CUDA 13.0.1 vs 13.2, PyTorch 2.12.0.dev vs transformers 4.57.3), known limitations (P2P blocked, SGLang multi-node hangs on Spark), and the key decisions that explain why certain choices were made.

Themes and Lessons

Across this twelve-message sequence, several overarching themes emerge that are valuable for anyone managing distributed AI infrastructure.

Systematic triage is essential. When multiple systems fail simultaneously, the operator must quickly assess what is reachable, what is not, and where effort should be concentrated. The assistant's decision to abandon the unreachable RTX PRO 6000 setup and focus on the Sparks was a pragmatic choice that maximized progress.

Cleanup is as important as deployment. The old GLM services that auto-started after the reboot had to be methodically dismantled before the new deployment could proceed. The assistant's structured cleanup — stop services, remove containers, verify memory, repeat on worker — is a template for reliable state management across multi-node systems.

Monitoring transforms waiting into observation. Rather than setting a single timer and hoping for the best, the assistant implemented a polling loop that provided real-time feedback, graceful failure handling, and automatic early termination. This is the difference between passive waiting and active observation.

Model-specific knowledge is infrastructure knowledge. The content: None response could have triggered an unnecessary debugging cycle. Instead, the assistant recognized it as correct behavior for a reasoning model under tight token constraints. Understanding the output format of the models you serve is not optional — it is a core operational requirement.

State documentation creates continuity. The comprehensive state dump at the end of the sequence ensures that the session can resume without losing context, even after a long pause or a context window limit. This is infrastructure management at scale: not just running services, but maintaining the knowledge of what is running and why.

Conclusion

The twelve-message sequence spanning messages 6734 through 6745 tells a complete story of infrastructure management in the age of large language models. It begins with a system in disarray — unreachable hosts, auto-started stale services, a deployment wiped clean by a reboot. It proceeds through methodical diagnosis, strategic triage, precise cleanup, careful reconstruction, patient monitoring, and nuanced interpretation of model output. It ends with a verified, working deployment and a comprehensive state document that captures everything learned along the way.

What makes this sequence remarkable is not any single insight or command, but the cumulative demonstration of operational discipline. The assistant never panics, never jumps to conclusions, never applies a fix without understanding the root cause. It systematically works through each layer of the problem — network, services, containers, GPU memory, cluster orchestration, model loading, API response format — until every layer is verified correct.

In the end, the Qwen3.5-122B-A10B-FP8 model is serving requests across two DGX Spark nodes connected via InfiniBand, with the reasoning parser correctly separating thinking from content, and the entire deployment surviving a full system reboot. The infrastructure that went dark has been brought back to light — not through magic, but through the careful application of engineering discipline at every step.