The Moment of Trust: Launching a 122B-Parameter Model Across Two DGX Spark Nodes
Message Overview
The subject message ([msg 6739]) is deceptively brief — a single bash command issued by the assistant to launch the vLLM server hosting Qwen3.5-122B-A10B-FP8 across two NVIDIA GB10 DGX Spark systems:
2 nodes, 2 GPUs. Launching vLLM serve:
>
[bash] ssh aurora@10.1.230.180 'nohup /home/aurora/spark-vllm-qwen35.sh serve > /home/aurora/vllm-qwen35-serve.log 2>&1 & echo "Serve launched, waiting for model load (~15 min)..."' 2>&1 Serve launched, waiting for model load (~15 min)...
In just 47 words of assistant text and a single remote command, this message represents the culmination of a long and arduous deployment journey. It is the moment when all prior preparation — network debugging, OOM workarounds, Ray cluster formation, GPU cleanup — converges into a single act of trust: launching the server and waiting.
Context and Motivation: Why This Message Was Written
To understand why this message exists, one must trace back through the preceding conversation. The user had requested a relaunch of the models ([msg 6731]). The assistant checked both major deployments — the RTX PRO 6000 system and the two DGX Spark nodes — and found the RTX unreachable due to a network issue and the Sparks running old GLM containers instead of the Qwen3.5 service ([msg 6732]-[msg 6735]). The user clarified: "Just on the sparks, those were rebooted" ([msg 6736]).
This reboot was significant. The DGX Spark nodes had been running a previous deployment of GLM-4.7-Flash using Docker containers managed by systemd services. After the reboot, those containers auto-started, occupying the GPU memory that the Qwen3.5 deployment needed. The assistant's first task was therefore a cleanup operation: stopping the old systemd services (vllm-cluster.service, vllm-proxy.service, vllm-embeddings.service, vllm-reranker.service), force-removing the old Docker containers (vllm_node, reranker, vllm_embeddings), and verifying that both GPUs were clear of model allocations ([msg 6737]).
With the GPUs freed, the assistant then rebuilt the Ray cluster from scratch. This involved running the head node startup script on the first Spark (192.168.200.12), then SSHing to the second Spark (192.168.200.13) to start the worker node connecting back to the head. The assistant verified the cluster was healthy with ray status, confirming two GPUs were visible ([msg 6738]).
Only after this entire chain of dependencies was satisfied — GPUs clear, Ray cluster formed, networking correct — did the assistant issue the serve command in message 6739. The message is thus the final step in a carefully orchestrated deployment sequence, the "launch" button after all pre-flight checks have passed.
How Decisions Were Made
The most notable decision in this message is what is not done. The assistant does not verify that the serve process started successfully. It does not check for immediate errors, does not tail the log, does not confirm the process is running. Instead, it launches via nohup in the background, echoes a placeholder message, and moves on.
This is a deliberate architectural choice rooted in the nature of large-model deployment. Loading a 119GB FP8 model across two nodes with tensor parallelism takes on the order of 10–15 minutes. The assistant cannot verify success in real time — it must wait. The nohup and backgrounding (&) ensure the process survives the SSH session termination. The log redirection to vllm-qwen35-serve.log preserves the output for later inspection.
The assistant's estimate of "~15 min" is not arbitrary. Earlier in the session ([msg 6715]), the assistant had observed the model loading at approximately 18 seconds per shard across 39 shards, calculating a ~12-minute baseline for weight loading alone. Adding CUDA graph capture and server initialization, the 15-minute estimate was a reasoned upper bound. This estimate was refined through direct observation of the same model on the same hardware — a textbook example of experiential knowledge informing operational decisions.
The choice of the shell script path (/home/aurora/spark-vllm-qwen35.sh serve) is also significant. This script had been iteratively refined over the course of the session ([msg 6709], [msg 6710], [msg 6711], [msg 6717], [msg 6718], [msg 6719]) to include critical environment variables: RAY_memory_monitor_refresh_ms=0 to disable the OOM killer that had previously terminated the process during CUDA graph capture, --gpu-memory-utilization 0.85 to leave headroom, and proper NCCL socket interface bindings. By the time of message 6739, the script represented accumulated debugging wisdom.
Assumptions Embedded in the Message
The assistant makes several assumptions in this message, most of which are justified by prior verification but remain unstated:
The Ray cluster is correctly formed. The assistant verified this in the immediately preceding message ([msg 6738]), where ray status confirmed two active GPUs. However, the assistant does not re-verify before launching serve. If the Ray cluster had degraded in the seconds between verification and launch, the serve command would fail silently.
The model weights are present on disk. The 119GB FP8 model had been downloaded from HuggingFace and rsynced to the second Spark earlier in the session. The assistant assumes the files survived the reboot — a reasonable assumption given they reside on persistent storage, but not explicitly verified.
Networking configuration is stable. The InfiniBand RoCE interconnect between the two Sparks (192.168.200.x subnet) had been configured with NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME pointing to the correct interface (enp1s0f1np1). The assistant assumes this configuration persists in the Docker container's environment.
The shell script is identical on both nodes. The assistant had previously copied the script from the head node to the worker ([msg 6712]), but the serve command only runs on the head. The worker's copy is used for Ray startup, not for serving.
The model will load within the estimated timeframe. The 15-minute estimate is an educated guess, not a guarantee. If the model had failed to load (due to OOM, CUDA errors, or NCCL initialization failures), the assistant would not discover this until the next message's monitoring loop.
Mistakes and Incorrect Assumptions
The most notable discrepancy is the time estimate. The assistant predicted "~15 min" for model loading, but in practice the model became available in approximately 10–11 minutes. The subsequent monitoring loop ([msg 6740]) shows the model still loading at 15:14:59 and available at 15:16:00 — roughly 11 minutes after launch. This is not a significant error; the estimate was conservative, which is the correct operational posture.
A more subtle issue is that the assistant does not capture the serve process ID or implement any health check before the monitoring loop. If the process had crashed immediately (e.g., due to a configuration syntax error in the shell script), the assistant would have waited the full 15 minutes before discovering the failure. A more robust approach might have included a brief sleep followed by a pgrep or docker ps check to confirm the process was alive before beginning the long wait.
The assistant also assumes that the Docker container (vllm-qwen35) from the Ray startup phase is the correct container for serving. The shell script's serve subcommand presumably uses the same container, but this is an implicit dependency — if the container had been stopped or removed between the Ray startup and the serve launch, the command would fail.
Input Knowledge Required
To understand this message fully, one needs knowledge of:
The hardware topology. Two NVIDIA GB10 DGX Spark nodes, each with 120GB unified memory and an ARM Cortex-X925 CPU, connected via InfiniBand RoCE on the 192.168.200.x subnet. The head node is also accessible on the external network at 10.1.230.180.
The model characteristics. Qwen3.5-122B-A10B-FP8 is a 122-billion-parameter Mixture-of-Experts model with approximately 10B active parameters per token, quantized to FP8. The model checkpoint is approximately 119GB on disk, requiring tensor parallelism across both GPUs.
The software stack. vLLM 0.17.1rc1 running inside a Docker container (hellohal2064/vllm-qwen3.5-gb10), orchestrated by Ray for multi-node coordination. NCCL uses the InfiniBand RoCE interface for inter-node communication.
The prior debugging history. The session had already resolved: Ray networking issues where the external IP was unreachable from the worker node, Ray OOM killer termination during CUDA graph capture, and NCCL initialization failures. Each of these was fixed through iterative edits to the launch script.
The deployment script structure. The spark-vllm-qwen35.sh script has subcommands for head, worker, serve, and stop. The serve subcommand starts the vLLM OpenAI-compatible API server with TP=2 across the two nodes.
Output Knowledge Created
This message produces:
A running vLLM server process on the head DGX Spark node, backgrounded via nohup. The process loads the 119GB model weights across both GPUs via NCCL, captures CUDA graphs for optimized inference, and starts an OpenAI-compatible API endpoint on port 30000.
A log file at /home/aurora/vllm-qwen35-serve.log on the head node, containing the full server output including model loading progress, CUDA graph capture status, and any errors.
An implicit contract with the user: the assistant will follow up with verification. The "waiting for model load (~15 min)..." message signals that the assistant is aware of the time commitment and will check back.
A checkpoint in the deployment process. After this message, the assistant transitions from "setup" mode to "monitoring" mode. The subsequent message ([msg 6740]) implements a polling loop that checks every 60 seconds whether the model API is responsive, demonstrating the shift from active configuration to passive verification.
The Thinking Process Visible in the Message
While the message itself contains no explicit reasoning block, the thinking is visible in its structure and timing:
The brevity signals confidence. The assistant does not second-guess the configuration, does not add redundant verification, does not hedge with conditionals. It simply states the state ("2 nodes, 2 GPUs") and executes. This confidence is earned through the extensive debugging that preceded it — every networking issue, OOM crash, and NCCL hang had been identified and fixed.
The time estimate reflects experiential knowledge. The "~15 min" is not a guess but a calculation based on observed performance: 18 seconds per shard, 39 shards, plus overhead for CUDA graph capture and server initialization. The assistant had watched this exact model load before ([msg 6715]) and extrapolated.
The choice of nohup over systemd or docker run is pragmatic. The assistant is working within the constraints of an SSH session. nohup ensures the process survives SSH disconnection. The log file provides a durable record. This is infrastructure management at the command line — not elegant, but effective.
The message serves as a status update to the user. By including the human-readable "Serve launched, waiting for model load (~15 min)..." in the echoed output, the assistant communicates both the action taken and the expected timeline. The user is not left wondering what happens next.
Conclusion
Message [msg 6739] is a masterclass in operational minimalism. In one SSH command, the assistant launches a 122-billion-parameter model across two geographically distributed ARM-based AI accelerators, building on hours of prior debugging. The message's brevity is not a lack of effort but a sign of maturity — the confidence that comes from having already solved the hard problems. It is the quiet moment before the storm of verification, the point where preparation gives way to execution, and the assistant commits to a 15-minute wait with nothing but a log file and a hope that the configuration holds.