The Full Circle: Deployment, Benchmarking, and Discovery in Multi-Node Qwen3.5 Inference

Introduction

The deployment of a 122-billion-parameter language model across two NVIDIA DGX Spark systems represents a remarkable engineering achievement — but the story does not end when the server starts serving tokens. In the final stretch of this opencode session, the assistant navigates a complete lifecycle: from the triumphant moment of a successful multi-node model load, through methodical benchmarking that reveals the hardware's true character, to the sobering discovery that infrastructure state is never as stable as it seems. This article synthesizes the work across the final messages of this chunk, tracing the arc from deployment triumph through performance evaluation to operational reality.

The Moment of Arrival: A Multi-Node Model Comes to Life

The central achievement of this chunk is the successful deployment of Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes connected via InfiniBand RoCE. After a grueling debugging session that spanned networking mismatches, Ray OOM killers, and framework incompatibilities, the assistant finally achieved a working deployment. Message 6722 announced the milestone: "Server is UP and serving! The model loaded successfully, CUDA graphs captured, and the API is responding on port 30000 with Qwen3.5-122B-A10B-FP8 across 2x DGX Spark with TP=2" [163].

This declaration was not made lightly. The path to this moment required solving six distinct infrastructure challenges, each of which could have derailed the entire deployment [165]:

  1. SGLang incompatibility: The official lmsysorg/sglang:spark image was too old to support Qwen3.5, forcing a pivot to vLLM via the community image hellohal2064/vllm-qwen3.5-gb10.
  2. Multi-node NCCL hangs: SGLang's multi-node NCCL initialization hung indefinitely, requiring a switch to vLLM + Ray for distributed orchestration.
  3. Ray IP registration: Ray auto-detected the head node's external IP (10.1.230.180), which was unreachable from the second Spark. The fix required forcing --node-ip-address to the InfiniBand subnet (192.168.200.x).
  4. Cross-node connectivity: The worker node couldn't reach the head on the external IP, requiring all distributed communication to be forced through the enp1s0f1np1 RoCE interface via GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME.
  5. Ray OOM killer: Ray's memory monitor (95% threshold) killed the vLLM process during CUDA graph capture — a particularly insidious failure because it happened after the model had loaded successfully. The fix required RAY_memory_monitor_refresh_ms=0 to disable the OOM killer and reduced --gpu-memory-utilization to leave headroom.
  6. vLLM version compatibility: vLLM 0.14 lacked Qwen3.5 support, requiring the community image with vLLM 0.17.1rc1. The orchestration of the final successful launch was itself a technical achievement. Message 6720 [161] shows a carefully constructed shell command that copies the updated launch script to both nodes, stops any existing containers, starts the Ray head on the first Spark, starts the Ray worker on the second, waits for cluster formation, and launches the vLLM serve process via nohup in the background. The command's structure reveals the assistant's deep understanding of the deployment lifecycle: cleanup before creation, sequential dependency management, and background execution for long-running processes.

The 15-Minute Wait: From Active Debugging to Passive Monitoring

After launching the serve process, the assistant entered a period of suspended anticipation. Message 6721 [162] captures this transition: "All launched. Now I need to wait for model loading (~12 minutes) and CUDA graph capture." The assistant issued a single bash command with a 900-second sleep followed by a compound health check — a deliberate design that minimized network round-trips while providing a definitive answer after the expected loading time.

The 900-second sleep was not arbitrary. The assistant had empirically measured the model loading time in a previous attempt, observing approximately 18 seconds per safetensors shard. With 39 shards total, this yields roughly 702 seconds (~12 minutes) for weight loading alone. Adding CUDA graph capture time — which had caused the OOM failure previously — the assistant conservatively estimated 15 minutes. This calibrated patience reflects a learned understanding of the system's performance characteristics.

The output from the server log, visible in the truncated tool output, showed API routes being registered — a positive signal that the server had passed model loading and was in the final stages of startup. The registered routes included /v1/messages, /v1/messages/count_tokens, /inference/v1/generate, /scale_elastic_ep, and /is_scaling_elastic_ep. The last two are particularly notable: they indicate this vLLM build supports dynamic expert scaling, a relatively advanced feature for MoE models.

Verification: The Model is Alive and Reasoning

The true functional test came in message 6723 [164], where the assistant issued a simple curl request to the chat completions endpoint:

curl -s http://localhost:30000/v1/chat/completions ... | python3 -m json.tool

The prompt was deliberately minimal: "Hello! What model are you? Reply briefly." The response confirmed everything: "I am Qwen3.5, the latest large language model developed by Tongyi Lab. How can I assist you today?"

This single curl command represents the culmination of an extraordinarily complex deployment effort. Every token in that response traveled across a physical InfiniBand link between two machines, through NCCL's collective communication primitives, through tensor parallelism sharding, through CUDA graph-optimized kernels on Blackwell GPUs. The simplicity of the output belies the complexity of the system that produced it.

The assistant showed deep technical insight by correctly interpreting the initial content: None response not as an error, but as the model correctly outputting its reasoning process into the separate reasoning field. This confirmed that the reasoning parser was active and working as intended — a critical validation for a reasoning model like Qwen3.5.

Benchmarking: The Memory-Bound Reality Check

The user's response to the successful deployment was characteristically concise: "benchmark?" [166]. This single-word query marked the transition from deployment to evaluation, from infrastructure to performance. The assistant responded by writing a benchmark script, copying it to the Spark head node, and executing it with concurrency levels of 1, 2, 4, 8, and 16 concurrent requests.

The benchmark results [170] revealed the true character of the DGX Spark hardware:

| Concurrency | Aggregate tok/s | Per-request tok/s | Avg TTFT | |-------------|-----------------|-------------------|----------| | 1 | 26 | 27 | 0.48s | | 2 | 41 | 21 | 0.86s | | 4 | 71 | 19 | 1.48s | | 8 | 94 | 12 | 2.29s | | 16 | 144 | 10 | 3.20s |

At single-request throughput of ~27 tok/s, the DGX Spark setup is clearly not competitive with discrete GPU configurations. The assistant's analysis [171] correctly identified the root cause: unified memory bandwidth. The DGX Spark's LPDDR5x memory, shared between CPU and GPU, has significantly lower bandwidth than the GDDR7 memory found in discrete GPUs like the RTX PRO 6000 Blackwell. During autoregressive decoding, the dominant cost is loading model weights from memory to compute units for each token. With unified memory, this bandwidth is shared with the CPU and system, creating a bottleneck that manifests clearly in the benchmark data.

The assistant provided three points of comparison that illuminated the results:

  1. DGX Spark (this benchmark): 27 tok/s single-request — characterized as "reasonable" for a 122B MoE model in FP8 on this hardware.
  2. RTX PRO 6000 setup: 277 tok/s with MTP speculation — roughly 10x faster, attributed to the fundamentally different memory architecture (discrete GDDR7 vs unified LPDDR5x).
  3. Community benchmark: ~75 tok/s for GPT-OSS-120B with MXFP4 — providing external validation that the Spark's performance is in the expected range, with the note that MXFP4 quantization would be even more memory-efficient. The assistant's analysis demonstrated sophisticated systems thinking: it didn't just dump numbers but connected the hardware characteristic (LPDDR5x bandwidth) to the observed behavior (memory-bandwidth-bound decode), providing a diagnosis that would inform future infrastructure decisions.

The Relaunch Request: When Infrastructure State Surprises You

The user's next command was even more concise: "relaunch the models" [172]. Two words that, within the context of the conversation, carried the full weight of the multi-system topology — both the DGX Spark pair and the RTX PRO 6000 LXC container.

The assistant's response [173] was a masterclass in operational reasoning. It first articulated its understanding: "On the Sparks: Qwen3.5-122B-A10B-FP8 via vLLM. On the RTX PRO 6000 (LXC container): Qwen3.5-122B-A10B BF16 via SGLang with MTP." Then it issued parallel health checks to both systems — a deliberate design choice enabled by the tool-calling architecture.

The results were alarming. The RTX PRO 6000 host at 10.1.230.174 returned "No route to host" — a network-level failure indicating the machine was completely unreachable. The Spark endpoint's health check returned ambiguous results.

The assistant then pivoted to infrastructure-level diagnostics [174], SSHing into the head Spark node and running docker ps to see what containers were actually running. The output revealed a critical finding:

vllm_node       Up 24 minutes
reranker        Up 24 minutes
vllm_embeddings Up 24 minutes
---
vllm_node       Up 24 minutes

The Qwen3.5 containers were gone. Instead, the old GLM-based services (vllm_node, reranker, vllm_embeddings) were running, having been started approximately 24 minutes prior. This was the same GLM-4.7-Flash setup that the assistant had stopped earlier in the session to free GPU memory for the Qwen3.5 deployment. Something — likely an auto-restart mechanism, a systemd service, or a Docker restart policy — had resurrected the old containers and replaced the Qwen3.5 deployment.

This discovery fundamentally changed the assistant's understanding of the system state. The API endpoint checks that returned empty results were now explained: the old GLM containers were running on a different port or with different API paths than the Qwen3.5 deployment. The assistant wasn't dealing with a crashed service — it was dealing with a replaced service.

The Full Circle: Lessons in Infrastructure Management

The arc of this chunk — from deployment triumph through benchmarking to the discovery of replaced containers — encapsulates several fundamental lessons about managing complex AI serving infrastructure:

1. Deployment is never truly "done." The moment a service is running, entropy begins to accumulate. Auto-restart mechanisms, system updates, and external interventions can silently alter the state of a system. The assistant's discovery that the old GLM containers had returned is a vivid reminder that infrastructure requires continuous verification, not just initial setup.

2. Verification must be multi-layered. The assistant's initial health check (querying the API endpoint) was correct but insufficient. When that failed, dropping down to docker ps — the container runtime's equivalent of "is it plugged in?" — revealed the true state of the system. This layered diagnostic approach is essential for distributed systems where failures can occur at any level of the stack.

3. Benchmarking reveals character, not just numbers. The benchmark results didn't just tell the user how fast the model was — they revealed the fundamental bottleneck of the DGX Spark architecture. The assistant's analysis connecting LPDDR5x bandwidth to decode throughput demonstrated that meaningful benchmarking requires understanding the hardware's architectural constraints.

4. Context enables compression. The user's two-word command "relaunch the models" and the assistant's correct interpretation demonstrate how shared context enables extraordinarily efficient communication. After hundreds of messages of collaboration, the user can communicate intent with maximum efficiency, trusting the assistant's comprehensive understanding of the system state.

5. The best-laid plans meet infrastructure reality. The assistant's carefully orchestrated deployment, verified through multiple rounds of debugging and confirmed by a successful benchmark, was silently undone by an automated process. This is not a failure of the deployment but a feature of complex systems: they have state, and that state changes in ways that operators must continuously discover.

Conclusion

This chunk of the opencode session captures the complete lifecycle of a multi-node AI model deployment: from the hard-won moment of a successful launch, through the revealing light of performance benchmarking, to the sobering discovery that infrastructure state is never static. The assistant demonstrated deep technical expertise across networking, memory management, distributed systems, and performance analysis. But perhaps the most important lesson is the simplest: in distributed AI serving, the work is never finished. Every successful deployment is just the beginning of a continuous cycle of verification, monitoring, and maintenance — a cycle that the assistant navigated with skill, transparency, and systematic reasoning throughout this session.