The Moment of Truth: Verifying a Multi-Node LLM Deployment Across Two DGX Sparks

On the surface, message [msg 6723] appears to be a simple act: a curl command hitting a REST API endpoint and receiving a JSON response. The assistant sends a chat completion request to a locally running vLLM server, asking "Hello! What model are you? Reply briefly." and receives back a perfectly reasonable answer: "I am Qwen3.5, the latest large language model developed by Tongyi Lab. How can I assist you today?"

But this message is anything but simple. It is the culmination of an extraordinarily complex deployment effort spanning dozens of messages, multiple failed approaches, intricate networking diagnostics, and hard-won infrastructure battles. This single curl command represents the moment when all the pieces finally clicked into place — the verification that a 122-billion-parameter FP8 model was successfully running across two NVIDIA DGX Spark nodes, connected via InfiniBand, orchestrated by Ray, and serving inference through vLLM.

The Weight of Context: Why This Message Was Written

To understand why this particular message exists, one must appreciate the journey that preceded it. The assistant had been tasked with deploying the Qwen3.5-122B-A10B-FP8 model — a massive 119GB reasoning model — across a pair of DGX Spark systems (NVIDIA's ARM-based desktop supercomputers with GB10 Blackwell GPUs and 120GB of unified memory each). This was not a simple single-node deployment. The model was too large for one Spark's GPU memory, requiring tensor parallelism across both nodes over an InfiniBand RoCE interconnect.

The preceding messages reveal a cascade of failures and recoveries. The assistant first attempted to use SGLang's official Spark image, only to discover it lacked Qwen3.5 support. A pivot to vLLM's hellohal2064/vllm-qwen3.5-gb10 image followed. Then came the Ray networking nightmare: Ray auto-detected the head node's external IP (10.1.230.180), which was unreachable from the second Spark. The worker node couldn't establish PyTorch's distributed (c10d) connections. The assistant fixed this by forcing --node-ip-address to the InfiniBand subnet (192.168.200.x), setting GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME to the correct RoCE interface, and restructuring the launch script to use per-node VLLM_HOST_IP values. Then Ray's OOM killer (triggered at 95% memory threshold) killed the process during CUDA graph capture, requiring RAY_memory_monitor_refresh_ms=0 and reduced --gpu-memory-utilization. Each fix was a battle.

By message [msg 6721], the server was finally up. The assistant confirmed the API was responding on port 30000. Message [msg 6722] declared victory: "Server is UP and serving!" But that declaration was based only on the /v1/models endpoint returning a model list — it confirmed the server was running, but not that it could actually generate tokens. Message [msg 6723] is the true functional test: can this multi-node, multi-GPU, Ray-orchestrated, InfiniBand-linked contraption actually run inference?

The Anatomy of a Verification Query

The assistant's choice of test query reveals careful thinking. The prompt "Hello! What model are you? Reply briefly." is deliberately minimal. It tests several things simultaneously:

  1. Basic generation works: The model must produce coherent text, not garbage or an error.
  2. Model identity is correct: The response should confirm the right weights were loaded.
  3. The response is brief: The instruction tests whether the model follows basic constraints.
  4. The API is OpenAI-compatible: The curl request uses the standard /v1/chat/completions endpoint with the expected JSON schema, confirming the deployment is usable by standard clients. The assistant also pipes the output through python3 -m json.tool, which pretty-prints the JSON. This is a small but telling detail — it shows the assistant expected success and wanted a clean, readable output to present as evidence. There is no error handling, no fallback, no contingency. The assistant is confident enough in the preceding work that this curl is purely ceremonial verification.

Input Knowledge Required

To fully appreciate this message, one must understand several layers of context:

Hardware topology: Two DGX Spark nodes, each with a single NVIDIA GB10 Blackwell GPU (SM121 architecture, ARM Cortex-X925 CPU), 120GB unified memory, connected via InfiniBand RoCE on a 192.168.200.x subnet. The external network is 10.1.230.x, but the Sparks cannot reach each other on that subnet — only the jump host (the Proxmox server at 10.1.230.180) has access to both.

Software stack: The hellohal2064/vllm-qwen3.5-gb10 Docker image contains vLLM 0.17.1rc1, specifically built for Qwen3.5 on GB10 hardware. Ray 2.53 handles multi-node orchestration. NCCL uses NET/IBext_v11 for inter-node tensor parallelism. The model is Qwen3.5-122B-A10B-FP8, a 122-billion-parameter model with 10B active parameters per token (MoE architecture), quantized to FP8.

The deployment script: The assistant's spark-vllm-qwen35.sh script manages the full lifecycle — starting Ray head/worker, launching vLLM serve, and stopping services. It encapsulates all the networking fixes discovered through trial and error.

The failure history: Without knowing about the Ray OOM killer, the c10d connection failures, the SGLang pivot, and the networking fixes, this message looks trivial. With that context, it reads as a triumph.

Output Knowledge Created

This message produces several important outputs:

  1. Functional verification: The model generates coherent, correct responses. The deployment is not just "running" but working.
  2. Model identity confirmed: The response "I am Qwen3.5, the latest large language model developed by Tongyi Lab" confirms the correct model weights were loaded and are producing expected outputs.
  3. Latency baseline: The response time is not explicitly measured here, but the fact that a 122B model across two nodes returns a response within the curl timeout establishes a rough performance baseline.
  4. API compatibility confirmed: The standard OpenAI chat completions schema works, meaning any OpenAI-compatible client can use this deployment. The assistant's subsequent message ([msg 6724]) expands on this, noting that "thinking/reasoning content is working, and tool calls are enabled." The verification in message [msg 6723] established the foundation for those more detailed observations.

Assumptions and Their Validity

The assistant makes several assumptions in this message:

That the API endpoint is reachable from the jump host: The curl targets localhost:30000 on the head Spark (via SSH tunneling through the jump host). This assumes the vLLM server bound to all interfaces and that port 30000 is accessible. This assumption proved correct.

That a simple prompt will work without special handling: The model is a reasoning model (Qwen3.5), which typically outputs its reasoning process before the final answer. The assistant does not request reasoning explicitly, and the response contains only the final answer without visible reasoning tokens. This suggests the model's reasoning parser is active and stripping the thinking content, or the model chose not to reason for such a simple query.

That the model can serve immediately after loading: The assistant waited 900 seconds (15 minutes) before the previous check, then immediately issued this curl. The assumption that the server was fully initialized and ready for inference proved correct.

That python3 -m json.tool is available: The assistant pipes through Python's JSON pretty-printer, assuming Python 3 is in the PATH inside the SSH session. This is a reasonable assumption given the environment.

What This Message Reveals About the Assistant's Thinking

The assistant's reasoning is visible in the structure of the message. It does not simply run the curl — it constructs a deliberate test. The -s flag suppresses curl's progress output for clean JSON. The 2>&1 merges stderr into stdout so any errors appear in the output. The python3 -m json.tool formatting makes the result immediately readable.

The choice of test prompt is also revealing. The assistant could have tested with a more complex query, a reasoning chain, or a tool call. Instead, it chooses the simplest possible interaction: identity question + brevity constraint. This is classic debugging methodology — start with the simplest test that can fail, then progressively increase complexity. A failure here would indicate fundamental problems (wrong model, broken generation, API misconfiguration). Success here allows the assistant to proceed to more nuanced tests (reasoning output, tool calling, throughput benchmarks).

The assistant also does not include any conditional logic or error recovery in this message. There is no "if the response contains X, then do Y" — just the raw curl and its output. This confidence is earned through the preceding battle. By message [msg 6723], the assistant has already solved the hard problems. This message is the victory lap.

The Broader Significance

In the context of the entire session, message [msg 6723] represents a major milestone. The session began with environment setup on a Proxmox server with RTX PRO 6000 Blackwell GPUs, progressed through multiple model deployments (GLM-5-NVFP4, Kimi-K2.5, Qwen3.5-397B), and eventually expanded to the DGX Spark cluster. Each deployment taught lessons about hardware compatibility, driver versions, NCCL configuration, and memory management. This message proves that the assistant can deploy a state-of-the-art 122B parameter reasoning model across a multi-node ARM-based cluster — a genuinely impressive systems engineering achievement.

The response itself — "I am Qwen3.5, the latest large language model developed by Tongyi Lab" — is unremarkable text. But the infrastructure behind it is remarkable: two ARM-based desktop supercomputers, connected by InfiniBand, running a custom vLLM build, orchestrated by Ray, with all the networking quirks of heterogeneous subnets and IOMMU configurations resolved. Every token in that response traveled across a physical InfiniBand link between two machines, through NCCL's collective communication primitives, through tensor parallelism sharding, through CUDA graph-optimized kernels on Blackwell GPUs. The simplicity of the output belies the complexity of the system that produced it.