The Pivot Point: Probing vLLM's Multi-Node CLI in a Cross-Spark Deployment

ssh aurora@10.1.230.180 'docker run --rm --entrypoint bash hellohal2064/vllm-qwen3.5-gb10 -c "vllm serve --help 2>&1 | grep -E \"tensor|parallel\" | head -5"' 2>&1 | tail -10

At first glance, message [msg 6658] appears to be nothing more than a routine reconnaissance command — a quick probe into a Docker container's help output to discover CLI flags. But in the broader arc of a complex multi-node deployment spanning two NVIDIA DGX Spark systems, this single line represents a critical inflection point. It is the moment the assistant definitively abandons one failed approach (SGLang multi-node) and commits to an alternative (vLLM with Ray), after hours of debugging a deadlocked NCCL initialization. Understanding why this particular command was issued, what it reveals about the assistant's reasoning, and how it shaped the subsequent deployment requires unpacking the technical drama that led to this juncture.

The Context: A Multi-Node NCCL Deadlock

The assistant had been tasked with deploying the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter FP8 mixture-of-experts reasoning model — across two DGX Spark nodes connected via InfiniBand RoCE. The DGX Spark is NVIDIA's compact ARM-based desktop supercomputer (GB10, SM121 Blackwell GPU, 120GB unified memory), and multi-node inference across two such systems requires careful distributed setup.

The initial attempt used SGLang, a popular inference engine. The assistant configured a head-worker pair, launched both containers, and waited. What followed was a classic distributed systems failure mode: the NCCL logs showed channel setup completing successfully (NET/IBext_v11 transport established, send/receive pairs configured), but the process then hung indefinitely. Across multiple retries spanning messages [msg 6630] through [msg 6647], the assistant tried adding --disable-custom-all-reduce, adjusting NCCL debug settings, and waiting progressively longer intervals (30 seconds, 60 seconds, 120 seconds, 180 seconds). Nothing worked. The logs consistently ended with CustomAllreduce is disabled because this process group spans across nodes and never progressed to weight loading.

The assistant's diagnosis was precise: "Completely stuck for 7+ minutes now — no progress at all. Something is wrong. This is a deadlock after the first NCCL communicator init. SGLang creates a second process group (likely for the Gloo backend or a pipelining group) that's hanging." This showed deep understanding of the SGLang distributed initialization sequence — the assistant knew that init_process_group creates a TCP store-based rendezvous, then NCCL establishes communication channels, and a second process group creation was likely where the hang occurred.

The Pivot: Discovering a Dedicated vLLM Image

After exhausting SGLang options, the assistant pivoted to vLLM. This was not a random choice — the assistant had previously noted that "vLLM is proven to work multi-node on these Sparks (the existing setup used it)." The existing infrastructure on the DGX Sparks already used a vllm-node Docker image for serving GLM models, and the assistant knew that vLLM's Ray-based multi-node architecture was battle-tested on this hardware.

A web search ([msg 6649]) revealed hellohal2064/vllm-qwen3.5-gb10 — a Docker image specifically built for Qwen3.5 on DGX Spark/GB10, updated just 20 days prior. The assistant pulled this image ([msg 6650]), verified it contained Qwen3.5 model files (qwen3_5.py, qwen3_5_mtp.py, custom config classes) ([msg 6655]), and transferred it to the second Spark node via docker save | ssh | docker load ([msg 6656]), achieving a successful transfer.

The Subject Message: Probing the CLI

Message [msg 6658] is the very next step after the image transfer. The assistant has the image on both nodes, knows it supports Qwen3.5, and now needs to understand how to launch it for multi-node inference. The command runs inside the container:

vllm serve --help 2>&1 | grep -E "tensor|parallel" | head -5

This is a targeted probe. The assistant is not reading the entire help output — it's filtering for flags related to tensor parallelism and distributed execution. The --help output of vllm serve is extensive (hundreds of lines), and the assistant knows exactly what it's looking for: --tensor-parallel-size, --pipeline-parallel-size, --distributed-executor-backend (which selects between Ray and MPI), and related options.

The choice of grep -E "tensor|parallel" is deliberate. The assistant needs to answer several questions before writing the launch script:

What is the correct flag for tensor parallelism? vLLM uses --tensor-parallel-size (not -tp or --tp), and the assistant needs to confirm the exact syntax.
Does this vLLM version support --distributed-executor-backend? This flag controls whether vLLM uses Ray or MPI for multi-node. For the DGX Spark setup with two separate physical nodes, Ray is required.
Are there any GB10-specific or ARM-specific flags? The image is specifically built for DGX Spark, so there might be custom flags.
What is the default pipeline parallelism behavior? With only one GPU per node, pipeline parallelism isn't useful, but tensor parallelism across nodes is essential. The head -5 limit shows the assistant expects only a few matching lines — it's a quick sanity check, not an exhaustive analysis. The tail -10 on the outer command is a defensive measure to capture any error output that might appear before the grep results.

Assumptions and Implicit Knowledge

This message makes several assumptions that reveal the assistant's mental model:

Assumption 1: vLLM's CLI is the primary interface. The assistant assumes that vllm serve --help will contain the necessary information to configure multi-node inference. This is correct for vLLM, which has a well-documented CLI, but it assumes the flags haven't changed between vLLM versions (the image uses vLLM 0.17.1rc1, a release candidate).

Assumption 2: The image's entrypoint supports --entrypoint bash. The command uses Docker's --entrypoint bash to override the default entrypoint and run an arbitrary command. This assumes the container has bash and that vllm is on PATH. The assistant had already verified this in previous messages ([msg 6654]).

Assumption 3: Grep patterns "tensor|parallel" will capture all relevant flags. This is a reasonable heuristic but could miss flags like --distributed-executor-backend (which contains neither "tensor" nor "parallel") or --worker-use-ray (an older flag). The assistant compensates for this by following up with additional probes in subsequent messages ([msg 6659] tries python3 -m vllm.entrypoints.openai.api_server --help).

Assumption 4: SSH key forwarding and Docker access work on the remote node. The command chains ssh to the head node, which then runs docker run. This assumes the SSH session has the necessary Docker socket access and that the hellohal2064/vllm-qwen3.5-gb10 image is present (it was transferred in the previous step).

The Outcome: What This Message Achieved

While the raw output of this command is not shown in the conversation (the next message [msg 6659] shows a different command that crashed), the probe served its purpose. In subsequent messages, the assistant successfully launched a Ray cluster across both Spark nodes, started the vLLM server with --tensor-parallel-size 2 --distributed-executor-backend ray, loaded the 119B model, and achieved ~27 tok/s single-request throughput with correct reasoning output.

This message is a textbook example of incremental infrastructure probing — a pattern where the assistant systematically discovers the configuration surface of a new tool before committing to a launch script. Rather than guessing flags or reading documentation blindly, it queries the tool's own help system, filters for relevant options, and builds understanding from the ground up. This approach minimizes the risk of syntax errors and version mismatches, which is especially important when working with release-candidate software (vLLM 0.17.1rc1) on non-standard hardware (ARM64 Blackwell).

The Thinking Process: What We Can Infer

The assistant's reasoning at this point, reconstructed from the conversation flow, goes something like this:

"SGLang multi-node is deadlocked — the NCCL second group creation hangs every time. I've tried disabling custom allreduce, waiting longer, and checking NCCL debug output. Nothing works. The forum thread confirms this is a known issue. But I have a vLLM image that's specifically built for Qwen3.5 on GB10, and the existing Spark infrastructure already uses vLLM successfully. I've transferred the image to both nodes. Now I need to figure out the exact launch flags for multi-node tensor parallelism. Let me check vllm serve --help for the tensor/parallel flags so I can write the Ray cluster launch script."

This is pragmatic debugging at its finest. The assistant doesn't chase the SGLang deadlock further — it recognizes a fundamental incompatibility (likely the Gloo backend or a second NCCL communicator group that SGLang creates for pipelining) and pivots to a known-working alternative. The probe is the first step in building the new deployment from scratch, learning the tool's interface through direct interrogation rather than documentation.

Conclusion

Message [msg 6658] is a small command with outsized significance. It marks the transition from debugging a dead end to building a working solution. In the space of a single SSH-wrapped Docker command, the assistant closes the chapter on SGLang multi-node and opens the chapter on vLLM with Ray. The command itself — a grep-filtered help probe — embodies a methodology of incremental, tool-assisted discovery that characterizes the most effective infrastructure engineering. It is the quiet pivot point on which the entire DGX Spark deployment turns.