The Art of Infrastructure Archaeology: Deploying Qwen3.5 Across Dual DGX Sparks

Introduction

In the sprawling, multi-month saga of deploying cutting-edge language models across heterogeneous hardware, few moments are as pivotal as the one captured in message 6559 of this opencode session. This message is not a triumphant declaration of success, nor a desperate plea for help—it is something far more subtle and instructive: a moment of strategic reconnaissance, where an AI assistant pauses to survey the landscape before committing to a course of action.

The message sits at a critical juncture. The user has just issued a new mission: deploy the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark systems (GB10 form factor, each with 120GB of unified memory, ARM Cortex-X925 CPUs, and InfiniBand interconnect). The assistant has spent the preceding messages exploring both nodes, checking hardware configurations, examining existing services, and researching deployment recipes. Message 6559 is the synthesis of that reconnaissance—a reasoning passage followed by targeted verification commands that will determine the entire deployment strategy.

This article examines that single message in depth: what the assistant knew, what it assumed, what decisions it made (and deferred), and what the message reveals about the broader challenge of deploying large language models on exotic hardware. It is a case study in infrastructure archaeology—the art of understanding what already exists before building something new.

The Message in Full

Let us first quote the subject message exactly as it appeared in the conversation:

Agent Reasoning

>

Key findings:

>

1. Model: Qwen/Qwen3.5-122B-A10B-FP8 exists on HF - official from Qwen 2. Head Spark: Has Docker vllm-node image with CUDA 13.1, TORCH_CUDA_ARCH_LIST=12.1a - this is the spark-specific vLLM Docker image 3. IB between sparks: rocep1s0f1 / enp1s0f1np1 is UP with IP 192.168.200.12 on head, 192.168.200.13 on second 4. Second spark has no internet (curl to HF timed out / no response) 5. Existing launch-cluster.sh script supports multi-node deployment with auto-detection 6. The Docker image has CUDA 13.1 and TORCH_CUDA_ARCH_LIST=12.1a (SM121 Blackwell GB10)

>

So my plan is to leverage the existing multi-node infrastructure in spark-vllm-docker/launch-cluster.sh, download the FP8 model on the head spark where internet is available, then distribute it to the second spark.

>

But wait—the user wants to deploy Qwen3.5-122B-A10B, which is a hybrid GDN model. On the kpro6 setup, we needed SGLang with --attention-backend triton because flashinfer doesn't support hybrid GDN architectures. The vLLM Docker image on the Sparks is an older version (v0.11), so I need to verify whether it actually supports the Qwen3.5 architecture before committing to this approach. Looking at the memory requirements more carefully - with FP8 quantization, all 125B parameters need to fit in memory even though only 10B are active at inference time. That's roughly 125GB total, which with tensor parallelism across 2 nodes means about 62.5GB per node. Given each node has around 120GB unified memory and accounting for OS overhead, this should fit, but I need to verify the actual model size on disk to be sure.

>

[Tool calls: exa_web_fetch_exa to HuggingFace model page, bash to check vLLM version, bash to check transformers/torch versions]

The message contains three tool calls dispatched in parallel: a web fetch to the HuggingFace model page for Qwen3.5-122B-A10B-FP8, a bash command to check the vLLM version inside the existing Docker container, and another bash command to check the transformers and PyTorch versions. These are verification probes, designed to answer the questions the assistant raised in its reasoning.

The Reasoning: A Window Into Strategic Decision-Making

The "Agent Reasoning" section of this message is where the assistant's thinking is most visible, and it reveals a remarkably sophisticated decision-making process. The assistant does not simply accept the first plausible path; it actively interrogates its own plan, identifies potential failure modes, and adjusts its strategy accordingly.

The Initial Plan

The assistant begins by enumerating five key findings, each derived from the exploration work of the preceding messages (6554–6558). These findings paint a picture of the existing infrastructure:

  1. The model exists officially on HuggingFace, so there is no question of using a community-quantized variant.
  2. The head Spark already has a Docker-based vLLM image, purpose-built for the GB10 hardware (indicated by TORCH_CUDA_ARCH_LIST=12.1a, which targets SM121 Blackwell).
  3. The InfiniBand link between the two Sparks is operational, with IPs 192.168.200.12 and 192.168.200.13 on the enp1s0f1np1 interface.
  4. The second Spark lacks internet access—a critical constraint that means the model must be downloaded on the head node and transferred.
  5. An existing launch-cluster.sh script supports multi-node deployment, suggesting the previous operator had already solved the cluster orchestration problem. From these findings, the assistant formulates an initial plan: "leverage the existing multi-node infrastructure in spark-vllm-docker/launch-cluster.sh, download the FP8 model on the head spark where internet is available, then distribute it to the second spark." This is the path of least resistance—reuse what already works, adapt only what is necessary.

The Pivot: Recognizing Architectural Incompatibility

Then comes the critical pivot. The assistant writes: "But wait—" and proceeds to identify a fundamental architectural concern. The Qwen3.5-122B-A10B is a hybrid GDN (Gated Dense Network) model—a Mixture-of-Experts architecture where only 10B of the 122B total parameters are active at any given time. On a previous setup (the "kpro6" system, which was the Proxmox VM with RTX PRO 6000 Blackwell GPUs from earlier segments), the assistant had learned that flashinfer—the default attention backend in both SGLang and vLLM—does not support hybrid GDN architectures. The solution there was to use SGLang with --attention-backend triton.

This is a crucial piece of learned knowledge being applied to a new context. The assistant recognizes that the same architectural constraint applies here: if the existing vLLM Docker image (which it estimates as "an older version (v0.11)") does not support hybrid GDN models, the entire plan of reusing the existing infrastructure collapses.

The assistant's reasoning here reveals a sophisticated understanding of the dependency chain. It is not enough for the model to fit in memory; the inference engine must also support the model's architecture. And support for Qwen3.5's hybrid GDN architecture is a relatively recent addition to both SGLang and vLLM—it may not be present in the older Docker image.

The Memory Calculation

The assistant also performs a rough memory budget calculation. With FP8 quantization, 122B parameters require approximately 122GB of storage (1 byte per parameter in FP8, plus some overhead for the 10B active parameters' additional data structures). With tensor parallelism across two nodes, each node would need to hold roughly 62.5GB of model weights. Each DGX Spark has 120GB of unified memory (shared between CPU and GPU in the GB10's unified memory architecture), so after accounting for OS overhead (~15GB as seen in the free -h output from message 6555), approximately 105GB would be available per node. This leaves about 42GB per node for KV cache and other runtime data—tight but feasible.

The assistant flags the need to verify "the actual model size on disk," which is a wise precaution. Model sizes on HuggingFace can vary depending on quantization format, sharding, and whether the safetensors include optimizer state or other metadata.

The Verification Commands: Testing the Hypotheses

The three tool calls in this message are not arbitrary—they are carefully chosen to test the assumptions underlying the deployment plan.

Checking the HuggingFace Model Page

The web fetch to https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8 serves multiple purposes. First, it confirms the model exists and is the official Qwen release (not a community quant). Second, it retrieves the model card, which contains critical information about the architecture, quantization format, and any special deployment requirements. Third, it may reveal the total model size, shard count, and file listing—all essential for planning the download and transfer strategy.

In the context of the assistant's reasoning, this fetch also addresses the "hybrid GDN" concern. The model card for Qwen3.5-122B-A10B-FP8 would explicitly state the architecture type, confirming or refuting the assistant's assumption about GDN support requirements.

Checking the vLLM Version

The bash command docker exec vllm_node bash -c "pip show vllm ..." returns vLLM 0.14.0rc2.dev262+g7fe255889.cu130. This is a critical data point that partially invalidates the assistant's assumption. The assistant had estimated "an older version (v0.11)," but the actual version is 0.14.0rc2—a much more recent release candidate. This matters because vLLM 0.14.x may include support for Qwen3.5's hybrid GDN architecture, whereas v0.11 likely would not.

However, the version string also reveals something else: it is a development build (dev262) with a specific git hash, compiled against CUDA 13.0. This is not a standard release—it is a custom build, likely from NVIDIA's internal vLLM fork for DGX Spark support. The TORCH_CUDA_ARCH_LIST=12.1a in the Docker image environment confirms this is a Spark-optimized build targeting SM121 Blackwell compute capability.

Checking Transformers and PyTorch

The second bash command reveals transformers 5.0.0rc3 and torch 2.9.1+cu130. These are extremely recent versions—transformers 5.0.0 is a major release that includes support for the latest Qwen architectures, and PyTorch 2.9.1 is a nightly build. This is encouraging: the software stack is modern enough to potentially support Qwen3.5.

The combination of these three verification results gives the assistant a much clearer picture. The vLLM version is newer than assumed, the transformers library is bleeding-edge, and PyTorch is compiled for the correct CUDA version. The path forward is narrowing: the existing Docker image may actually work, or at worst, a minor upgrade may be needed.

Assumptions and Their Validity

Every decision in this message rests on a web of assumptions. Some are explicit, some implicit. Examining them reveals both the strengths and potential blind spots of the assistant's reasoning.

Explicit Assumptions

  1. "The vLLM Docker image on the Sparks is an older version (v0.11)" — This assumption is tested and partially disproven by the verification command. The actual version is 0.14.0rc2, which is significantly more recent. However, the assistant's caution is justified: assuming a newer version without checking could lead to a failed deployment.
  2. "With FP8 quantization, all 125B parameters need to fit in memory even though only 10B are active" — This is correct for MoE models. Even though only a subset of experts are activated per token, all parameters must be loaded into memory because the router can select any expert at any time. The 122B → 125B rounding is a reasonable approximation.
  3. "TP=2 means about 62.5GB per node" — This assumes perfect sharding, which is approximately correct for transformer layers. Some metadata (tokenizer config, model config) is duplicated across nodes, but the weights themselves are evenly split.

Implicit Assumptions

  1. The existing Docker image supports the Qwen3.5 architecture — This is the most critical untested assumption. The assistant has verified the vLLM version but not whether that version includes the specific model architecture support. vLLM 0.14.0rc2 may or may not have the Qwen3.5 model class registered.
  2. The InfiniBand link is sufficient for tensor parallelism — The IB link shows "Rate: 40" (likely 40Gbps) on the second Spark, which is relatively slow for inter-node TP. The head node's IB status was not fully shown. TP requires extremely low-latency, high-bandwidth communication for each forward pass. 40Gbps IB may be marginal for 122B parameter model sharded across two nodes.
  3. The launch-cluster.sh script can be adapted without modification — The assistant plans to "leverage the existing multi-node infrastructure," but the script was designed for the previous model (GLM-4.7-Flash-NVFP4). It may have hardcoded paths, model names, or port configurations that need changing.
  4. The second Spark's lack of internet is the only networking issue — The assistant assumes that once the model is transferred, the second Spark can participate in inference without internet access. This is true for the inference engine itself, but Ray (which vLLM uses for multi-node orchestration) may require internet access for certain operations like downloading dependencies or registering with a Ray cluster.

Input Knowledge Required

To fully understand this message, a reader would need knowledge in several domains:

Hardware Architecture

Model Architecture

Software Stack

Operational Context

Output Knowledge Created

This message produces several concrete outputs that advance the deployment:

  1. vLLM version confirmed: 0.14.0rc2.dev262+g7fe255889.cu130 — a development build, newer than assumed, compiled for CUDA 13.0 with SM121 support
  2. Transformers version confirmed: 5.0.0rc3 — bleeding-edge, likely supporting Qwen3.5 architecture
  3. PyTorch version confirmed: 2.9.1+cu130 — nightly build matching the CUDA toolkit version
  4. Model existence confirmed: The HuggingFace fetch retrieves the model card, confirming the FP8 variant is official and available
  5. Memory budget validated: The assistant's calculation suggests the model can fit across two nodes with reasonable KV cache headroom These outputs collectively narrow the decision space. The assistant now knows that the software stack is modern enough to potentially support the target model, and the primary remaining question is whether vLLM 0.14.0rc2 specifically includes Qwen3.5 model support.

The Thinking Process: A Masterclass in Diagnostic Reasoning

What makes this message remarkable is not any single insight but the structure of the thinking itself. The assistant demonstrates a diagnostic reasoning pattern that would be familiar to any experienced systems engineer:

  1. Survey the landscape: Enumerate what is known (findings 1-6)
  2. Formulate a plan: "Leverage existing infrastructure"
  3. Stress-test the plan: "But wait—" — identify the weakest assumption
  4. Trace the dependency chain: Architecture support → vLLM version → Docker image age
  5. Gather evidence: Deploy verification commands targeting the critical unknowns
  6. Recalculate: Update the mental model with new data This is the opposite of "move fast and break things." It is a deliberate, cautious approach that prioritizes understanding over action. The assistant knows that a failed deployment could take hours to recover from (model download alone could be 30+ minutes for a 119GB model), so it invests time upfront in verification. The "But wait—" moment is particularly instructive. It represents the assistant catching itself before committing to a suboptimal path. In the context of AI safety and reliability, this self-correction capability is crucial. An assistant that blindly executes its first plan is dangerous; one that interrogates its own assumptions is trustworthy.

Mistakes and Incorrect Assumptions

While the message is generally sound, there are a few points where the assistant's reasoning could be challenged:

The vLLM Version Miscalculation

The assistant states "The vLLM Docker image on the Sparks is an older version (v0.11)" based on an earlier check (message 6556) that showed the image was "2 months old." This is a reasonable inference—a two-month-old image from January 2026 would indeed predate Qwen3.5's release (March 2026). However, the actual version turns out to be 0.14.0rc2, which is a development release candidate that may include bleeding-edge model support. The assistant's assumption was too pessimistic.

The "125B" Rounding

The assistant rounds "122B-A10B" to "125B" for the memory calculation. While this is a reasonable approximation (the difference is only ~2.5%), it could matter in the tight memory budget of the DGX Spark. The actual model size on disk (which the HuggingFace fetch would reveal) might be closer to 119GB as seen in the chunk summary, making the 62.5GB-per-node estimate slightly high.

The IB Bandwidth Assumption

The assistant does not explicitly consider whether 40Gbps InfiniBand is sufficient for TP=2 on a 122B parameter model. For reference, inter-node TP typically requires 200-400 Gbps for efficient scaling on models of this size. 40Gbps may introduce significant communication overhead, potentially negating the benefits of multi-node deployment. This is a silent assumption that could surface as a performance issue later.

Conclusion

Message 6559 is a masterclass in infrastructure archaeology and strategic reasoning. It captures the moment when an AI assistant, faced with a complex deployment challenge on unfamiliar hardware, pauses to survey the landscape, interrogate its assumptions, and gather critical evidence before committing to a course of action.

The message reveals the assistant's deep understanding of the dependency chain connecting model architecture, inference engine support, hardware capability, and software versions. It demonstrates the ability to learn from past experience (the kpro6 GDN/flashinfer issue) and apply that knowledge to a new context. And it shows a healthy skepticism toward its own plans—the willingness to say "But wait—" and reconsider.

For the reader, this message offers a window into the invisible work of systems engineering: the reasoning, the verification, the assumption-checking that happens before any command is executed. In a field where most visible work is the execution (the bash commands, the config files, the deployment scripts), the thinking captured here is the part that separates a fragile deployment from a robust one.

The message ends with the assistant in a state of informed uncertainty. It has narrowed the unknowns but not eliminated them. The next messages will show whether the verification commands confirm or refute the remaining assumptions, and whether the deployment proceeds as planned or requires yet another pivot. That is the nature of infrastructure work: a continuous cycle of reconnaissance, planning, verification, and adaptation.