Benchmarking Qwen3.5-122B-A10B-FP8 on Dual DGX Spark: A Memory-Bound Reality Check

Introduction

In the sprawling narrative of an opencode coding session spanning dozens of segments and thousands of messages, message [msg 6730] stands as a quiet moment of assessment—a brief, data-rich pause after an intense multi-hour infrastructure battle. The assistant had just finished deploying the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark systems (GB10, Blackwell SM121, ARM Cortex-X925 CPUs, 120GB unified memory each, connected via InfiniBand RoCE). The user's simple prompt—"benchmark?"—triggered a methodical performance evaluation whose results are presented and analyzed in this single message. What makes this message remarkable is not the raw numbers themselves, but the depth of contextual interpretation the assistant provides, situating the benchmark results within the broader hardware landscape of the session.

The Message: A Benchmark Table with Embedded Analysis

The subject message reads in full:

Results are in. The first warmup request was 12s TTFT (Triton kernel JIT compilation), but subsequent requests are much faster.

>

Qwen3.5-122B-A10B-FP8 on 2x GB10 DGX Spark (TP=2, vLLM 0.17):

>

| C | Agg tok/s | Per-req tok/s | Avg TTFT | |---|---|---|---| | 1 | 26 | 27 | 0.48s | | 2 | 41 | 21 | 0.86s | | 4 | 71 | 19 | 1.48s | | 8 | 94 | 12 | 2.29s | | 16 | 144 | 10 | 3.20s |

>

For context: - 27 tok/s single-request on 2x DGX Spark (ARM, unified memory, IB interconnect) is reasonable for a 122B MoE model in FP8 - The RTX PRO 6000 setup (4x discrete GPUs with PCIe Gen5) gets 277 tok/s with MTP speculation — ~10x faster, which makes sense given it has much faster discrete GDDR7 vs unified LPDDR5x memory - The Spark throughput matches community benchmarks (the forum thread showed ~75 tok/s for GPT-OSS-120B which is a similar-sized MoE model, but that was with MXFP4 quantization)

>

The bottleneck on Spark is the unified memory bandwidth — LPDDR5x is much slower than GDDR7 for weight loading during decode. The model is memory-bandwidth-bound at low concurrency.

On the surface, this is a straightforward benchmark report. But every sentence carries the weight of the preceding deployment struggle and reveals the assistant's deep understanding of the hardware-software stack.

The Road to This Message: A Deployment Odyssey

To understand why this message was written, one must appreciate the journey that preceded it. The assistant had just completed a grueling multi-node deployment on two DGX Spark systems—compact ARM-based AI supercomputers from NVIDIA with unified memory architecture (CPU and GPU share the same 120GB LPDDR5x pool). The deployment required solving six distinct infrastructure challenges ([msg 6724]):

  1. SGLang incompatibility: The official lmsysorg/sglang:spark image was too old to support Qwen3.5, forcing a pivot to vLLM.
  2. Multi-node NCCL hangs: SGLang's multi-node NCCL initialization hung indefinitely, requiring a switch to vLLM + Ray.
  3. Ray IP registration: Ray auto-detected the head node's external IP (10.1.230.180), which was unreachable from the second Spark, requiring --node-ip-address to force the InfiniBand subnet IP (192.168.200.x).
  4. Cross-node connectivity: The worker node couldn't reach the head on the external IP, requiring all distributed communication to be forced through the enp1s0f1np1 interface.
  5. Ray OOM killer: Ray's memory monitor (95% threshold) killed the vLLM process during CUDA graph capture, requiring RAY_memory_monitor_refresh_ms=0 and reduced --gpu-memory-utilization.
  6. vLLM version: vLLM 0.14 lacked Qwen3.5 support, requiring the community image hellohal2064/vllm-qwen3.5-gb10 with vLLM 0.17.1rc1. Each of these challenges was diagnosed and resolved through iterative debugging—checking logs, adjusting environment variables, restarting containers, and verifying connectivity. The final deployment used NCCL over NET/IBext_v11 for inter-node tensor parallelism (TP=2), with both nodes communicating over InfiniBand RoCE at 192.168.200.x addresses.

Benchmark Methodology and Interpretation

The assistant wrote a benchmark script (bench_spark_qwen35.py, [msg 6727]) and ran it with concurrency levels of 1, 2, 4, 8, and 16 concurrent requests. Each test generated 2000 total tokens (4 requests at C=1, 4 at C=2, 8 at C=4, etc.). The warmup request exhibited a 12-second time-to-first-token (TTFT), which the assistant correctly attributed to Triton kernel JIT compilation—a one-time cost as the vLLM runtime compiles and optimizes CUDA kernels for the specific model architecture on the SM121 Blackwell GPU.

The results reveal a clear pattern: aggregate throughput scales with concurrency (from 26 tok/s at C=1 to 144 tok/s at C=16), but per-request throughput degrades (from 27 tok/s down to 10 tok/s), and average TTFT increases (from 0.48s to 3.20s). This is characteristic of a memory-bandwidth-bound system: as more requests compete for the same memory bus, each individual request slows down, but the system as a whole can keep more of the pipeline busy.

The assistant's analysis zeroes in on the root cause: unified memory bandwidth. The DGX Spark's LPDDR5x memory, shared between CPU and GPU, has significantly lower bandwidth than the GDDR7 memory found in discrete GPUs like the RTX PRO 6000 Blackwell. During autoregressive decoding, the dominant cost is loading model weights from memory to compute units for each token. With unified memory, this bandwidth is shared with the CPU and system, creating a bottleneck that manifests clearly in the benchmark data.

Comparative Analysis: Three Hardware Tiers

The assistant provides three points of comparison that illuminate the results:

1. DGX Spark (this benchmark): 27 tok/s single-request. The assistant characterizes this as "reasonable" for a 122B Mixture-of-Experts model in FP8 precision on the Spark hardware. This is a measured assessment—not triumphant, not dismissive, but grounded in an understanding of the hardware constraints.

2. RTX PRO 6000 setup: 277 tok/s with MTP speculation. This comparison comes from earlier in the session ([msg 6724] and preceding segments), where the assistant had deployed the same model on a Proxmox host with 4× RTX PRO 6000 Blackwell GPUs (discrete GDDR7, PCIe Gen5). The ~10x performance gap is attributed to the fundamentally different memory architectures: GDDR7 provides vastly higher bandwidth than LPDDR5x, and discrete GPUs don't share that bandwidth with the CPU.

3. Community benchmark: ~75 tok/s for GPT-OSS-120B with MXFP4. This reference to a forum thread provides external validation that the Spark's performance is in the expected range. The assistant notes that the community result used MXFP4 quantization (4-bit), which would be even more memory-efficient than the FP8 used here, explaining the higher throughput despite a similar model size.

Assumptions and Potential Blind Spots

The assistant makes several assumptions in this analysis that deserve scrutiny:

The Triton JIT explanation for warmup TTFT: The 12-second warmup is attributed to Triton kernel compilation. This is plausible—vLLM uses Triton for many fused attention and MoE kernels, and the first inference pass triggers compilation. However, there could be other contributing factors: CUDA graph capture, NCCL connection establishment between the two nodes, or model weight loading from disk to GPU memory (though the model was already loaded). The assistant's assumption is reasonable but not proven.

Memory bandwidth as the sole bottleneck: The analysis focuses exclusively on memory bandwidth. While this is almost certainly the dominant factor for a 122B parameter model during decode, there may be secondary bottlenecks: the InfiniBand RoCE interconnect latency between the two nodes (TP=2 requires communication for every token), the ARM CPU's ability to schedule and dispatch work, or the vLLM scheduler overhead at high concurrency. The assistant's single-factor explanation is elegant but may oversimplify.

Comparison fairness: The comparison to the RTX PRO 6000 setup is informative but not apples-to-apples. The RTX PRO setup used MTP (Multi-Token Prediction) speculation, which provides a 12-45% throughput improvement ([msg 6724]). The DGX Spark benchmark did not use speculation. Additionally, the RTX PRO setup had 4 GPUs vs the Spark's 2, and used PCIe Gen5 vs InfiniBand RoCE. The assistant acknowledges the speculation difference implicitly but doesn't break down the multiple contributing factors.

The Thinking Process Visible in the Message

The message reveals a sophisticated analytical process. The assistant doesn't just dump numbers—it structures them, contextualizes them, and extracts a diagnosis. The progression is:

  1. Acknowledge the warmup artifact: Explain the 12s TTFT as Triton JIT compilation, reassuring the reader that steady-state performance is what matters.
  2. Present the data cleanly: A formatted table with four columns (concurrency, aggregate throughput, per-request throughput, average TTFT).
  3. Provide hardware context: Remind the reader of the exact hardware configuration (2x DGX Spark, ARM, unified memory, IB interconnect).
  4. Compare to known baselines: Reference the RTX PRO 6000 setup from earlier in the session and community benchmarks.
  5. Diagnose the bottleneck: Identify unified memory bandwidth as the root cause.
  6. Explain the mechanism: Connect the hardware characteristic (LPDDR5x bandwidth) to the observed behavior (memory-bandwidth-bound decode). This is not a rote report. It is an expert analysis that draws on deep knowledge of GPU architecture, memory hierarchy, and inference optimization.

The Broader Significance

This message marks a critical inflection point in the session. After hours of infrastructure wrestling—installing drivers, building flash-attn from source, patching SGLang for SM120 compatibility, configuring Ray networking, and fighting OOM killers—the assistant finally has a working multi-node deployment and can answer the fundamental question: "How fast is it?"

The answer—27 tok/s single-request, 144 tok/s at high concurrency—is neither spectacular nor disappointing. It is simply informative. It tells the user (and the assistant) that the DGX Spark, while a remarkable compact AI system, is not competitive with discrete GPU setups for large model inference. The unified memory architecture that makes the Spark so elegant for development and small-model work becomes a liability when serving 122B parameter models.

This knowledge shapes future decisions. If the goal is high-throughput serving of large models, the RTX PRO 6000 setup is the right path. If the goal is a compact, low-power, always-on inference node for moderate workloads, the Spark is viable. The benchmark provides the data needed to make that trade-off.

Conclusion

Message [msg 6730] is a masterclass in contextual benchmarking. It takes a simple table of throughput numbers and transforms it into a hardware diagnosis, a comparative analysis across three system configurations, and a decision-support tool for future infrastructure choices. The assistant's ability to interpret the results through the lens of memory architecture—connecting the LPDDR5x bandwidth limitation to the observed throughput scaling—demonstrates a level of systems thinking that goes far beyond running a benchmark script. In the broader arc of the opencode session, this message closes the deployment chapter and opens the evaluation chapter, providing the quantitative foundation for the infrastructure decisions that follow.