Benchmarking Qwen3.5-122B-A10B-FP8 Across Two DGX Spark Nodes: A Performance Snapshot

The Message

ssh aurora@10.1.230.180 'python3 /home/aurora/bench_spark_qwen35.py 1 2 4 8 16' 2>&1
Warming up...
  Warmup: tokens=50, ok=True, ttft=11.99s

   C |  Agg tok/s |  Per-req tok/s |   Avg TTFT |   Tokens |  Wall(s) |   Ok
--------------------------------------------------------------------------------
   1 |       26.1 |           26.8 |      0.48s |     2000 |     76.5 |    4
   2 |       41.2 |           21.4 |      0.86s |     2000 |     48.5 |    4
   4 |       71.3 |           18.8 |      1.48s |     4000 |     56.1 |    8
   8 |       93.5 |           12.4 |      2.29s |     ...

Context and Motivation

This message, sent by the AI assistant in an opencode coding session, represents the culmination of a lengthy and technically demanding deployment effort. The assistant had just finished deploying Qwen3.5-122B-A10B-FP8, a 122-billion-parameter Mixture-of-Experts (MoE) language model, across two NVIDIA DGX Spark systems — compact ARM-based AI workstations powered by the GB10 Grace Blackwell superchip, each equipped with 120GB of unified memory and connected via InfiniBand RoCE networking.

The user's preceding message was a single word: "benchmark?" ([msg 6725]). This terse query was a natural next step after the assistant had announced the successful deployment in [msg 6724], summarizing the architecture, the six key challenges overcome, and the final configuration. The user wanted quantitative validation — not just that the model served responses, but how well it performed under load. The assistant responded by writing a benchmark script ([msg 6727]), copying it to the Spark head node ([msg 6728]), and then executing it with concurrency levels of 1, 2, 4, 8, and 16 concurrent requests ([msg 6729], the subject message).

What the Message Reveals

The benchmark output is compact but information-dense. The assistant ran a warmup request first, which took 11.99 seconds for time-to-first-token (TTFT). This extreme latency is characteristic of the first inference call on NVIDIA GPUs using Triton — the CUDA kernels must be JIT-compiled and cached before they can run at full speed. Subsequent requests benefit from the cached kernels, which is why the concurrency-1 row shows an average TTFT of only 0.48 seconds.

The results table uses five columns:

The Reasoning Behind the Benchmark Design

The assistant made several deliberate choices in designing this benchmark. The concurrency levels — 1, 2, 4, 8, 16 — form a geometric progression that reveals how the system scales under increasing load. This is a standard approach for latency-throughput characterization: start with the uncontracted single-request case to establish baseline latency and throughput, then double the load to observe how the continuous batching engine absorbs additional requests.

The choice of 500 tokens per request (2000 tokens across 4 requests at C=1) is also significant. Short requests would measure latency-dominated behavior; long requests would measure throughput-dominated behavior. At ~500 tokens, each request is long enough to reach steady-state generation but short enough to complete within a reasonable wall-clock time for the full benchmark suite. The total benchmark ran for about 76 seconds at C=1 and progressively faster at higher concurrency (48.5s at C=2, 56.1s at C=4 for double the tokens).

The assistant did not include a --num-prompts or request count parameter visible in the invocation, suggesting the benchmark script hardcoded the number of requests per concurrency level. The "Ok" column shows 4 requests at C=1 and C=2, and 8 at C=4 — doubling proportionally with concurrency, which is a sensible design that keeps the total token count roughly balanced across runs.

Assumptions and Limitations

The benchmark makes several implicit assumptions. First, it assumes the model is already loaded and the server is warm — the 11.99s warmup TTFT confirms that the first request triggers kernel compilation, but the benchmark does not measure cold-start time separately. Second, it assumes the network between the two DGX Spark nodes (InfiniBand RoCE at ~640MB/s as observed earlier in the session) is not the bottleneck. For a 122B MoE model split across two nodes with tensor parallelism, inter-node communication is critical; the benchmark results implicitly validate that the IB link is adequate for this workload.

The truncated output is a practical limitation of the bash tool's timeout and output capture. The assistant did not re-run the benchmark to capture the full table, instead relying on the subsequent message to summarize the C=16 results. This is a reasonable trade-off — the key scaling trend is visible even in the partial output.

One notable omission is the absence of variance metrics. The table shows averages but no percentiles (P50, P95, P99) for TTFT or per-request latency. For production deployments, tail latency is often more important than average throughput. The assistant's follow-up analysis in [msg 6730] focused on aggregate throughput comparisons rather than latency distributions, suggesting this was a quick characterization rather than a rigorous production benchmark.

Input Knowledge Required

To fully understand this message, the reader needs to know:

  1. The hardware: Two DGX Spark systems (GB10 Grace Blackwell, ARM Cortex-X925 CPU, 120GB unified LPDDR5x memory, InfiniBand RoCE interconnect). The assistant's earlier summary in [msg 6724] provides this context.
  2. The model: Qwen3.5-122B-A10B-FP8 — a 122B-parameter MoE model with 10B active parameters per token, quantized to FP8. The FP8 quantization reduces memory footprint and bandwidth requirements compared to BF16, which is critical for the Spark's unified memory architecture.
  3. The serving stack: vLLM 0.17.1rc1 (a custom build from hellohal2064/vllm-qwen3.5-gb10), Ray distributed runtime for multi-node orchestration, NCCL over InfiniBand for tensor-parallel communication, and TP=2 (tensor parallelism across the two nodes).
  4. The deployment challenges: The assistant had to solve Ray IP resolution issues (head node registered with external IP unreachable from worker), Ray OOM killer interference during CUDA graph capture, and NCCL socket interface configuration — all documented in the preceding messages.
  5. The benchmark script: Written by the assistant in [msg 6727], though its full source is not shown in the conversation. The script presumably sends chat completion requests to the vLLM OpenAI-compatible API endpoint and measures throughput and latency.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. Single-request throughput: ~27 tok/s for a 122B MoE model on 2x DGX Spark is a meaningful data point for the community. The assistant's follow-up comparison in [msg 6730] notes that the RTX PRO 6000 discrete GPU setup achieves ~277 tok/s — roughly 10x faster — which contextualizes the Spark's unified memory bandwidth limitation.
  2. Scaling behavior: Aggregate throughput scales from 26 tok/s at C=1 to 144 tok/s at C=16 (a 5.5x increase for 16x concurrency), demonstrating that vLLM's continuous batching effectively absorbs concurrent requests even across a multi-node tensor-parallel configuration.
  3. TTFT scaling: Average TTFT increases from 0.48s at C=1 to 3.20s at C=16, a 6.7x increase. This is acceptable for many interactive applications but would be problematic for real-time use cases.
  4. Validation of the deployment: The benchmark confirms that the multi-node Ray cluster, the NCCL InfiniBand transport, the FP8 quantization, and the vLLM server are all functioning correctly under load. Every request completed successfully (the "Ok" column matches expected counts).

The Broader Significance

This benchmark message sits at a critical juncture in the conversation. The preceding ~50 messages document an arduous journey: building SGLang from source, patching Blackwell backend compatibility, debugging NaN outputs, fixing FP8 KV cache accuracy, reconfiguring GPU topology across Proxmox containers, diagnosing P2P DMA corruption under SEV-SNP IOMMU, and finally pivoting from SGLang to vLLM for multi-node support on the DGX Sparks. The benchmark is the first quantitative validation that all this effort has produced a working, performant system.

The 11.99s warmup TTFT is itself a story — it represents the Triton kernel compilation overhead that the assistant had wrestled with in earlier segments when configuring FP4 and MoE backends for Blackwell. The fact that subsequent requests drop to sub-second TTFT confirms that the kernel caching is working correctly.

The benchmark also serves as a baseline for future optimization work. The assistant's analysis in [msg 6730] correctly identifies the bottleneck: "The bottleneck on Spark is the unified memory bandwidth — LPDDR5x is much slower than GDDR7 for weight loading during decode." This diagnosis points toward potential mitigations: smaller quantization (MXFP4), fewer active parameters, or model distillation — all of which the assistant had explored in earlier segments with other models.

Conclusion

Message [msg 6729] is deceptively simple — a single SSH command with a benchmark invocation. But it represents the payoff for an extended, multi-segment engineering effort spanning driver installation, CUDA toolkit configuration, kernel compilation, distributed networking, and memory management. The benchmark numbers are not just performance metrics; they are the quantitative signature of a complex distributed system working correctly across two ARM-based AI accelerators connected by InfiniBand, serving one of the largest open-weight language models available. The assistant's choice of concurrency levels, the interpretation of the warmup penalty, and the subsequent analysis all demonstrate a deep understanding of the inference serving stack and the hardware it runs on.