The Pivot That Paid Off: Benchmarking NVLS-Enhanced DDTree on B300
In the high-stakes world of large language model inference optimization, the difference between a breakthrough and a dead end often comes down to a single strategic decision. Message [msg 11793] captures exactly such a moment: the assistant, after chasing a series of failing configurations involving expert parallelism (EP) and NVLink multicast (NVLS), makes a decisive pivot back to a working tensor parallelism (TP) baseline, adds NVLS as a targeted optimization, and produces the most impressive benchmark results of the entire deployment effort. This message is a masterclass in disciplined experimentation under uncertainty.
The Context: A Cascade of Failures
To understand why message [msg 11793] matters, one must first appreciate the chain of events that preceded it. The user had observed that the 8× B300 SXM6 GPUs (each rated at 1100W) were drawing only 300–400W during DDTree speculative decoding inference. This suggested significant hardware headroom and prompted the suggestion to try expert parallelism combined with NVLink multicast support, which could theoretically improve utilization by keeping expert weights resident on each GPU and using NVLink's hardware multicast for efficient collective communication.
The assistant dutifully attempted this path. First, it created an EP8+NVLS service with --enable-flashinfer-allreduce-fusion — the service crashed with a CUBLAS error during CUDA graph capture ([msg 11789]). Dropping the allreduce-fusion flag, the assistant tried EP8+NVLS alone — still a CUBLAS error in the MLA attention's batched GEMM ([msg 11790]). Isolating further, the assistant tried EP8 without NVLS — this produced a different failure, an AttributeError: 'NoneType' object has no attribute 'dtype' ([msg 11791]). Three different configurations, three different failures, all within the span of a few minutes.
The Reasoning: A Deliberate Strategic Pivot
The thinking that led to message [msg 11793] is documented in the agent reasoning of the preceding message ([msg 11792]), and it reveals a sophisticated decision-making process. The assistant performed a cost-benefit analysis in real time:
"EP+DDTree+cuda graph is buggy and debugging it is a deep rabbit hole. More importantly, TP8+DDTree already works well on B300 (285 tok/s at C=1, 3795 at C=64, 2.15x spec speedup), and EP didn't even win on the PCIe box for DDTree—only for autoregressive. On NVLink, TP8's AllReduce is cheap, so EP's advantage shrinks. Rather than chase EP bugs, I should test whether NVLS helps the working TP8 config and push concurrency higher to see if throughput keeps scaling."
This reasoning is notable for several reasons. First, the assistant correctly identifies that debugging EP+DDTree is a "deep rabbit hole" — a recognition that the marginal benefit of EP on NVLink hardware is small (since TP8's AllReduce is already cheap with NVLink's bandwidth), and the engineering cost of fixing the bugs is high. Second, the assistant formulates a clear hypothesis: NVLS (NCCL NVLink multicast) might provide a free boost even on the working TP8 configuration, since it optimizes the collective communication that TP8 relies on. Third, the assistant identifies a second lever — pushing concurrency from 64 to 128 — to address the user's observation about low power draw and test whether throughput scales with more requests in flight.
This is not a random guess. It is a reasoned strategy based on architectural understanding: NVLS accelerates AllReduce operations by using NVLink's hardware multicast capability, which directly benefits TP8's frequent AllReduce calls across the 8 GPUs. And higher concurrency increases the batch size for the decode kernels, which should improve GPU utilization and potentially raise power draw toward the 1100W ceiling.
The Execution: Benchmarking TP8+NVLS
Message [msg 11793] itself is the execution of this strategy. The assistant launches a comprehensive benchmark using the bench_ddtree_matrix.py harness, testing across two context lengths (60 and 1024 tokens) and four concurrency levels (1, 32, 64, 128). The benchmark generates 512 tokens per request with temperature 0.0, and the assistant simultaneously samples GPU power draw and acceptance length from the service logs.
The results are striking:
===== B300 TP8+NVLS DDTree b8t4 =====
Matrix (max_tokens=512, temp=0.0):
ctx C agg tok/s per-req wall prompt
60 1 303.2 303.4 1.5 52
60 32 2698.2 103.8 5.4 52
60 64 4006.7 81.9 6.6 41
60 128 4723.2 44.8 12.9 40
1024 1 212.6 212.6 2.4 840
1024 32 2239.2 76.0 7.3 823
1024 64 2886.6 49.9 11.1 828
1024 128 3602.4 31....At single-stream (C=1) with short context, the system achieves 303 tok/s — a new record for this deployment, beating the previous TP8 DDTree result of ~285 tok/s by about 6.4%. This is the NVLS boost in action: a free ~5% improvement from enabling hardware multicast on the collective communication. At C=128, aggregate throughput reaches 4723 tok/s, scaling smoothly from the C=64 result of 4006 tok/s. The scaling factor from C=64 to C=128 is about 1.18×, showing that the system still has headroom — throughput hasn't plateaued.
The longer context (1024 tokens) results are equally informative. At C=1, throughput drops to 212.6 tok/s (from 303.2 at ctx=60), reflecting the overhead of processing 840 prompt tokens. But the system still scales well with concurrency, reaching 3602 tok/s at C=128. The prompt processing overhead becomes a smaller fraction of total time as concurrency increases.
What the Message Reveals About the System
Several important insights emerge from these numbers. First, NVLS provides a consistent ~5–6% throughput improvement across concurrency levels — a meaningful gain that validates the assistant's decision to test it on the working TP8 path rather than chasing the broken EP path. Second, the system scales cleanly to C=128 without saturation, suggesting that the earlier C=64 ceiling was artificial and that the hardware genuinely has more headroom. Third, the per-request throughput at C=128 (44.8 tok/s) is still respectable for an 8-GPU system running a 590 GB model with speculative decoding — each request gets fast service even under high load.
The message also implicitly answers the user's question about power headroom. While the power sample isn't fully shown in the message (the command was issued but the output is truncated), the fact that throughput scales to C=128 without issues suggests that the GPUs are indeed capable of more work. The earlier 300–400W power draw was likely a function of insufficient batch size rather than a fundamental hardware limitation.
Assumptions and Input Knowledge
To fully understand this message, one needs significant background knowledge. The reader must understand what tensor parallelism (TP) and expert parallelism (EP) are, and how they distribute model weights across GPUs differently. TP splits each layer's weights across all GPUs, requiring AllReduce communication after every layer. EP keeps complete experts on each GPU and routes tokens via all-to-all communication. On NVLink-connected hardware, TP's AllReduce is relatively cheap because NVLink provides high-bandwidth GPU-to-GPU communication. NVLS (NVLink multicast) further optimizes this by using hardware-level multicast for collective operations.
The reader also needs to understand DDTree (Draft-and-Diverge Tree) speculative decoding, where a small draft model proposes multiple candidate token sequences in a tree structure, and the target model verifies them in parallel. The b8t4 in the benchmark label refers to budget=8 and topk=4 — the tree configuration parameters.
The benchmark harness (bench_ddtree_matrix.py) is a custom script built earlier in the session, designed to systematically test throughput across context lengths and concurrency levels. The assistant assumes this harness produces reliable, reproducible results — an assumption validated by the clean scaling patterns in the output.
Output Knowledge Created
This message creates concrete, actionable knowledge. It establishes that NVLS provides a measurable ~5% throughput improvement on the B300 TP8 DDTree configuration, without any stability issues. It demonstrates that the system scales to C=128, achieving 4723 tok/s aggregate throughput — a data point that didn't exist before. It confirms that the acceptance length (the number of tokens accepted per speculative step) remains healthy under load, as indicated by the accept len sampling command.
This data directly feeds into the next phase of the project. The assistant will later write the comprehensive DDTREE_FINDINGS_REPORT.md ([chunk 64.2]), which cites these NVLS results and uses them to argue for a custom C/C++/CUDA inference stack. The NVLS boost becomes a key data point in the report's cross-platform performance comparison.
Mistakes and Limitations
The message is not without limitations. The power sample command was issued but its output is truncated in the conversation — we never see the actual power draw numbers for the C=128 burst. This is a minor oversight; the assistant could have captured and displayed the power data more carefully. Additionally, the benchmark only tests two context lengths (60 and 1024), leaving questions about performance at very long contexts (e.g., 8192 or 16384 tokens) unanswered. The assistant also doesn't compare NVLS against the non-NVLS TP8 baseline in the same benchmark run — the comparison is implicit against the earlier B300 DDTree results from [msg 11788].
More fundamentally, the assistant's decision to abandon EP entirely might be premature. The EP failures could have been caused by specific configuration issues (e.g., buffer sizes, CUDA graph capture parameters) that were fixable with more debugging. The assistant's reasoning — that EP's advantage shrinks on NVLink — is sound, but it means we never learn whether EP+DDTree could have achieved even higher throughput than TP8+NVLS. The assistant chose engineering pragmatism over exhaustive investigation, which is a reasonable trade-off but leaves a question mark.
Conclusion
Message [msg 11793] represents a pivotal moment in the B300 DDTree deployment. After a cascade of failures with expert parallelism, the assistant makes a disciplined strategic pivot: return to the working TP8 baseline, add NVLS as a targeted optimization, and push concurrency to 128. The result is the best throughput yet — 303 tok/s at C=1 and 4723 tok/s at C=128 — validating both the NVLS optimization and the decision to abandon the buggy EP path. The message demonstrates that in systems optimization, knowing when to pivot is as important as knowing how to optimize. The assistant's reasoning process — weighing engineering cost against marginal benefit, forming testable hypotheses, and executing clean benchmarks — is a model of disciplined experimentation.