The Benchmark That Proved It: Validating the CUDA 13 Upgrade with FlashInfer Attention

In the long arc of a complex machine learning infrastructure project, there are moments of grinding frustration — hours spent debugging build errors, wrestling with version mismatches, and chasing dead-end optimizations. And then there are moments of quiet validation: a single command that returns a number confirming the entire effort was worthwhile. Message [msg 5356] in this opencode session is precisely such a moment. It is deceptively simple — a one-line bash command followed by its output — but it represents the culmination of dozens of preceding messages, multiple failed attempts, and a systematic diagnostic process that transformed a potential regression into a genuine performance gain.

The Context: A Desperate Need for CUDA 13

To understand why this message matters, one must appreciate the journey that led to it. The project involved deploying the K2.5 language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5, using SGLang as the inference server. For days, the assistant had been chasing an elusive optimization: enabling FlashInfer allreduce fusion and Torch symmetric memory on Blackwell GPUs (architecture SM120). Repeated attempts had failed because these features required CUDA compute capability support that simply did not exist in the CUDA 12.8 toolkit installed on the system. The error message was unambiguous: "No supported CUDA architectures found for major versions [9, 10]."

The only path forward was a full CUDA stack upgrade to version 13, which added SM120 support. But this was no trivial undertaking. The assistant had to navigate ABI compatibility challenges, rebuild the entire Python environment, source compatible pre-built wheels for sgl-kernel and flashinfer, and carefully manage version dependencies across PyTorch, CUDA runtime libraries, and NCCL. The upgrade process spanned dozens of messages — installing CUDA 13.0.1 alongside the existing 12.8 installation, installing PyTorch 2.9.1 built against cu130, patching SGLang's source code to recognize SM120 in its torch_symm_mem and kimi_k25.py modules, and setting environment variables to bypass irrelevant compatibility checks.

The First Baseline: A Disappointing Regression

When the assistant finally launched the first baseline server under the new CUDA 13 stack ([msg 5340]), the initial results were discouraging. The server started, but the benchmark returned only 83.9 tok/s — a 6% regression from the previous baseline of 89.5 tok/s. This was a critical moment. Had the CUDA 13 upgrade actually hurt performance? The assistant could have accepted this result and moved on, but instead it paused to diagnose.

Examining the server logs ([msg 5352]), the assistant identified two potential culprits. First, the attention backend was set to triton by default, whereas the previous working configuration had likely used flashinfer. Second, DeepGemm was enabled and warning about accuracy degradation on Blackwell. The assistant made a decision: kill the server, restart with --attention-backend flashinfer, and re-benchmark.

This decision was not automatic. The assistant had to reason about which variable to change. The triton attention backend might have different kernel fusion strategies, different memory access patterns, or different CUDA graph compatibility than flashinfer. Given that the entire point of the CUDA 13 upgrade was to enable Blackwell-native optimizations in FlashInfer, it made sense to test with the flashinfer backend first before investigating other potential causes of the regression.

The Message Itself: A Quiet Triumph

The subject message ([msg 5356]) is remarkably terse. The assistant writes simply:

Now run the benchmark:

Then executes a bash command that runs benchmark_eagle3.py against the SGLang server at http://localhost:30000. The output shows five runs with remarkable consistency:

• Run 1: 2048 tokens in 22.13s = 92.6 tok/s
• Run 2: 2048 tokens in 22.14s = 92.5 tok/s
• Run 3: 2048 tokens in 22.14s = 92.5 tok/s
• Run 4: 1837 tokens in 19.80s = 92.8 tok/s
• Run 5: 2048 tokens in 22.11s = 92.6 tok/s The overall result: 92.6 tok/s. This is a 3.5% improvement over the old baseline of 89.5 tok/s, and a stunning 10.4% improvement over the failed triton-backend run of 83.9 tok/s. The consistency of the numbers is itself noteworthy. The standard deviation across runs is negligible — less than 0.1 tok/s. This tells us the system is stable, not subject to thermal throttling, memory bandwidth contention, or other transient factors. The CUDA 13 upgrade, combined with the FlashInfer attention backend, has produced a clean, repeatable improvement.

What This Message Reveals About the Assistant's Thinking

The assistant's reasoning is visible in the structure of this message. It does not simply run the benchmark and report results — it frames the action with "Now run the benchmark," implying that this step follows logically from the previous work of restarting the server with the corrected attention backend. The assistant is treating this as a verification step, not an exploration step. It expects a specific outcome (improvement over the triton backend) and is testing that hypothesis.

The choice to use benchmark_eagle3.py with --server-url http://localhost:30000 also reveals assumptions. The assistant knows the server is on port 30000 (not the default 8000) because it learned this in an earlier message ([msg 5351]) where it discovered the v0.5.9 default port had changed. The benchmark script itself is a custom tool designed for this project, measuring end-to-end token generation throughput with a specific prompt length and generation length configuration (max_tokens=2048, 5 runs with 2 warmup runs).

Input Knowledge Required

To fully understand this message, a reader needs to know:

• The project involves running a large language model (K2.5) on 8× RTX PRO 6000 Blackwell GPUs using SGLang
• The CUDA stack had just been upgraded from 12.8 to 13.0 to enable SM120 support
• The first baseline attempt used the triton attention backend and produced 83.9 tok/s
• The assistant had identified the attention backend mismatch and restarted with --attention-backend flashinfer
• The server took an unusually long time to load (~600s) with the flashinfer backend but eventually started
• The previous stable baseline was 89.5 tok/s

Output Knowledge Created

This message creates several critical pieces of knowledge:

1. The CUDA 13 upgrade is validated. The 3.5% improvement proves that the upgrade was not only necessary for enabling Blackwell-native features but also beneficial for baseline performance.
2. FlashInfer attention backend is the correct choice for this hardware. The 10.4% gap between triton (83.9) and flashinfer (92.6) attention backends on the same CUDA 13 stack is dramatic.
3. The system is stable and consistent. The near-identical per-run results indicate no thermal or resource contention issues.
4. The benchmark methodology is sound. Five runs with warmup provide reliable measurements.

The Broader Significance

This message is a turning point in the session. Immediately after receiving these results, the assistant updates the todo list and declares the baseline validated ([msg 5357]). Then it pivots to the next critical test: enabling FlashInfer allreduce fusion with --enable-flashinfer-allreduce-fusion ([msg 5358]), the optimization that was the original motivation for the entire CUDA 13 upgrade. The assistant has confirmed that the foundation is solid before building the next layer.

In the subsequent messages, the FlashInfer allreduce fusion test succeeds where it had previously failed, and the assistant achieves a breakthrough 96.1 tok/s with EAGLE-3 speculative decoding — transforming it from a net-negative 54.1 tok/s to a net-positive configuration. But none of that would have been possible without the validation in this message. The 92.6 tok/s baseline proved that the CUDA 13 stack was stable and performant, giving the assistant confidence to proceed with the more ambitious optimizations.

This message also illustrates a broader principle of systems engineering: when a complex upgrade produces a regression, the correct response is not to revert but to diagnose. The assistant's willingness to investigate the attention backend mismatch — rather than accepting the 83.9 tok/s result or abandoning the CUDA 13 upgrade — turned a potential failure into a measurable success. The 3.5% baseline improvement from the upgrade alone, before any of the Blackwell-specific optimizations, was a bonus that justified the entire effort even before the bigger wins materialized.