The Build That Unlocked 3–6× Decode Speedup

A Single Bash Command and What It Meant

In the middle of a high-stakes optimization session for a custom speculative decoding engine running on NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant issued a single bash command. On its face, the message is unremarkable—a remote build invocation piped through tail -20 to keep the output tidy:

[assistant] [bash] timeout 300 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'cd /root/kdtree-engine && CUDA_HOME=/usr/local/cuda-13.0 bash scripts/build_nvcc.sh 2>&1 | tail -20'
[1/5] kernel objects
[2/5] engine objects
[3/5] shared C-ABI lib
[4/5] kernel unit tests + bench
[5/5] engine tests + demo (needs cublas)
done -> build/

Five build stages, all green. "done -> build/". The message is terse, almost anticlimactic. But this build was the culmination of an intense engineering effort: the assistant had just written a custom flash attention CUDA kernel from scratch—the verify_attn_flash.cu—designed specifically for the sm_120 architecture found on Blackwell consumer GPUs. This kernel would go on to deliver a 3–6× end-to-end decode speedup over the existing Triton-based implementation, transforming the performance profile of the entire speculative decoding pipeline.

This article examines that single message: why it was written, what it reveals about the engineering process, and how a seemingly mundane build step can represent a critical inflection point in a complex optimization campaign.

The Context: A Custom Kernel for an Unsupported Architecture

To understand why this build mattered, we need to step back. The assistant had been deploying the Kimi K2.6 model with DFlash speculative decoding on RTX PRO 6000 Blackwell GPUs (compute capability sm_120). The existing verify attention kernel—the component that scores and merges candidate tokens from the draft model against the target model's predictions—was implemented in Triton and used the standard MLA (Multi-head Latent Attention) approach. At long context lengths, this kernel was catastrophically slow: decode throughput dropped to 0.7 tokens per second at 185k tokens, with GPU tensor core utilization at only ~3%.

The root cause was identified in earlier analysis ([msg 12225] and surrounding messages): the Triton MLA kernel used page_size=1 for KV cache access, causing scattered global memory reads at approximately 14 GB/s effective bandwidth—roughly 130× below the GPU's 1.8 TB/s peak. The assistant investigated whether optimized MLA kernels from the ecosystem (FlashMLA, cutlass-MLA, flashinfer-MLA) could help, but discovered that all of them were compiled exclusively for sm_90a (Hopper), sm_100a, or sm_103a (Blackwell数据中心). None supported sm_120, the architecture of the RTX PRO 6000 Blackwell consumer card. The user gave the go-ahead to build an owned sm_120 kernel.

What followed was an intensive design and implementation phase. The assistant wrote a detailed per-phase plan (plans/0002-sm120-verify-kernel-defrag.md) and then implemented Phase 1: a KV-split flash-decode MLA verify kernel. The key innovation was online softmax with KV tiling: instead of loading the entire KV cache into shared memory (which the naive kernel required, and which failed at prefixes beyond ~25k due to the 100 KB shared memory limit), the flash kernel streams KV tiles into shared memory one at a time, recomputing the softmax normalization online. This eliminated the shared memory bottleneck and, more importantly, reduced global memory reads of the KV latent from 64× per query (one per head) down to 8× (one per head-tile), yielding a theoretical 8× reduction in the dominant memory cost.

The assistant wrote the kernel, integrated it into the C-ABI interface, extended the test suite to A/B test both kernels against golden reference outputs, and added benchmark entries. All of this happened across messages [msg 12227] through [msg 12234]. Then, in message [msg 12235], the assistant synced the entire repository to the CT200 server via rsync. Message [msg 12236]—the subject of this article—is what came next: the build.

Why This Build Was Different

A build is a build. In a typical development workflow, compiling code is a routine checkpoint—you write, you compile, you fix errors, you repeat. But this build carried unusual weight for several reasons.

First, it was a remote build on target hardware. The kernel was developed on a local machine but needed to compile and run on the CT200 server, which had the actual RTX PRO 6000 Blackwell GPUs. The assistant used CUDA_HOME=/usr/local/cuda-13.0 to point to the correct CUDA toolkit. Any mismatch between the local development environment and the remote deployment target—different CUDA versions, different driver versions, different GPU architectures—could cause compilation failures that wouldn't appear locally. The rsync in the previous message had already succeeded, but rsync doesn't compile code. This build was the first real test of whether the kernel source was compatible with the target environment.

Second, the build script had been modified. The assistant had edited scripts/build_nvcc.sh ([msg 12230]) to add verify_attn_flash.cu to both the KOBJ list and the compilation loop. This was a surgical change to an existing build system, and any mistake—a typo in the filename, an incorrect path, a missing dependency—would surface here. The clean build output confirmed that the edit was correct and that the new kernel object compiled and linked without issues.

Third, the build encompassed multiple artifacts. The five-stage output reveals the full scope:

Kernel objects — The CUDA kernels themselves, including the new verify_attn_flash.cu alongside the existing verify_attn.cu, tree_accept.cu, and capi.cu.
Engine objects — The higher-level engine code that orchestrates kernel launches.
Shared C-ABI lib — The shared library exposing a C-compatible interface, which SGLang loads at runtime.
Kernel unit tests + bench — The test binaries that would validate correctness and measure performance.
Engine tests + demo — Higher-level integration tests (noted as needing cublas, which was available on the target). Each stage building successfully meant that the entire software stack—from low-level CUDA kernels to the C-ABI layer to the test harness—was coherent.

The Assumptions Embedded in a Single Command

The build command encodes several assumptions, all of which proved correct:

The rsync had transferred all necessary files. The assistant assumed that the source tree was complete and that no files were missed. The --exclude patterns in the rsync (build artifacts, git metadata, pycache, reference model files) were carefully chosen to exclude regenerable or platform-specific content while keeping everything needed for compilation.
The CUDA toolkit 13.0 supports sm_120. The CUDA_HOME environment variable pointed to /usr/local/cuda-13.0. This assumption was non-trivial: sm_120 is a relatively new architecture, and not all CUDA toolchains support it. The build succeeding confirmed that CUDA 13.0 includes the necessary architecture targets and that nvcc can generate code for sm_120.
The build script handles the new files correctly. The assistant's edit to the build script added verify_attn_flash to the compilation. The assumption was that the script's existing loop structure—which iterates over kernel source files and compiles each one with the appropriate flags—would work seamlessly with the new file. It did.
The remote machine has sufficient resources. Building CUDA kernels is memory-intensive. The timeout 300 (5-minute limit) was a safety measure against hangs, but the assumption was that the CT200 server had enough RAM, disk space, and GPU memory headroom to complete the build. With the production SGLang service consuming ~94 GB per GPU, this was a real concern—but the build completed well within the timeout.
The kernel source is syntactically correct. This is the most fundamental assumption. The assistant had written the flash kernel in message [msg 12227], with careful attention to shared memory budgets, thread mapping, and online softmax logic. But until nvcc parses the code, syntax errors, type mismatches, and template instantiation failures are all possible. The clean build validated the assistant's implementation.

What the Build Output Reveals (and What It Doesn't)

The output is deliberately minimal: five status lines and a completion message. The tail -20 filter hides the verbose compiler output, showing only the stage markers. This terseness is itself a design choice—the assistant prioritized signal over noise. A build that succeeds doesn't need detailed diagnostics; a build that fails would need the error messages, which would appear in the truncated output if they occurred in the last 20 lines.

The build output reveals that compilation succeeded, but it does not reveal whether the kernel produces correct results. Correctness is tested in the next message ([msg 12237]), where the assistant runs the verification tests against golden reference outputs. The build output also doesn't reveal performance—that comes later in the benchmark phase ([msg 12238]), where the initial microbenchmarks show the flash kernel running at 0.6× the naive kernel at short prefixes, improving to 0.9× at 16k, and handling 65k prefixes that the naive kernel simply cannot run.

This separation of concerns—build, then correctness, then performance—is a hallmark of disciplined engineering. Each gate must be passed before the next one opens.

The Broader Significance

In the larger arc of the session, message [msg 12236] is the quiet moment before the storm. The build succeeds, the tests pass (next message), and then the assistant pivots to making the kernel capture-safe for CUDA graphs, optimizing occupancy, and implementing KV defragmentation. The 3–6× decode speedup that eventually materializes is built on the foundation of this clean build.

But there's a deeper lesson here about the nature of optimization work. The flash kernel's initial microbenchmark results were underwhelming—0.6× speedup at short prefixes, barely reaching parity at longer ones. The naive kernel was faster at short contexts because it had simpler control flow and fewer tile iterations. It was only after the assistant profiled the TP8 regime and discovered that occupancy starvation was the real bottleneck—not memory bandwidth—that the kernel was retuned with larger NSPLIT and vectorized loads, unlocking the dramatic 3–6× gain.

The build in message [msg 12236] didn't guarantee success. It only guaranteed that the code was syntactically valid. The real work—the iterative diagnosis, profiling, and tuning—came after. But without this build, none of that could happen. Every optimization journey begins with a clean compile, and this message captures that essential, unglamorous truth.

Conclusion

Message [msg 12236] is a single bash command that builds a custom CUDA kernel on a remote server. It is brief, almost perfunctory. But in the context of the session, it represents the transition from design to reality, from source code to executable artifact. The assistant had designed a flash attention kernel tailored to the sm_120 architecture, written it, integrated it into a complex build system, synced it to the target hardware, and now—with five lines of build output—confirmed that it compiles. The gates were open for testing, profiling, and the iterative optimization that would follow.

The message is a reminder that even the most sophisticated engineering work rests on mundane foundations. A build is not glamorous, but it is necessary. And when it succeeds, it creates the conditions for everything that follows.