From Planning to Execution: The Pivot Point in Building a Custom sm_120 Verify Kernel

Introduction

In the middle of a months-long effort to deploy speculative decoding for a large language model on Blackwell RTX PRO 6000 GPUs, there comes a moment when planning must yield to building. Message [msg 12215] captures precisely that transition. The assistant, having just received the user's directive to "Write down a plan/spec for each phase, then execute," responds by shifting mental gears from architectural deliberation to hands-on implementation. The message is brief—barely a reasoning paragraph followed by a single bash command and its output—but it represents a critical inflection point in the engineering workflow. This article examines that message in depth: why it was written, what decisions it embodies, the assumptions it carries, and the thinking process that bridges a comprehensive plan to its concrete realization.

The Context: A Plan Already Written

To understand [msg 12215], one must first understand what immediately preceded it. In [msg 12213], the assistant produced an extraordinarily detailed execution plan for building an owned sm_120 DDTree verify kernel and integrating it into SGLang, along with a K/V defragmentation strategy. That plan was the culmination of days of investigation: the assistant had discovered that every optimized MLA attention kernel (FlashMLA, cutlass-MLA, flashinfer-MLA) ships only for sm_90a, sm_100a, and sm_103a architectures—none support sm_120, the architecture of the RTX PRO 6000 Blackwell GPUs in use. The service was running on Triton JIT-compiled kernels, which were fundamentally memory-bound, achieving only ~14 GB/s effective bandwidth against a 1.8 TB/s peak. The plan laid out four phases: scaffolding and parity harness (Phase 0), the flash MLA verify kernel itself (Phase 1), the SGLang custom backend (Phase 2), live validation (Phase 3), and K/V defrag (Phase 4). It specified guiding engineering choices, validation gates, risks, and deliverables.

The user's response in [msg 12214] was succinct: "Write down a plan/spec for each phase, then execute." This is not a request for more planning—the assistant had already written a plan. The user is asking for two things: first, formalize the plan into a written specification document (the "plan/spec for each phase"), and second, begin executing it. The user is signaling that deliberation is over; it is time to build.

The Message Itself: A Shift in Modality

Message [msg 12215] opens with the assistant's reasoning:

I'm shifting into execution mode now. I need to set up a structured plan document first, then work through each phase methodically. Let me start by examining the existing repo structure to understand the context before I begin. I need to explore the repository structure to locate the exact paths for these kernel files, build scripts, and tests so I can read them systematically. I'm in build mode now. I'll write the per-phase spec doc, then execute starting with Phase 0/1. First let me ground the spec in the actual code — reading the existing kernel, C-ABI, harness, and build scripts.

This reasoning reveals a deliberate cognitive shift. The assistant is consciously moving from "planning mode" to "execution mode" or "build mode." This is not a trivial change—it involves reorienting from abstract reasoning about architecture and trade-offs to concrete actions like reading files, understanding existing code structure, and writing code. The assistant recognizes that to write a meaningful spec document, it must first ground itself in the actual repository layout. It needs to know where the existing kernel files live, how the build scripts work, what test harnesses exist, and how the C-ABI layer is structured. Only with this concrete knowledge can it produce a spec that accurately reflects the codebase it will modify.

The assistant then executes a bash command to explore the repository:

cd /home/theuser/glm-kimi-sm120-rtx6000bw/kdtree-engine && echo "=== tree ===" && ls -R src/kernels tests scripts 2>/dev/null | head -60 && echo "=== python ===" && ls python/

The output reveals the project structure: src/kernels/ contains capi.cu, tree_accept.cu, tree_accept.cuh, tree_build.cu, tree_build.cuh, verify_attn.cu, and verify_attn.cuh. The tests/ directory contains bench_kernels.cu, various test files, and reference data. The scripts/ directory has build_and_test.sh, build_nvcc.sh, and run_tests_nvcc.sh. This exploration is the first concrete step of Phase 0—understanding the scaffolding before building upon it.

Why This Message Matters

On the surface, [msg 12215] appears to be a simple reconnaissance step: look at the repo, see what exists. But its significance lies in what it represents. This is the moment when an ambitious engineering plan—the product of days of investigation, debugging, profiling, and architectural reasoning—collides with the messy reality of an existing codebase. The plan in [msg 12213] speaks in abstractions: "one streaming kernel, no merge_state," "KV-tile outer / heads inner," "subclass TritonAttnBackend." These are sound architectural decisions, but they cannot be implemented without understanding the exact shape of the existing verify_attn.cu, the C-ABI interface in capi.cu, the test infrastructure, and the build system.

The assistant's reasoning makes this explicit: "First let me ground the spec in the actual code." This is a recognition that a spec written in isolation, without reference to the actual codebase, would be fragile. It might propose interfaces that don't match existing conventions, or suggest modifications that conflict with the existing architecture. By examining the repository first, the assistant ensures that the written spec will be implementable.

The Thinking Process: Methodical and Grounded

The reasoning in [msg 12215] reveals several characteristics of the assistant's thinking process:

Sequential decomposition. The assistant breaks the task into clear steps: first write the structured plan document, then execute phase by phase. Within that, it further decomposes: first explore the repo, then read the key files, then write the spec, then implement. This is classic engineering methodology—decompose a complex task into manageable, ordered steps.

Grounding before abstraction. Rather than writing the spec from memory of the plan, the assistant insists on re-examining the actual code. This is a deliberate choice to prioritize accuracy over speed. The assistant could have written the spec immediately based on the plan in [msg 12213], but it chooses to verify its understanding against the real codebase first. This reflects an awareness that plans, no matter how well-reasoned, can contain hidden assumptions that only surface when confronted with actual code.

Awareness of the transition. The assistant explicitly names the shift it is making: "shifting into execution mode," "in build mode now." This metacognitive awareness is notable—the assistant recognizes that different modes of thinking are required for planning versus building, and it consciously switches between them. This is a sophisticated cognitive behavior that mirrors how experienced engineers operate.

Assumptions and Their Implications

The message carries several implicit assumptions:

That the existing repository structure is stable and representative. The assistant assumes that the files it sees in ls -R are the files it will need to modify. This is a reasonable assumption, but it could be wrong if the repository has been restructured or if there are uncommitted changes. The assistant does not check git status or verify that the working tree is clean.

That the build scripts (build_nvcc.sh, etc.) are the correct entry points for compilation. The assistant assumes that the existing build infrastructure is appropriate for the new kernel. This is likely correct, but the new kernel may require different compilation flags or additional dependencies that the existing scripts don't handle.

That reading the files will be sufficient to understand the architecture. The assistant plans to "read them systematically," but reading alone may not reveal runtime behaviors, implicit conventions, or the interactions between components. Some knowledge may only be accessible through building and running tests.

That the spec can be written after reading the code, before implementing. This assumes a clean separation between specification and implementation. In practice, writing the spec often reveals ambiguities that require further investigation, and implementation inevitably uncovers issues that the spec didn't anticipate. The assistant's approach is sound, but it should be prepared for the spec to evolve during implementation.

Input Knowledge Required

To fully understand [msg 12215], a reader needs:

Knowledge of the project context. The message is part of a larger effort to deploy the GLM-5-NVFP4 / Kimi K2.6 model with DFlash speculative decoding using DDTree on RTX PRO 6000 Blackwell GPUs. Without this context, the references to "verify_attn," "tree_accept," and "sm_120" are opaque.

Knowledge of CUDA GPU architecture. The distinction between sm_90 (Hopper), sm_100/sm_103 (Blackwell data center), and sm_120 (Blackwell consumer/RTX PRO) is central to the entire effort. sm_120 lacks the warp-group matrix multiply instructions (wgmma) and Tensor Memory Accelerator (TMA) available on Hopper and Blackwell data-center GPUs, which is why existing optimized kernels cannot be used.

Knowledge of speculative decoding and DDTree. The "verify" kernel performs the verification step in speculative decoding: it computes attention over the draft tree to determine which tokens to accept. The tree mask and KV indirection are specific to the DDTree algorithm.

Knowledge of SGLang's attention backend architecture. The plan to subclass TritonAttnBackend and override only the verify method requires understanding SGLang's AttentionBackend ABC, its registry, and how custom masks are passed.

Output Knowledge Created

This message produces several forms of knowledge:

Repository topology. The output of the bash command provides a concrete map of the codebase. We learn that verify_attn.cu and verify_attn.cuh exist alongside capi.cu (the C-ABI layer), that there are build scripts in scripts/, and that tests include both unit tests (test_verify_attn.cu) and benchmarks (bench_kernels.cu). Reference data for tests lives in tests/refs/ as .kdtr files.

The starting point for Phase 0. The assistant now knows exactly which files to read next: verify_attn.cu for the existing naive kernel, capi.cu for the C-ABI interface, build_nvcc.sh for the compilation flags, and test_verify_attn.cu for the test harness. This grounds the next steps.

Confirmation of the development approach. The repository structure confirms that the project already has a well-organized kernel development workflow with separate source, test, and script directories. The assistant can follow existing patterns rather than inventing new ones.

The Broader Significance

Message [msg 12215] exemplifies a pattern that recurs throughout complex engineering projects: the transition from planning to execution. Plans are seductive—they allow us to reason about ideal solutions without the mess of implementation. But every plan eventually meets reality, and that meeting is where engineering truly happens. The assistant's disciplined approach—grounding the spec in actual code before writing it, explicitly naming the cognitive shift, decomposing the work into ordered steps—is a model for how to make that transition successfully.

The message also highlights the importance of concrete knowledge. The plan in [msg 12213] was sophisticated and well-reasoned, but it could not have been executed without knowing the exact paths, file names, and conventions of the codebase. The bash command in [msg 12215] is not busywork—it is the essential first step of Phase 0, the scaffolding that makes everything else possible.

Conclusion

Message [msg 12215] is a pivot point. It marks the moment when the assistant stops asking "what should we build?" and starts asking "how do we build it?" The reasoning reveals a methodical, grounded approach: explore the repository, understand the existing code, then write the spec and implement. The assumptions are reasonable, the thinking process is clear, and the output—a concrete map of the codebase—provides the foundation for the work to follow. In the broader narrative of this coding session, this message is where planning becomes execution, and where abstract architecture meets the real files that will be changed.