The Quiet Art of Integration: Wiring a Custom CUDA Kernel into a Living Codebase

Introduction

In any substantial software engineering effort, the most celebrated moments are those of creation: the design document that crystallizes a new architecture, the kernel implementation that translates mathematical insight into parallel execution, the benchmark that reveals a dramatic speedup. But between creation and validation lies a less glamorous, equally essential phase: integration. This is the work of connecting a new component to the existing codebase — adding declarations to headers, updating build configurations, extending test suites, and wiring up benchmark harnesses. It is the plumbing that makes creation usable.

Message <msg id=12228> captures one such integration moment in the development of a custom CUDA flash attention kernel for the NVIDIA RTX PRO 6000 Blackwell (sm_120) architecture. The message itself is brief — a single edit to a header file, accompanied by a concise reasoning note. But that brevity belies the depth of context, decision-making, and engineering discipline that surrounds it. This article examines that message in detail: why it was written, what decisions it reflects, the assumptions it encodes, and the knowledge it both consumes and produces.

The Message

The subject message reads in full:

## Agent Reasoning Now I'm adding the function declaration to the verification header, integrating it into the main CUDA file, updating the build configuration, extending the test suite to validate the results, and adding a benchmark entry to track performance. Now wire it into the header, C-ABI, build script, test, and bench: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/kdtree-engine/src/kernels/verify_attn.cuh Edit applied successfully.

At first glance, this appears to be a routine edit: add a function declaration to a header file. But the reasoning reveals that this single edit is the first domino in a chain of integration steps. The assistant is not merely adding a declaration; it is initiating the process of "wiring" a newly written kernel into every layer of the software stack — the C-ABI interface that makes it callable from Python, the build script that compiles it, the test suite that validates its correctness, and the benchmark harness that measures its performance.

The Context: What Led Here

To understand why this message was written, one must understand the journey that preceded it. The assistant had been engaged in a multi-phase effort to deploy and optimize the Kimi K2.6 model with speculative decoding on Blackwell GPUs. A critical bottleneck had been identified: the "verify attention" kernel — the component that computes attention scores during the tree-verification step of speculative decoding — was performing poorly at long context lengths.

The root cause was architectural. The existing verify attention implementation used Triton (a Python-based compiler for GPU kernels) with a page size of 1, causing scattered, non-coalesced memory access to the KV cache. Effective bandwidth was approximately 14 GB/s — roughly 130× below the GPU's peak of 1.8 TB/s. The KV cache's MLA (Multi-head Latent Attention) format stores a compressed latent vector per token that is shared across all attention heads, but the naive kernel re-reads this latent from global memory for each head independently, multiplying memory traffic by the number of heads (64 in the Kimi K2.6 model).

The assistant had spent the preceding messages designing and implementing a custom flash attention kernel to address this bottleneck. The design process, captured in <msg id=12225> and <msg id=12227>, was extensive and iterative. The assistant explored multiple architectural approaches:

• A register-tiled design where one block handles all 64 heads, distributing the output accumulator across threads' registers
• A grouped-heads approach where heads are processed in batches (head-tiles) to keep the shared memory footprint manageable
• Various tile size configurations (HT=8 with TK=16, HT=16 with TK=16, HT=8 with TK=32) balancing shared memory budget against latency reduction The final design settled on a block structure where each block processes one query position and a tile of heads (HT=8), loading the KV latent into shared memory once and reusing it across all heads in the tile. This reduces global memory reads of the KV cache by a factor of 2×HT = 16× compared to the naive approach, while keeping shared memory usage under the 100 KB limit of the sm_120 architecture. Online softmax enables processing arbitrary-length prefixes without storing full score matrices. The kernel implementation was written in <msg id=12227> as verify_attn_flash.cu — a substantial CUDA source file containing the flash attention kernel with KV-tiling, online softmax, and masked attention support. But a kernel source file alone is not useful; it must be compiled, linked, and made callable from the rest of the system. That is the purpose of message <msg id=12228>.## The Integration Chain The assistant's reasoning enumerates five distinct integration targets: 1. The header file (verify_attn.cuh) — adding the function declaration so that other compilation units can reference the kernel 2. The main CUDA file (capi.cu) — integrating the kernel into the C-ABI shim that exposes it to Python via ctypes 3. The build configuration — ensuring the new source file is compiled as part of the project 4. The test suite — adding correctness tests that compare the flash kernel's output against the naive oracle 5. The benchmark harness — adding a benchmark entry to track the kernel's performance across different configurations This is a textbook example of the "edit one file, propagate everywhere" pattern that characterizes robust software engineering. Each integration point serves a distinct purpose: the header makes the kernel visible to the compiler; the C-ABI makes it callable from Python; the build configuration makes it part of the binary; the tests ensure it produces correct results; and the benchmarks ensure it actually improves performance. The message's edit — adding a declaration to verify_attn.cuh — is the first and most fundamental step. Without it, subsequent integration steps would fail at compile time. The header file is the contract between the kernel implementation and its consumers: it declares the function signature, the types of its parameters, and any compile-time constants that callers must agree upon. By editing this file first, the assistant establishes the interface that all other integration points will depend on.

Assumptions Embedded in the Message

The message, though brief, encodes several significant assumptions:

Assumption 1: The kernel is correct. The assistant is proceeding with integration before running any tests on the newly written kernel. This is a deliberate workflow choice: write the kernel, wire it into the build system, then compile and test everything together. The assumption is that any compilation errors or correctness bugs will be caught in the subsequent build-and-test cycle. This is a reasonable assumption for an experienced CUDA developer working in a well-structured project with existing test infrastructure.

Assumption 2: The interface is stable. By adding the function declaration to the header, the assistant implicitly assumes that the kernel's signature will not change during debugging. If the kernel requires additional parameters or a different memory layout, the header will need to be updated, and all downstream integration points will need corresponding changes. The assistant's confidence likely stems from having designed the kernel against the existing oracle interface, ensuring compatibility from the start.

Assumption 3: The build system can handle the new file. The project uses a custom build script (build_nvcc.sh) rather than CMake or Make. The assistant assumes that adding the new source file to the build script is straightforward — a matter of appending the filename to a list of source files. This is a safe assumption given the project's structure, but it reflects a preference for simple, explicit build configurations over complex meta-build systems.

Assumption 4: The test infrastructure is extensible. The existing test suite, driven by run_tests_nvcc.sh, generates reference data using Python scripts and compares kernel outputs against these references. The assistant assumes that adding a new test case — comparing the flash kernel's output against the same oracle — is a matter of adding a few lines to the test runner. This is a reasonable assumption, but it depends on the flash kernel producing bit-exact results with the oracle, which may not hold if numerical precision differences arise.

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains:

CUDA programming model: Understanding that a kernel written in a .cu file must be declared in a header (.cuh) to be visible to other compilation units, and that the C-ABI shim (capi.cu) uses extern "C" to export functions that Python's ctypes can call.

Project architecture: The kdtree-engine project is structured with kernel source files in src/kernels/, headers in the same directory, a C-ABI wrapper in capi.cu, a build script in scripts/, tests in tests/, and benchmarks in the same directory. Understanding this layout is necessary to know which files need editing.

Attention mechanism: The kernel implements MLA (Multi-head Latent Attention) with absorbed KV computation, a specialized attention variant used by DeepSeek and Kimi models. The verify attention operation is central to speculative decoding with draft trees.

SGLang integration context: The kernel is ultimately destined to replace Triton-based attention in the SGLang serving framework. The C-ABI interface is the bridge between the custom CUDA kernel and SGLang's Python runtime.

Output Knowledge Created

This message creates knowledge in the form of an updated header file that declares the new flash attention kernel. This declaration serves as:

• A compile-time contract between the kernel implementation and its callers
• A documentation artifact that specifies the kernel's parameters and their types
• A dependency anchor that the build system uses to determine compilation order
• A discovery point for developers who need to understand what kernels are available The message also creates procedural knowledge: the assistant's reasoning documents the sequence of integration steps, providing a roadmap for future kernel additions. Any developer who needs to add a new CUDA kernel to this project can follow the same pattern: write the kernel, declare it in the header, wire it into the C-ABI, add it to the build, test it, and benchmark it.

The Thinking Process

The assistant's reasoning reveals a methodical, plan-driven approach. The phrase "Now I'm adding the function declaration" signals that the assistant is executing a pre-established plan — the spec document written in <msg id=12226> and the todo list created in <msg id=12225>. The assistant is not improvising; it is following a checklist.

The reasoning also reveals an awareness of the full integration surface. The assistant does not say "add the declaration to the header" — it enumerates five distinct integration targets, demonstrating a systems-level understanding of how a kernel moves from source file to running service. This breadth of awareness is characteristic of experienced infrastructure engineers who have learned that "it compiles" is only the first of many gates a kernel must pass.

The use of the word "wiring" is telling. It evokes an electrical metaphor: the kernel is a component that must be connected to the power supply (build system), the control panel (C-ABI), the quality assurance station (tests), and the instrument panel (benchmarks). Each connection is necessary for the component to function within the larger system.

Conclusion

Message <msg id=12228> is a small edit with large implications. It marks the transition from kernel design to kernel integration — the moment when a standalone CUDA source file begins its journey toward becoming a production component in a live inference service. The message's brevity is a sign of discipline: the assistant knows exactly what needs to be done and executes without hesitation. But the reasoning reveals the depth of understanding beneath that execution: an awareness of the full integration chain, the assumptions being made, and the steps that follow.

In the broader narrative of the session, this message is the pivot point. The preceding messages were about analysis and design — understanding the bottleneck, exploring architectural options, and writing the kernel. The subsequent messages will be about validation — compiling, testing, debugging, and benchmarking. Message <msg id=12228> is the bridge between those two phases, the moment when creation meets integration. It is, in its quiet way, as important as the kernel itself.