The Query-Split Kernel: A Pivotal Moment in Building a Native DDTree Inference Engine

Message Summary

[assistant] [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/kdtree-engine/src/engine/ops.cu Edit applied successfully.

On its surface, this message is unremarkable — a terse confirmation that a file edit succeeded. The assistant's tool call to modify ops.cu returned without error, and the system logged the result. But this single line represents the culmination of an extraordinarily dense chain of reasoning spanning dozens of prior messages, and it marks a critical architectural decision in the construction of a native C/C++/CUDA inference engine for the Kimi K2.6 model with speculative decoding via Draft-and-Diffusion Tree (DDTree).

Context: Building an Inference Engine from Scratch

To understand why this message matters, one must appreciate the broader context. The assistant was in the midst of building a complete native inference engine for the Kimi K2.6 model — a massive Mixture-of-Experts (MoE) transformer with Multi-head Latent Attention (MLA) — targeting 8× RTX PRO 6000 Blackwell GPUs. This was not a thin wrapper around existing libraries; it was a ground-up implementation in C++ with custom CUDA kernels, designed to support DDTree speculative decoding, a technique that uses a small "drafter" model to propose candidate tokens in a tree structure, which the large "target" model then verifies in a single batched forward pass.

The engine was organized into a new kdtree-engine/ repository. Phase 0 had established the build infrastructure (CMake + CUDA 13 targeting sm_120 for Blackwell), a binary container format (KDTR) for sharing test data between Python and C++, and faithful NumPy reference implementations of the DDTree algorithms. Phase 1 delivered three validated custom CUDA kernels: a GPU best-first tree builder, a tree-verify MLA-absorb attention kernel with visibility masking, and a greedy tree-accept kernel. All 27 kernel tests passed bit-exact against the references.

Phase 2 — the phase in which this message appears — was the construction of the MVP native engine itself: a full DeepSeekV3/Kimi-style MLA+MoE transformer in FP32, with cuBLAS GEMMs serving as placeholders for the eventual INT4 Marlin path. The engine needed RMSNorm, NeoX-style Rotary Position Embedding (RoPE), SwiGLU activations, MoE routing with a shared expert, KV cache with post-verify compaction, and the complete DDTree speculative decode loop wiring all three custom kernels together.

The Decision Behind the Edit: Why a Query-Split Kernel?

The edit to ops.cu was not arbitrary. It was driven by a specific architectural decision that the assistant had been wrestling with in the preceding messages, particularly in [msg 11926]. The core issue was how to handle the query projections in the MLA attention mechanism.

In the MLA absorb formulation used by DeepSeekV3 and Kimi K2.6, the query projection produces a combined tensor that contains both "nope" (no position encoding) and "rope" (rotary position encoding) components concatenated together. The nope component is used directly in the attention score computation, while the rope component must have rotary position encoding applied before it can be used. These two components need to be separated before the attention computation can proceed.

The assistant considered two approaches:

  1. Extract into separate contiguous buffers: Split the combined query tensor into two separate tensors — one for the nope portion and one for the rope portion — using a dedicated kernel. This makes the subsequent operations (attention score computation for nope, RoPE application for rope) straightforward since each operates on contiguous memory.
  2. Read with strides from the combined buffer: Keep the combined tensor intact and use strided access patterns to read the nope and rope components as needed. This avoids the memory overhead and kernel launch of a split, but introduces complexity in the attention kernel and RoPE kernel, which would need to handle non-contiguous memory layouts. The assistant chose option 1, reasoning: "I'll go with extracting into contiguous buffers since it's clearer." This decision prioritized code clarity and correctness over the marginal performance gain of avoiding a copy. In a hyper-optimized inference stack, every kernel launch and memory copy matters, but at the MVP stage, getting the math right and being able to debug it easily is paramount. The query-split kernel was thus added to ops.cu — the file that houses the engine's CUDA kernel implementations — via this edit.

Input Knowledge Required

To understand the significance of this edit, one needs substantial domain knowledge spanning multiple levels:

The Thinking Process Revealed

The assistant's reasoning in [msg 11926] reveals a meticulous, almost obsessive, approach to correctness. The thinking traces through multiple levels of abstraction:

At the highest level, the assistant is designing the model API — deciding what buffers are needed, how the forward pass should be structured, and how the KV cache should be managed. It considers the three scenarios the forward pass must handle: prefill (processing L tokens at once), autoregressive decode (one token at a time), and tree verification (multiple query tokens with masking).

At the intermediate level, the assistant works through the exact indexing for tree verification — a notoriously tricky problem involving off-by-one errors in cache positions, root node placement, and visibility masking. The reasoning shows the assistant tracing through SGLang's approach, identifying inconsistencies, and ultimately designing a cleaner scheme: "Rather than trying to exactly match SGLang's slot bookkeeping, let me define a clean, self-consistent scheme for my MVP."

At the lowest level, the assistant is making concrete implementation decisions: whether to extract query components into separate buffers or use strided access, how to structure the MoE routing (compute router logits via GEMM, apply top-k and softmax on the host, then run each expert's SwiGLU transformation), and what precision to use for numerical operations.

Assumptions and Potential Pitfalls

The query-split approach makes several implicit assumptions:

  1. The memory overhead is negligible: The split kernel allocates separate buffers for the nope and rope components. For the tiny test model (hidden=256, 4 heads), this is trivially small. For the full K2.6 model with 64+ heads and a hidden dimension of 7168, the overhead is still modest — roughly n_heads × qk_nope and n_heads × qk_rope floats per query token — but it's non-zero.
  2. The kernel launch overhead is acceptable: Adding a kernel launch to the forward pass increases latency. For the MVP, this is fine; for the hyper-optimized production stack, the assistant might revisit this decision and fuse the split into the attention kernel.
  3. Contiguous buffers are always preferable: This is generally true for GPU kernels, but there are edge cases where strided access with appropriate alignment can be equally efficient. The assistant's choice prioritizes clarity, which is the correct call for an MVP.
  4. The query-split is a separable operation: The assumption is that splitting the query into nope and rope components is a pure data rearrangement that doesn't depend on any other computation. This is correct for the MLA absorb formulation, where the query projection produces the combined tensor and the split is purely structural.

Output Knowledge Created

This edit produced a concrete artifact: a query-split CUDA kernel in ops.cu that takes the combined query tensor from the MLA projection and separates it into contiguous nope and rope buffers. This kernel becomes a building block for the full forward pass, which the assistant would go on to implement in model.cu.

More broadly, the edit represents a commitment to a specific architectural approach. The decision to use contiguous buffers with a dedicated split kernel shapes the design of every downstream component: the attention kernel receives cleanly separated inputs, the RoPE kernel operates on a contiguous rope buffer, and the overall forward pass has clear data flow boundaries. This is the kind of structural decision that, once made, propagates through the entire codebase.

The Broader Narrative

This message sits at a fascinating inflection point in the session. The assistant had just completed the most intense reasoning phase of the engine build — working through the exact semantics of tree verification indexing, the KV cache management scheme, and the forward pass structure. The edit to ops.cu was the first concrete implementation step following that reasoning. It was the moment when the abstract design crystallized into code.

The subsequent messages would build on this foundation: the model header and implementation files, the weight loading system, the forward pass wiring, and ultimately the complete DDTree decode loop. But this edit — adding the query-split kernel to ops.cu — was where the architecture first touched silicon. It was the point at which the assistant stopped reasoning about abstractions and started writing the code that would run on the Blackwell GPUs.

In a session spanning hundreds of messages, thousands of lines of code, and dozens of architectural decisions, this single edit confirmation is a quiet landmark. It represents the transition from design to implementation, from reasoning to building, from "what should we do?" to "here is the code that does it."