The Architecture of Capture-Safety: Refactoring a CUDA Kernel for Graph Replay

Introduction

In the high-stakes world of large language model inference, every microsecond counts. When serving models like Kimi K2.6 with speculative decoding, the difference between a responsive system and a sluggish one often comes down to how efficiently the GPU executes attention computations. This article examines a single message from an opencode coding session—message index 12303—in which an AI assistant undertakes a critical refactoring: making a custom CUDA verify-attention kernel "capture-safe" for CUDA graph replay. The message is a window into the intricate reasoning, design trade-offs, and technical constraints that define modern GPU kernel engineering for LLM inference.

The message in question occurs at a pivotal moment in the session. The assistant has already built a custom sm_120 verify-attention kernel that achieves a 2× speedup over the Triton baseline for long-context decoding. However, this kernel runs in "eager mode"—meaning each invocation executes immediately on the GPU stream—which forces the entire serving system to disable CUDA graph optimization globally. The user's directive is clear: "Fix cudagraphs, then do defrag and optimise marshaling." The assistant must now refactor its kernel to be compatible with CUDA graph capture, a constraint that imposes severe restrictions on what operations are permissible during execution.

What follows is a masterclass in systems-level reasoning. The assistant walks through workspace allocation strategies, grid sizing trade-offs, memory constraints, pointer stability requirements, and the subtle semantics of CUDA graph capture—all while maintaining the correctness and performance of the underlying attention computation. This article dissects that reasoning in detail.

The CUDA Graph Capture Problem

To understand the message, one must first understand CUDA graph capture. CUDA graphs allow a sequence of GPU operations (kernel launches, memcpy operations, etc.) to be captured, serialized into a graph object, and then replayed with new data. The critical constraint is that during capture, every operation must be "capturable"—it must use only resources that persist across replays, and it must not introduce host-device synchronization points.

The assistant's original verify kernel violated these constraints in several ways:

  1. Host synchronizations: The Python code called .item() on tensors, which forces a device-to-host memory transfer and stalls the GPU pipeline until the value is available.
  2. Metadata buffer copies: The Python code rebuilt combined indices and visibility masks using .cat(), .contiguous(), and .to() operations, creating new tensors with different memory addresses. During graph replay, the captured kernel would read from the original (stale) addresses.
  3. Raw cudaMalloc calls: The workspace allocation bypassed PyTorch's caching allocator, which manages memory in graph-compatible pools. The assistant's reasoning begins by identifying these issues and formulating a capture-safe design:
"The core requirement: during capture I must do no host syncs, no raw cudaMalloc, and pass SGLang's static metadata buffers directly (so replay reads the updated buffers) — no .to()/.contiguous()/.cat() copies."

This is the foundational insight. The kernel must consume SGLang's pre-allocated static buffers directly, computing indices and masks in-kernel rather than in Python. The assistant recognizes that SGLang's framework already maintains these static buffers for CUDA graph support—the challenge is adapting the custom kernel to use them natively.

The Workspace Allocation Dilemma

The most extensively reasoned portion of the message concerns workspace allocation. The verify kernel requires a temporary buffer to store partial attention results across the split-K decomposition. The question is: how should this workspace be allocated to be compatible with CUDA graph capture?

The assistant cycles through several strategies, each with different trade-offs:

Strategy 1: Per-call torch.empty

"Let me use torch.empty per call. But that's an allocation each step in eager too (minor overhead). Fine."

This is the simplest approach: allocate a fresh workspace tensor on every invocation. The assistant initially favors this because torch.empty uses PyTorch's caching allocator, which is compatible with CUDA graph capture. During capture, the allocation goes into the graph's private memory pool and persists across replays. The overhead of per-call allocation in eager mode is deemed acceptable.

Strategy 2: Persistent module-level workspace

"Actually, reconsider: a persistent module-level workspace avoids per-call alloc and is simplest for capture (allocate once, reuse pointer). But sizing for max bs..."

The assistant then considers a persistent workspace tensor allocated once and reused across all invocations. This avoids the per-call allocation overhead entirely and provides a fixed pointer that's trivially capture-safe. The challenge is sizing: the workspace must be large enough for the maximum batch size the server might encounter, which is unknown at startup.

Strategy 3: Persistent workspace with lazy growth

The assistant eventually settles on a hybrid approach: a module-level tensor that grows as needed during eager warmup before capture begins. By the time graphs are captured, the workspace is already sized for the maximum batch size encountered during warmup. This combines the capture-safety of a fixed pointer with the flexibility of adaptive sizing.

This back-and-forth reasoning reveals a key insight about the assistant's decision-making process: it's not just choosing a strategy, but evaluating each against multiple criteria (capture compatibility, memory efficiency, simplicity, overhead) and iterating toward a solution that balances all constraints. The final choice—persistent workspace with lazy growth—is the most sophisticated option, requiring the most implementation effort but offering the best trade-offs.

The NSPLIT Parameter: A Study in Optimization Trade-offs

The NSPLIT parameter controls how many parallel splits the verify kernel uses for the flash-attention-style reduction. A higher NSPLIT means more parallelism (better occupancy) but also larger workspace memory and more empty blocks at short context lengths. A lower NSPLIT saves memory but may underutilize the GPU's SMs.

The assistant's reasoning about NSPLIT is a microcosm of the entire message's approach to optimization:

"Let me set NSPLIT = 64 (module constant matching the kernel's expectation; the kernel takes NSPLIT as param). Actually the kernel takes NSPLIT as a param and uses it for grid + chunk. I pass NSPLIT=64 always (capture-safe, fixed)."

The initial choice of 64 is driven by the desire to cover long context lengths (up to 200k tokens) with reasonable chunk sizes. With NSPLIT=64, each split handles approximately 3,125 tokens at 200k context—a comfortable workload.

But then the assistant reconsiders:

"Wait, but is NSPLIT=64 optimal? For short context it's wasteful (empty blocks). For capture-safety it must be fixed."

This is a critical realization. With NSPLIT=64 fixed, a short-context request (say, 100 tokens) would still launch 64 grid blocks, most of which would immediately early-out because their split range is empty. The overhead of launching these empty blocks is small but non-zero, and at short context lengths where the kernel is already slower than Triton, every microsecond matters.

The assistant then explores reducing NSPLIT to 16:

"To keep memory usage reasonable, I'm considering reducing NSPLIT from 64 to 16, which cuts workspace size by 4x (from ~2.4GB down to ~600MB for bs=32) while still maintaining good occupancy even at long context lengths—16 splits across 1152 blocks gives enough parallelism across the SMs."

The memory arithmetic is revealing. With NSPLIT=64 and batch size 32, the workspace would require approximately 2.4 GB per GPU—a substantial fraction of the available memory. Reducing to NSPLIT=16 brings this down to 600 MB, a much more manageable footprint.

But the assistant worries about performance:

"The tradeoff is that the Phase 1 microbench used NSPLIT=32 for 65k tokens and got good performance, so halving to 16 splits might reduce occupancy slightly at long context, though it's still workable."

The assistant is balancing three competing constraints: capture-safety (requires fixed NSPLIT), memory capacity (lower NSPLIT saves memory), and performance (higher NSPLIT improves occupancy). The final choice of NSPLIT=16 represents a conservative compromise that prioritizes memory safety over peak performance, with the understanding that the kernel can be further tuned later.

This kind of multi-constraint optimization is characteristic of systems engineering. There is no single "correct" answer—only a solution that adequately satisfies all constraints within acceptable margins.

The Capture-Safe Kernel Interface

The assistant's reasoning also details the technical architecture of the capture-safe kernel interface. The key design principle is that the kernel must consume SGLang's native metadata buffers directly, without any Python-side reshaping or copying:

"The metadata buffers: I need to pass them directly via data_ptr() without any conversions or reshaping. The kernel expects specific dtypes — kv_indptr as int32, qo_indptr and mask_indptr as int64, kv_indices and out_cache_loc as int64, custom_mask as uint8."

This requires the kernel to handle index computation internally. Instead of Python building a combined list of KV positions (prefix indices + draft token cache locations), the kernel reads from two separate arrays—kv_indices for the prefix and out_cache_loc for the draft tokens—and selects between them based on the position relative to the prefix length.

Similarly, the visibility mask is no longer pre-gathered into a dense buffer. Instead, the kernel receives the raw custom_mask array and mask_indptr offsets, computing the mask index inline during execution. This eliminates two Python-level tensor operations that would break capture.

The assistant also considers the query tensor:

"For q, I'm reshaping it from [num_tok, H*qk] to [num_tok, H, qk] with view, which is free if q is already contiguous. Since q comes in as bfloat16 already, I should skip any .contiguous().to() calls that would create a copy and break capture pointer stability."

This attention to detail—ensuring that even a .contiguous() call is avoided—demonstrates a deep understanding of CUDA graph capture semantics. Any operation that creates a new tensor with a different memory address would cause the captured graph to read from stale pointers during replay.

Assumptions and Their Validity

The assistant makes several assumptions in its reasoning, most of which are well-justified but worth examining:

Assumption 1: torch.empty is capturable. The assistant assumes that allocating a tensor with torch.empty during CUDA graph capture is safe because PyTorch's caching allocator manages graph-compatible memory pools. This is correct for standard CUDA graph capture workflows, though the assistant acknowledges edge cases where the allocator might behave unexpectedly.

Assumption 2: Python attribute access is safe during capture. The assistant notes that accessing forward_batch.batch_size, q.shape[0], and layer dimensions as Python ints does not trigger device synchronization. This is correct—these are metadata values already available on the host.

Assumption 3: The kernel launch via ctypes happens on the capture stream. The assistant must ensure that the ctypes-based kernel launch uses the correct CUDA stream during capture. This is a subtle requirement: if the kernel launches on the wrong stream, it won't be captured in the graph.

Assumption 4: SGLang's static buffers remain valid across replays. The assistant assumes that SGLang's framework copies new data into the same static buffers before each graph replay, so the captured pointers remain valid. This is a fundamental property of SGLang's CUDA graph implementation and is well-documented.

Assumption 5: Memory pressure can be managed by adjusting mem-fraction-static. The assistant considers lowering the memory fraction from 0.94 to 0.88–0.90 to accommodate the workspace and graph overhead. This assumes that the KV cache pool can be slightly reduced without impacting serving quality, which is reasonable for a non-live-testing environment.

One potential mistake is the assistant's initial assumption that per-call torch.empty would be the simplest approach, only to later reconsider and adopt a persistent workspace. This isn't a bug—both approaches are valid—but the iteration reveals that the assistant is reasoning through the problem in real time, refining its understanding as it considers deeper implications.

Input Knowledge Required

To fully understand this message, a reader needs substantial background knowledge:

  1. CUDA graph capture semantics: Understanding what operations are capturable, how memory pools work during capture, and how replay differs from eager execution.
  2. Flash attention and split-K decomposition: The verify kernel uses a flash-attention-style algorithm where the KV sequence is split into chunks, each chunk computes partial attention, and a reduction step combines the partial results. The NSPLIT parameter controls the number of chunks.
  3. SGLang's serving architecture: Knowledge of how SGLang manages KV caches, static metadata buffers, CUDA graph capture for the forward pass, and the forward_extend hook where the verify kernel is injected.
  4. PyTorch's CUDA caching allocator: Understanding that torch.empty allocates from a managed pool that is compatible with CUDA graph capture, unlike raw cudaMalloc.
  5. Speculative decoding with DDTree: The verify kernel is part of a speculative decoding system where a draft model proposes candidate tokens and the target model verifies them in parallel. The "verify attention" computes attention for all draft tokens simultaneously.
  6. Tensor parallelism (TP): The system uses TP=8, meaning each GPU handles 8 attention heads. This affects occupancy calculations and kernel design.
  7. Memory bandwidth and occupancy: The reasoning about NSPLIT and workspace size involves understanding GPU memory hierarchy, SM occupancy, and the relationship between parallel work and memory bandwidth utilization.

Output Knowledge Created

The message produces several concrete outputs:

  1. A capture-safe kernel interface specification: The kernel now accepts SGLang's native buffers directly, with in-kernel index computation and mask lookup.
  2. A workspace allocation strategy: A persistent module-level tensor that grows during eager warmup and is reused across all captured graphs.
  3. A fixed NSPLIT parameter: Set to 16 (or configurable via environment variable), balancing memory usage and occupancy.
  4. A refactored Python backend: The kdtree_mla_backend.py file is rewritten to eliminate all capture-breaking operations, using data_ptr() for native buffer access and torch.empty for workspace allocation.
  5. A validation path: The patched_forward_extend function maintains a validation mode that can use host syncs for correctness checking, separate from the capture-safe _run_kernel path.
  6. Documentation of constraints: The reasoning captures the design constraints and trade-offs, serving as documentation for future maintainers.

The Thinking Process: A Window into Systems Engineering

What makes this message exceptional is the visible reasoning process. The assistant doesn't just implement a solution—it explores alternatives, evaluates trade-offs, and iterates toward a design that satisfies multiple constraints.

The reasoning exhibits several characteristic patterns of expert systems engineering:

Constraint-driven design: The assistant starts by enumerating the constraints (no host syncs, no raw cudaMalloc, no tensor copies) and then designs the solution to satisfy them. This is the opposite of feature-driven design, where one implements a feature and then checks if it meets constraints.

Multi-level optimization: The assistant considers trade-offs at multiple levels simultaneously—memory footprint, kernel occupancy, capture compatibility, implementation complexity—and finds a balance that works across all dimensions.

Progressive refinement: The workspace allocation strategy evolves from per-call allocation to persistent allocation to lazy-growth persistent allocation, each iteration addressing a concern identified in the previous approach.

Empirical grounding: The assistant references actual measurements from the Phase 1 microbench (NSPLIT=32 at 65k tokens) to ground its performance estimates, rather than relying on abstract reasoning alone.

Conservative bias: When uncertain about memory pressure, the assistant chooses a more conservative NSPLIT value (16) rather than pushing for maximum performance. This is appropriate for a production system where stability matters more than peak throughput.

Awareness of future work: The assistant notes that marshaling optimization (Phase 2e) can address per-step costs, acknowledging that the current design may not be optimal but is sufficient for the immediate goal of capture-safety.

Conclusion

Message 12303 captures a moment of deep technical reasoning in the development of a custom CUDA kernel for LLM inference. The assistant's task—making a verify-attention kernel compatible with CUDA graph capture—requires navigating a complex landscape of constraints, trade-offs, and subtle GPU programming semantics.

The message is valuable not just for the specific solution it produces, but for the reasoning process it reveals. The assistant demonstrates how to approach a constrained optimization problem in systems engineering: enumerate constraints, explore alternatives, evaluate trade-offs against multiple criteria, and iterate toward a solution that balances all requirements.

For anyone working on GPU kernel development, LLM serving infrastructure, or performance-critical systems programming, this message offers a rare glimpse into the decision-making process of an experienced practitioner. The capture-safe kernel it produces is a concrete artifact, but the thinking that produced it—the workspace allocation dilemma, the NSPLIT optimization, the pointer stability analysis—is the real contribution.

The session continues beyond this message, with the capture-safe kernel achieving a 3–6× decode speedup over Triton and the bottleneck shifting to MoE expert imbalance. But message 12303 is where the critical architectural decision is made, and where the assistant's reasoning shines brightest.