The Persistent-Logits Fix: A Targeted Intervention in a CUDA-Graph Heisenbug

Introduction

In the long and arduous debugging journey documented across Segment 72 of the DeepSeek-V4-Flash deployment on Blackwell GPUs, message [msg 13451] stands as a quiet but pivotal moment. The message itself is deceptively simple — a single line from the assistant:

[edit] /tmp/opencode/dsv4_indexer.py Edit applied successfully.

This terse confirmation belies the intense reasoning, hypothesis-testing, and diagnostic detective work that led to this point. The edit represents the culmination of a multi-hour investigation into a persistent, high-concurrency tool-call corruption bug that had plagued the system for days. It was the assistant's attempt to apply a targeted, low-risk fix based on a carefully developed theory about the root cause: that the CUDA graph memory planner was aliasing transient intermediate tensors in the bf16 indexer path, causing silent data corruption under graph replay.

To understand why this edit matters, we must trace the reasoning that produced it.

The Debugging Context: A Heisenbug in the bf16 Indexer Path

The corruption bug had been a stubborn adversary. Under real agentic load — hundreds of requests with multi-round tool calls — the system would sporadically produce corrupted outputs, with corruption rates of 13–18% in baseline stress tests. The bug was specific to a narrow set of conditions: it required bf16 index keys (the SGLANG_DSV4_BF16_INDEX_K=1 path), CUDA graph capture enabled, and decode batch sizes greater than one. Eager mode was clean. The fp8 path was clean. The index-K buffer data itself was pristine. The corruption was transient — it appeared during graph replay but left no trace in the underlying data structures.

The assistant had deployed an ingenious diagnostic tool: the GE_DIFF (Graph vs. Eager Differential) instrumentation, which cloned the indexer's inputs before the captured graph ran, then recomputed the logits eagerly afterward and compared the two results. The differential was designed to catch the exact moment when captured and eager computations diverged, pinpointing the corrupted tensor.

But the differential did something unexpected: it suppressed the bug. With GE_DIFF enabled, corruption dropped from ~15% to ~3%, and across hundreds of comparison samples, the captured logits matched the eager recompute perfectly — zero divergence. The instrumentation itself was perturbing the system enough to make the bug disappear. This was the signature of a Heisenbug: a defect that vanishes when you try to observe it.

The assistant recognized this as a crucial clue. If adding clone and copy operations in the indexer region shifted the captured-graph memory layout and largely removed the corruption, then the defect was graph-memory-layout-sensitive — a captured-graph intermediate aliasing or race condition in the bf16 read path. The GE_DIFF clones changed how the graph planner allocated memory, which moved addresses around and broke whatever aliasing was causing the corruption.

The Hypothesis: Graph-Pool Aliasing of Transient Intermediates

With the Heisenbug behavior established, the assistant formulated a precise hypothesis. The bf16 reader function (bf16_paged_mqa_logits_triton_sm120) allocated a large intermediate tensor on every call:

logits = torch.empty(
    (batch_size, max_seq_len), dtype=torch.float32, device=q_fp8.device
)

This torch.empty call created a transient buffer inside the CUDA graph memory pool. During graph capture, when a tensor's Python reference count dropped to zero, its memory was returned to the pool and could be immediately reused for a subsequent allocation. If the graph planner reused that memory while a kernel still held a dangling reference to it, the result would be silent data corruption — exactly the pattern observed: pristine index-K data, correct kernel math, but corrupted outputs that only appeared under graph replay.

The fp8 path (using deep_gemm) didn't exhibit this behavior because its memory allocation patterns were different — likely using persistent workspaces or differently sized intermediates that didn't trigger the same aliasing.

The fix, the assistant reasoned, was straightforward: make the logits buffer a persistent cached buffer rather than a per-call transient allocation. By caching the buffer at the module level — keyed by batch size, max sequence length, device, and dtype — the tensor would be permanently live across all replays and graph buckets. The graph planner would see it as permanently referenced and would never reuse its memory for other allocations. This would break the aliasing at its source.

The Decision: Why This Approach Was Chosen

The assistant had several options on the table. Option (a) was the persistent-logits cache — low risk, directly testable, and surgically targeted at the hypothesized mechanism. Option (b) was running the entire indexer eagerly outside the captured graph — more robust but riskier, requiring a "breakable backend" wrapper that could destabilize the decode path. Option (c) was running the entire decode eagerly — proven clean (0% corruption) but with unknown performance cost.

The assistant chose option (a) for several reasons. First, it was the most direct test of the hypothesis: if making logits persistent fixed the corruption, it confirmed that the logits buffer was the aliasing victim. Second, it was low-risk: the kernel fully overwrites the logits buffer on every call (writing either actual logits or -inf for masked positions), so reusing a cached buffer was safe — no stale data leakage. Third, within a captured graph, the indexer layers call the reader sequentially, so each layer's logits are consumed by its topk before the next layer runs, meaning a single cached buffer could be safely reused across layers.

The assistant also recognized a meta-insight: the codebase already cached arange buffers "to avoid device-tensor creation during CUDA graph capture" elsewhere in the same file. This confirmed that the bug class — transient allocations causing graph-pool aliasing — was known to the original developers. The persistent-logits fix was applying the same pattern to a tensor that had been missed.

The Edit: What Changed

The edit to /tmp/opencode/dsv4_indexer.py inserted a caching helper function before the bf16_paged_mqa_logits_triton_sm120 function definition and replaced the torch.empty call with a cache lookup. The cache was a dictionary keyed by device and shape tuple, returning a pre-allocated buffer if one existed or creating and storing a new one. The buffer was allocated eagerly — during the warmup calls before graph capture — so it existed in the default memory pool, not as a graph-pool transient.

A companion edit (message [msg 13452]) applied the same fix to the torch-based reader fallback path for completeness.

The Result: Partial Success, Deeper Insight

When the assistant tested the fix (message [msg 13453]), the results were revealing but not definitive. With GE_DIFF disabled (testing against the true 15% baseline), the persistent-logits fix produced corruption rates of 5%, 5%, and 12% across three runs — an average of ~7%, roughly half the baseline. The fix had reduced corruption but not eliminated it.

This was a crucial finding. The logits buffer was part of the aliasing surface — caching it shifted the memory layout enough to reduce corruption — but it was not the sole victim. Other transient allocations in the indexer/topk path remained uncached, and the graph planner was still aliasing something else. The Heisenbug had evaded a clean kill.

But the experiment was far from wasted. It confirmed the mechanism: the corruption was indeed graph-memory-layout-sensitive, and the logits buffer was one of the tensors involved. The partial suppression also ruled out simpler explanations — if the bug had been a kernel logic error or a data corruption in the index-K buffer, caching the logits would have had no effect at all. The fact that it halved the corruption rate was strong evidence for the aliasing hypothesis.

Assumptions and Their Validity

The assistant made several assumptions in choosing this approach. The first was that the CUDA graph memory planner reuses freed blocks during capture in a way that can cause aliasing. This assumption was validated by the partial fix — if the planner were fully capture-safe, caching a single tensor would have had no effect. The second assumption was that the logits buffer was the primary aliasing victim. This turned out to be partially correct — it was a victim but not the only one. The third assumption was that the kernel's full-overwrite semantics made buffer reuse safe. This was correct — no new corruption modes were introduced by the cache.

The key mistake was underestimating the number of transient allocations in the indexer path. The assistant later discovered that the topk_transform dispatch and other internal operations also produced uncached intermediates that could alias. The persistent-logits fix was necessary but not sufficient — it addressed one aliasing surface while others remained.

Knowledge Created

This message produced several important pieces of knowledge. First, it confirmed that the corruption mechanism was graph-memory-layout-sensitive — not a kernel logic error, not a data corruption in the KV cache, but an aliasing issue in the captured-graph memory pool. Second, it established that the logits buffer was one of the tensors involved in the aliasing, providing a concrete target for further analysis. Third, it demonstrated the effectiveness of persistent caching as a diagnostic technique — by observing how much corruption was suppressed, the assistant could estimate how many aliasing surfaces remained.

The partial fix also ruled out the "single point of failure" hypothesis. If the corruption had been caused by a single aliasing pair (e.g., logits always colliding with a specific other tensor), caching logits would have eliminated it entirely. The residual 7% corruption meant multiple aliasing relationships were at play, each with its own probability of collision.

Conclusion

Message [msg 13451] captures a moment of focused engineering judgment. Faced with a Heisenbug that had resisted days of investigation, the assistant chose a low-risk, targeted intervention based on a carefully developed theory about the root cause. The edit was surgically precise — changing a single allocation pattern in a single function — and the test results were informative even in partial failure. The persistent-logits cache halved the corruption rate, confirmed the aliasing mechanism, and narrowed the remaining search space. It was a step forward, not a dead end, and it paved the way for the definitive fix that would come later: disabling SGLANG_OPT_USE_MULTI_STREAM_OVERLAP, which eliminated the corruption entirely by removing the race condition at its architectural root.

In the broader narrative of Segment 72, this message represents the transition from characterization to intervention — the moment when the assistant stopped observing the bug and started fixing it. The fix wasn't complete, but it was correct, and the knowledge it produced was invaluable.