Chunk 71.1

This sub-session launched a comprehensive multi-agent investigation to root-cause the high-concurrency tool-call corruption (garbled DSML at C=60, clean at C=1). Initial research ruled out several high-profile serving-layer bugs (e.g., the detokenizer `batch_decode` issue #15042 was already fixed in the fork). The assistant built a multi-turn agentic repro harness that successfully reproduced the 18% corruption rate at 80 concurrent sessions. A controlled bisection campaign was initiated, testing the custom SM120 kernels and the bf16 index-K patch, which decisively isolated the bf16 index-K patch as the trigger. Running with fp8 keys eliminated the corruption, while bf16 keys consistently produced ~12-18% corruption and severe performance degradation (70 timeouts). Crucially, running the same bf16 code in a non-PD single-server configuration showed only ~2% corruption, localizing the bug to the PD transfer of the larger bf16 index-K buffer. The user firmly rejected an fp8 revert, directing the assistant to fix the performance/corruption issue while preserving the bf16 numerics. The decisive breakthrough came from testing the HiCache hypothesis: disabling HiCache entirely eliminated both the corruption (0% at 60 sessions) and the wedge (no transfer timeouts), while enabling it caused 12-18% corruption and stuck transfers. The root cause was identified as a classic race condition in the disaggregated prefill engine (sglang #22811). The main KV cache read path is properly gated by a `wait_layer_transfer` call, ensuring the async HiCache layer load completes before the data is read. However, the index-K buffer read path (`get_index_k_with_scale_buffer`) lacked this gate, allowing the sparse indexer to read stale or partially-loaded index data under concurrent load. The bf16 patch's 2x larger index-K buffer widened the race window, making the corruption reliably reproducible at high concurrency. The immediate mitigation deployed was to disable HiCache, restoring full stability and correctness. The proper fix involves adding the `wait_layer_transfer` synchronization gate to the index-K read path, which will allow HiCache and bf16 to coexist safely, preserving both the prefix-cache performance and the long-context recall benefits of the bf16 index-K patch.

The PD Transfer That Corrupted Everything: Tracing a bf16 Index-K Race Condition Under High Concurrency 1712 words

Message Articles