Chunk 69.0

This chunk pivoted from deployment optimization to **deep-dive debugging of a multi-turn context-loss failure**. The user reported that their agent harness (opencode) consistently loses context on long conversations — the model acts as if prior turns never happened (e.g., "to a file" after a tic-tac-toe request produced "this is the very first message"). The assistant initially blamed temperature/repetition, but the user pushed back, demanding a review of the deployment patches. **Systematic investigation** followed: the assistant pulled deployment logs, confirmed the actual prompts being sent (including that the harness uses `temperature=0.2`, not 0.6), examined the model's reference encoding to understand `drop_thinking` behavior (spec-compliant: prior reasoning is kept when tools are present), and — crucially — **audited every performance patch** applied to sglang. The key discovery is that the decode-only kernels (MMA split-K, Triton indexer) passed an 8k single-turn coherence test and don't touch prefill, so the long multi-turn failures point to **prefill-path numerical changes**: the **MHC bf16 GEMM** (unconditional, every layer, casts fp32 mixing weights to bf16) and the **MoE routed-scaling** implementation in `hash_topk.py`. The assistant produced a structured risk ranking, a diagnostic proxy script (`diag_proxy.sh`), and an isolation plan (A/B kernel toggling, golden reference comparison, context-fidelity needle tests). **Themes:** (1) **Performance vs correctness tension** — every speed optimization (bf16 tensor cores, split-K, Triton indexer) introduces numerical approximations that compound over long contexts; the MHC bf16 change is the most dangerous because it's unconditional and affects every layer. (2) **The importance of proper validation** — the earlier "coherency check" (gzip ratio) only caught repetition collapse, not context-fidelity; real multi-turn recall tests are needed. (3) **Deployment debugging rigor** — the assistant had to verify which MoE backend is actually active, which env flags are set, and which patches are inert to correctly rank risks. (4) **Spec vs implementation** — the `drop_thinking` behavior is correct per the model card, but the combination of long reasoning at `effort=max` + low temperature + prior reasoning in context creates a pathological feedback loop that degrades the model's ability to attend to history.

The Context Vanishes: A Debugging Odyssey Through Performance Patches, Spec Collisions, and Multi-Turn Amnesia 2735 words

Message Articles