Building the Bisection Harness: A Pivot from Static Analysis to Empirical Evidence

Message Overview

The subject message (index 13181) is a brief but pivotal assistant response in a long-running debugging session. It consists of a single tool call — writing a Python file to /tmp/opencode/repro_toolcall.py — accompanied by a short reasoning block explaining the intent. On the surface, it appears unremarkable: the assistant commits code, then writes a test harness. But this message represents a critical methodological turning point in a complex production debugging effort, where the assistant shifts from static analysis and hypothesis generation to systematic empirical investigation.

The full message as written:

## Agent Reasoning The git commit is done and the working tree is clean. Now I'm building a deterministic load tester that will hammer the server with parallel requests containing tools and lengthy context to expose any corruption issues—things like DSML appearing in the response, unexpected finish reasons, or garbled output. I'm modeling it after the session-bible setup with around ten tools and a long message history, specifically designed to trigger tool calls reliably. Writing the harness script... I'm refining the corruption detection logic to catch three cases: DSML leakage in the content, missing tool calls when tools are provided, and successful tool invocations. For the long context needed to trigger sparsity, I'll use a configurable parameter to control filler text volume—targeting around 8-10k tokens by repeating sentences. To make it more realistic and agentic, I should consider multi-turn conversations, but I'll start with a single long user message with tools defined and iterate if needed. I'm building a harness script with options for concurrent requests, context size, streaming vs non-streaming, and will deploy it to test with increasing concurrency levels.

Committed 7c6254e77, clean tree. Now I'll build a deterministic repro harness (mirrors the agent: ~10 tools, long context, asks for many tool calls) with corruption detection, so every bisection step has hard evidence.

[write] /tmp/opencode/repro_toolcall.py Wrote file successfully.

The Context: A Stubborn Production Bug

To understand why this message was written, one must appreciate the debugging landscape that preceded it. The team had been battling a high-concurrency output corruption bug for multiple segments. The symptom was unmistakable: under heavy parallel load (approximately 60 concurrent sessions), the DeepSeek-V4 model would produce garbled output — DSML (DeepSeek Markup Language) tags leaking into assistant responses, tool calls rendered as raw text instead of structured JSON, and general "token salad" that rendered the system unusable for agentic workloads. Yet at single-session concurrency (C=1), the system worked flawlessly.

This pattern — clean at low concurrency, corrupted at high concurrency — is the classic signature of a race condition, a buffer overrun, or a synchronization bug that only manifests when multiple requests compete for shared resources. The assistant had spent considerable effort narrowing the field. Several high-profile suspects had been ruled out: the detokenizer batch_decode bug (already fixed in the fork), the overlap scheduler (disabled with --disable-overlap-schedule), MTP/EAGLE speculative decoding (not configured), pipeline parallelism (not used), and radix cache corruption (disabled). What remained was a shortlist of custom code paths: the bf16 index-K patch, the custom SM120 Triton indexer, the SM120 MMA FlashMLA decode kernel, and the NVFP4 GEMM/MoE path.

The user had been emphatic that the same model worked flawlessly from cloud providers at high parallelism, which meant the bug was in the deployment-specific stack — the custom kernels, the disaggregated prefill (PD) setup, and the various patches the assistant had applied over the preceding segments. This constraint ruled out blaming upstream model behavior and forced the investigation to focus on the team's own code.

Why This Message Was Written: The Need for Deterministic Evidence

The assistant's reasoning reveals a clear motivation: after committing the current state of the bf16 index-K patch as a checkpoint, the next step was to build a tool that could produce hard, repeatable evidence of the corruption. Up to this point, the investigation had relied on observed behavior from production runs — the user reporting that "C=60 corrupts, C=1 doesn't." But production runs are messy: they vary in context length, tool schema, timing, and request ordering. A deterministic repro harness would allow the assistant to control every variable: the exact conversation history, the exact tool definitions, the exact concurrency level, the exact random seed.

The reasoning block shows the assistant thinking through the design of this harness in real time. It identifies three corruption signatures to detect:

DSML leakage — markup language appearing in the response content
Missing tool calls — the model failing to invoke tools when provided
Successful tool invocations — the positive case, confirming the system works This tripartite detection scheme is clever: it doesn't just look for corruption; it also confirms when the system is working correctly, providing a clear binary signal for each test run. The assistant also considers the need for long context (8-10k tokens) to trigger the sparse attention path, since the DSA indexer only activates beyond ~2048 tokens — a critical detail that earlier analysis had identified as a potential confound in the "C=1 always clean" observation.

Decisions Made in This Message

Several implicit and explicit decisions are embedded in this message:

Decision 1: Build a custom harness rather than use an existing one. The assistant could have used a generic HTTP load testing tool like oha, wrk, or a Python async stress tester. Instead, it chose to write a purpose-built script that mirrors the agentic tool-calling pattern of the production workload. This decision prioritizes ecological validity — the harness must trigger the exact same code paths as the real bug — over convenience.

Decision 2: Start with single-turn, iterate to multi-turn. The reasoning explicitly notes: "I should consider multi-turn conversations, but I'll start with a single long user message with tools defined and iterate if needed." This is a pragmatic incrementalism: build the simplest version first, confirm it can trigger the corruption, then add complexity. It reflects an awareness that multi-turn conversations introduce additional variables (KV cache management, context accumulation) that could confound the bisection.

Decision 3: Use configurable parameters for concurrency, context size, and streaming. The harness is designed as a parameter sweep tool, not a one-shot test. This allows the assistant to run the same test at C=1, C=10, C=30, C=60 and observe where the corruption onset occurs — a crucial capability for bisecting the root cause.

Decision 4: Commit the bf16 index-K patch as a checkpoint before starting bisection. This ensures the working tree is clean and the bisection experiments can be cleanly toggled with environment variables rather than git operations. It also preserves the current deployed state as a recoverable baseline.

Assumptions Made

The message reveals several assumptions, some explicit and some implicit:

Assumption 1: The corruption is deterministic under controlled conditions. The entire bisection strategy depends on the bug being reliably triggerable. If the corruption is truly a Heisenbug — sensitive to timing, memory layout, or random scheduling — then even a perfect harness might produce inconsistent results. The assistant implicitly assumes that by controlling concurrency, context length, and seed, the corruption rate will be stable enough to measure.

Assumption 2: The harness can reproduce the production corruption pattern. The assistant models the harness after "session-bible," a known working configuration. But if the production workload has some subtle property not captured (specific tool schemas, particular message ordering, timing between requests), the harness might fail to trigger the bug, leading to false negatives.

Assumption 3: Context length is the key variable for sparsity onset. The assistant targets 8-10k tokens to exceed the ~2048 token sparsity threshold. This assumes the DSA sparse indexer is indeed the relevant path. If the corruption occurs through a different mechanism that doesn't depend on sparsity, this assumption is harmless but the long context adds unnecessary complexity.

Assumption 4: The corruption manifests within a single request-response cycle. The harness sends a request and checks the response for corruption. But if the bug requires multiple turns to accumulate state corruption — as the user later reported with the "losing the plot" phenomenon — a single-turn harness might miss it entirely. The assistant acknowledges this by planning to add multi-turn support if needed.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

The DSML format — DeepSeek's custom markup language for tool calling, which the model emits as structured tags. Leakage of these tags into response text is the primary corruption signal.
The DSA sparse attention mechanism — DeepSeek's attention architecture uses a sparse indexer that only activates beyond a certain context length (~2048 tokens). This is why context length matters: short contexts never exercise the sparse path.
The bf16 index-K patch — A modification to store attention index keys in bf16 precision instead of fp8, matching the DeepSeek reference implementation. This patch is the leading suspect for the corruption and is toggled via SGLANG_DSV4_BF16_INDEX_K.
The PD (prefill-decode) disaggregation architecture — The system separates prefill and decode across different GPU groups. The corruption might be related to KV cache transfer between these groups.
The previous debugging history — The detokenizer bug exclusion, the overlap scheduler fix, the HiCache race condition discovery, and the wedge fix all inform why the assistant is now focused on building a harness rather than continuing static analysis.

Output Knowledge Created

This message produces:

A deterministic repro harness (/tmp/opencode/repro_toolcall.py) — a Python script that can drive parallel tool-calling requests against the SGLang server with configurable concurrency, context length, and streaming mode. This becomes the primary instrument for the bisection campaign.
A clean git checkpoint (commit 7c6254e77) — preserving the current state of the bf16 index-K patch so that bisection experiments can be cleanly isolated.
A methodological precedent — the decision to use empirical, reproducible testing rather than static analysis or production observation sets the tone for the remainder of the debugging effort. Subsequent messages will run this harness at various concurrency levels and environment flag configurations.

The Thinking Process: From Abstract to Concrete

The reasoning block in this message is particularly revealing of the assistant's cognitive process. It begins with a status update ("The git commit is done and the working tree is clean") — a checkpoint that provides psychological closure on the previous phase. Then it transitions to the future: "Now I'm building a deterministic load tester."

The thinking oscillates between concrete implementation details ("around ten tools and a long message history") and abstract design principles ("specifically designed to trigger tool calls reliably"). The assistant is simultaneously a programmer thinking about code structure and a detective thinking about evidence quality.

There's a notable refinement of the corruption detection scheme in real time: "I'm refining the corruption detection logic to catch three cases: DSML leakage in the content, missing tool calls when tools are provided, and successful tool invocations." This tripartite scheme — two negative signals and one positive signal — shows the assistant thinking about both sensitivity (catching all forms of corruption) and specificity (confirming when the system works).

The assistant also demonstrates awareness of the sparsity confound: "For the long context needed to trigger sparsity, I'll use a configurable parameter to control filler text volume—targeting around 8-10k tokens by repeating sentences." This directly addresses the earlier insight that "C=1 always clean" might be an artifact of short context rather than low concurrency.

The final sentence of the reasoning — "I'm building a harness script with options for concurrent requests, context size, streaming vs non-streaming, and will deploy it to test with increasing concurrency levels" — reveals the experimental design: a parameter sweep across concurrency, not just a single test. This is the thinking of a scientist designing an experiment, not just a developer writing a test.

Significance in the Larger Narrative

This message sits at a inflection point in the debugging saga. The preceding messages had been characterized by broad hypothesis generation, code auditing, and exclusion of known bugs. The assistant had produced an elaborate plan with six ranked hypotheses and a detailed bisection sequence. But plans are not evidence. This message represents the transition from planning to execution — from "we think the bug might be X" to "let us build a tool that will tell us definitively."

The harness built in this message will be used in subsequent messages to run the decisive A/B tests: fp8 index-K versus bf16 index-K at high concurrency, HiCache on versus HiCache off, CUDA_LAUNCH_BLOCKING on versus off. Without this harness, those tests would be impossible — the assistant would be reduced to guessing based on production observations, which are inherently noisy and irreproducible.

In a deeper sense, this message embodies a philosophy of debugging: when faced with a complex, intermittent production bug, the first priority is not to guess the root cause but to build the apparatus that can produce reliable evidence. The harness is an investment in epistemic quality — it transforms the investigation from speculation into science.