The Moment of Discovery: Uncovering SGLang's Reasoning Capture Bug

Introduction

In any complex software pipeline, the most critical debugging moments are those where an assumption collapses under the weight of direct observation. Message 3761 in this opencode session represents precisely such a moment: the assistant, after hours of building an EAGLE-3 training pipeline for the Kimi-K2.5 model, makes a raw HTTP request to the SGLang inference server and discovers that the reasoning content it had been faithfully collecting for thousands of samples was, in fact, never properly captured at all. This article examines that single message in depth—its context, its reasoning, the assumptions it shattered, and the cascade of decisions it triggered.

The Context: A Pipeline Built on a Faulty Foundation

To understand why message 3761 is so significant, one must appreciate what came before it. The assistant and user had been working for days on an elaborate EAGLE-3 speculative decoding pipeline. The workflow involved generating synthetic training data by sending prompts to a Kimi-K2.5 model running on SGLang, then using the responses (including the model's internal reasoning) to train a lightweight drafter model that could predict the base model's next tokens more efficiently.

The data generation pipeline (run_inference.py) had already produced thousands of samples across multiple datasets—B1_glaive, B2_opencodeinstruct, B3_magicoder, and others. The assistant had previously fixed a "reasoning capture bug" in an earlier 10K-sample version of this pipeline, and the assumption was that the current 100K-scale pipeline was correctly extracting the model's chain-of-thought reasoning into a separate reasoning field for training.

This assumption was first challenged in [msg 3746], when the user pasted two sample responses from the pipeline and observed: "Seems again we're not capturing reasoning correctly (this was also a bug previously fixed in 10k version)." The samples showed "reasoning": ""—empty—while the actual reasoning text was embedded inside "content". The bug had returned, or rather, had never truly been fixed.

The Investigation: Tracing the Data Flow

The assistant's response in [msg 3747] began the investigation by reading run_inference.py and identifying line 59: reasoning = getattr(msg, "reasoning", None) or "". This line assumed that the OpenAI-compatible chat completions API would return a reasoning attribute on the message object. But was SGLang actually providing it?

A series of diagnostic commands followed. In [msg 3748], the assistant SSH'd into the container and inspected the raw response files directly:

d = json.loads(line)
print("Keys:", list(d.keys()))
print("reasoning repr:", repr(d.get("reasoning", "")))
print("content[:200]:", repr(d.get("content", "")[:200]))

The output was damning: reasoning repr: '' while content began with ' The user is asking for the latest news headlines...'—the thinking text was embedded in content, not separated.

Further probing in [msg 3749] confirmed that every sample had has_think_in_content=True but reasoning=''. In [msg 3750], the assistant located the </think> tag at position 314 within the content string, confirming the structure: the model's reasoning was present in the output, but the API wasn't splitting it out.

The Raw Request: Message 3761 as the Pivot Point

Message 3761 is the assistant's analysis after making a direct curl request to the SGLang server's /v1/chat/completions endpoint ([msg 3760]). The raw response revealed the exact structure SGLang was returning:

{
    "choices": [{
        "message": {
            "role": "assistant",
            "content": " The user is asking a simple math question...\n\n This is a straightforward...\n\n I should provide the answer..."
        }
    }],
    "usage": {
        "reasoning_tokens": 0
    }
}

The assistant's analysis in message 3761 is precise and methodical:

There it is. The SGLang response has: - message.content — contains EVERYTHING including the think content (without <think> opening tag, but with </think>) - message.reasoning_content — null - usage.reasoning_tokens — 0

>

So SGLang is not splitting the reasoning into reasoning_content. The content starts with the thinking text (no <think> prefix), has </think> in the middle, then the actual answer.

This is the moment of clarity. The assistant now understands the exact shape of the problem:

  1. SGLang's chat completions endpoint does not automatically parse reasoning content. The --reasoning-parser server flag (which the assistant would discover moments later in [msg 3769]) was not set when the server was launched. Without it, SGLang treats the model's raw token output as a single blob of content.
  2. The model's output format is <think>reasoning</think>response, but SGLang strips the opening <think> tag while leaving </think> in place. This means the content string starts directly with the reasoning text, has </think> somewhere in the middle, and then the actual assistant response follows.
  3. The OpenAI-compatible API fields reasoning_content and reasoning_tokens are null/zero because SGLang doesn't populate them without the reasoning parser enabled.

The Proposed Fix and Its Assumptions

The assistant proposes a three-part fix in message 3761:

1. Not rely on reasoning / reasoning_content from the API 2. Parse the content field to split on </think> — everything before is reasoning, everything after is the actual content 3. Also prepend <think> since the model's output starts with thinking but SGLang strips the opening <think> tag

And then adds a prudent caveat:

Let me also check if there's a way to make SGLang return reasoning_content properly — but regardless, we need to fix the parsing.

This reveals an important assumption: the assistant assumes that parsing the content field manually is the correct approach, but also suspects there might be a server-side configuration that would make the problem go away entirely. This dual-track thinking—fix the client and investigate the server—is characteristic of good debugging.

However, there is a subtle mistake in the initial analysis. The assistant writes "without <think> opening tag, but with </think>" and later says "prepend <think> since the model's output starts with thinking but SGLang strips the opening <think> tag." This assumes that the opening tag is always stripped. In reality, the behavior depends on the model's tokenizer and how SGLang processes the raw logits. The <think> token (token ID 163606 for this model) is a special token that the model emits at the start of its reasoning. SGLang's chat template processing may or may not preserve it depending on configuration. The assistant's assumption that the opening tag is always stripped would need verification across different prompt types and generation settings.

The Knowledge Flow: Input and Output

Input knowledge required to understand this message includes:

The Thinking Process Revealed

The assistant's reasoning in message 3761 shows a clear diagnostic pattern:

  1. Observation: The raw API response has been obtained and examined.
  2. Comparison: The actual response structure is compared against the expected structure (what the OpenAI client library would expect).
  3. Gap identification: The specific discrepancies are enumerated (reasoning_content is null, reasoning_tokens is 0, content contains everything).
  4. Root cause inference: The assistant deduces that SGLang is "not splitting the reasoning into reasoning_content."
  5. Solution design: A three-part parsing fix is outlined.
  6. Secondary investigation: The assistant notes the need to check if a server-side option exists, showing awareness that the client-side fix might be suboptimal. The todo list update in the message is also revealing. The assistant changes the status of "Phase 2: Response generation" from running to "STOPPED - fixing reasoning capture bug" and adds a new high-priority todo: "Fix run_inference.py to parse <think>...</think> from content field." This shows the operational impact of the discovery—the entire data generation pipeline is halted until the fix is applied.

The Aftermath: What Followed

Immediately after message 3761, the user in [msg 3763] suggested: "Pretty sure we're running sglang with wrong reasoning parser(?)" This prompted the assistant to investigate the server configuration (<msg id=3764-3769>), where it discovered the --reasoning-parser flag with a kimi_k2 option that maps to Qwen3Detector—a detector that uses &lt;think&gt;/&lt;/think&gt; tags, exactly matching Kimi-K2.5's format.

This led to a more elegant solution: restart the SGLang server with --reasoning-parser kimi_k2 so that the API would properly populate reasoning_content and reasoning_tokens. The client-side parsing hack was no longer necessary.

However, the story doesn't end there. As the chunk summary reveals, the fix ultimately involved a more radical approach: rewriting run_inference.py to bypass the OpenAI-compatible chat completions API entirely and use SGLang's /generate endpoint with raw input_ids/output_ids. This was because even with the reasoning parser enabled, the chat completions API introduced ambiguities in how tool calls and special tokens were represented. The /generate endpoint, which works with raw token sequences, gave the pipeline complete control over what tokens to capture and how to interpret them.

Conclusion

Message 3761 is a textbook example of the debugging process in complex ML systems. It represents the transition from assumption-based reasoning to evidence-based reasoning. Before this message, the assistant assumed that the OpenAI-compatible API was correctly separating reasoning from content—after all, the code had been written to read the reasoning field, and earlier tests had seemed to work. But the direct observation of the raw API response shattered that assumption and revealed the true structure of the data.

The message also illustrates an important principle in systems integration: when two systems (the OpenAI client library and the SGLang server) communicate through a shared API schema, the client may interpret the server's responses based on assumptions that the server does not fulfill. The OpenAI client library expects reasoning_content to be populated; SGLang only populates it when explicitly configured to do so. The gap between expectation and reality is where bugs live.

For the EAGLE-3 training pipeline, this discovery was costly—thousands of samples had been generated with empty reasoning fields, rendering them useless for training the drafter model. But the insight gained—understanding exactly how SGLang represents reasoning content and how to configure it correctly—was invaluable. It transformed a mysterious "reasoning is empty" bug into a known, fixable configuration issue, and ultimately led to a more robust architecture using raw token endpoints that eliminated parsing ambiguity entirely.