The Unparsed Tokens Problem: When a Production Server Can't Speak Its Model's Language

The Message

The server seems to work, but it doesn't output correct tool-calls it seems, or doesn't parse tool-calls and thinking correct for the model, In a separate session when I called the model it outputed unparsed thinking and tool-call tokens: "... let me check if there are any existing files in the workspace. response <|tool_calls_section_begin|> <|tool_call_begin|> functions.glob:0 <|tool_call_argument_begin|> {"pattern": "*/"} <|tool_call_end|> <|tool_calls_section_end|>", the tokens with <|.. are correct for kimi, but didn't get parsed?

This message, sent by the user at a critical juncture in the deployment pipeline, encapsulates a subtle but devastating class of production bug: the model is generating the right tokens, but the serving infrastructure is failing to interpret them. The Kimi-K2.5 INT4 server is running, generating tokens, and returning HTTP 200 responses — yet it is fundamentally broken for its intended purpose as an agentic coding assistant. This article dissects the reasoning, context, assumptions, and knowledge boundaries surrounding this single message, which triggered the next phase of the production hardening effort.

Context: What Came Before

To understand why this message carries such weight, one must appreciate the journey that preceded it. The conversation leading up to this point (segments 33–38 of the opencode session) had been an exhaustive, multi-day optimization campaign for deploying large language models on a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 (no NVLink). The user and assistant had:

• Diagnosed that EAGLE-3 speculative decoding was performing worse than baseline due to NCCL all-reduce bottlenecks on PCIe
• Upgraded the entire CUDA stack from version 12.8 to 13.0 to unblock Blackwell-native optimizations
• Patched SGLang source code for SM120 (Blackwell architecture) support
• Enabled FlashInfer allreduce fusion and Torch symmetric memory
• Benchmarked extensively, eventually settling on a topk=1 + spec_v2 (overlap scheduling) configuration that matched or beat baseline throughput at high concurrency
• Created a comprehensive production documentation file (/root/production_v2.md)
• Built a systemd service (sglang-kimi.service) with auto-start on boot
• Verified the server was healthy and generating tokens The very last thing the assistant had done before this message was bind the server to 0.0.0.0:30000 so it could be accessed from outside the container. The user had asked "What port?", the assistant had replied "30000 — bound to 127.0.0.1", the user said "bind to 0.0.0.0", and the assistant had just finished restarting the service with the corrected binding. The server was healthy, listening on all interfaces, and generating tokens. And then the user tried to use it.

The Core Problem: Token-Level Correctness vs. API-Level Correctness

The user's message reveals a fundamental distinction that is easy to overlook in the heat of deployment: a model server can be technically operational — responding to health checks, loading weights correctly, and generating syntactically valid tokens — while being functionally broken for its intended use case.

The user reports that when they called the model in a separate session, it output:

... let me check if there are any existing files in the workspace.  response <|tool_calls_section_begin|> <|tool_call_begin|> functions.glob:0 <|tool_call_argument_begin|> {"pattern": "*/"} <|tool_call_end|> <|tool_calls_section_end|>

These tokens — <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|>, <|tool_call_end|>, <|tool_calls_section_end|> — are the correct special tokens for the Kimi-K2.5 model's tool-calling format. The model is generating them correctly. But SGLang's OpenAI-compatible API endpoint is returning them as raw text in the content field of the chat completion response, rather than parsing them into the structured tool_calls array that the API specification demands.

This is the difference between a model that can generate tool calls and a serving stack that understands them. The model is doing its job. The server is not.

Why This Problem Occurred: The Missing Parsers

SGLang, like many LLM serving frameworks, supports "tool-call parsers" and "reasoning parsers" — post-processing modules that intercept the raw token stream from the model and transform it into structured API output. For the OpenAI-compatible chat completions endpoint, this means:

1. Tool-call parsing: Detecting special tokens like <|tool_call_begin|> in the output and converting them into the structured tool_calls array with id, type, function, and arguments fields
2. Reasoning parsing: Detecting thinking/reasoning tokens (often wrapped in special delimiters) and separating them from the final assistant response, typically placing them in a reasoning_content field The Kimi-K2.5 model uses a specific vocabulary of special tokens for both purposes. The <|tool_calls_section_begin|> and related tokens are its native format for expressing tool calls, and the model likely uses <|thinking_begin|> / <|thinking_end|> or similar tokens for reasoning content. The user's observation — "the tokens with <|.. are correct for kimi, but didn't get parsed" — demonstrates a correct diagnosis. The model is emitting the right tokens. The server simply wasn't configured with the --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 flags that tell SGLang how to interpret these model-specific tokens.

Assumptions and Blind Spots

This message exposes several assumptions that had been operating beneath the surface of the deployment effort:

Assumption 1: A working server is a usable server. The assistant had focused intensely on getting the server to start, load the 547 GB model, and generate tokens. Health check returning 200? Check. Tokens coming out? Check. But "working" in the engineering sense (process running, model loaded, tokens generated) is not the same as "working" in the user's sense (API returning properly structured responses that an agentic application can consume).

Assumption 2: Model format knowledge is implicit. The assistant knew that Kimi-K2.5 used special tokens for tool calls — it had patched the model file (kimi_k25.py) earlier with EAGLE-3 delegation methods. But knowing about the tokens at the model architecture level is different from configuring the serving layer to parse them. The assistant had never explicitly added the parser flags to the launch command or the systemd service.

Assumption 3: The OpenAI-compatible API "just works." There's an implicit belief that if you serve a model through an OpenAI-compatible endpoint, the tool-calling format will be automatically handled. In reality, each model family has its own special token vocabulary, and the server must be explicitly told which parser to use. Without this configuration, the API returns raw, unparsed text — technically valid as a chat completion, but useless for programmatic tool-call handling.

Assumption 4: Previous testing was sufficient. The assistant had tested the server with a simple generation request ("Write a Python function to check if a number is prime") which doesn't trigger tool calls or reasoning. The test passed, but it didn't exercise the features that actually matter for the model's intended use case as an agentic coding assistant. The user's separate session revealed the gap.

The Knowledge Required to Understand This Message

To fully grasp the significance of this message, one needs:

1. Understanding of LLM serving architecture: Knowledge that serving frameworks like SGLang, vLLM, and TGI have a pipeline from raw token generation → post-processing → API response formatting, and that each stage can be independently configured.
2. Model-specific token vocabulary: Awareness that Kimi-K2.5 (and models in the Kimi family) uses custom special tokens like <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|>, and <|tool_call_end|> for structured tool calls, and similar tokens for reasoning/thinking content.
3. OpenAI API specification: Knowledge that the chat completions API has a specific format for tool calls — an array of tool_calls objects with id, type, function.name, function.arguments fields — and that raw text containing special tokens does not satisfy this specification.
4. SGLang parser configuration: Familiarity with the --tool-call-parser and --reasoning-parser command-line flags, which accept model-family identifiers (like kimi_k2) and enable the appropriate post-processing logic.
5. The deployment context: Understanding that the server had just been converted to a systemd service, that the launch command was embedded in the service file, and that any configuration change would require editing the service file, reloading systemd, and restarting the server (with a ~9.5 minute cold start).

The Thinking Process Visible in the Message

The user's message reveals a methodical debugging approach. They didn't just say "the tool calls don't work." They:

1. Tested the server independently: "In a separate session when I called the model" — they ran their own test, not relying on the assistant's verification.
2. Captured the exact raw output: They quoted the precise token sequence the model produced, including the special tokens and their positions relative to the natural language content.
3. Identified the correct behavior vs. observed behavior: "the tokens with <|.. are correct for kimi, but didn't get parsed" — they recognized that the tokens themselves were valid model output, but the serving layer failed to process them.
4. Framed the problem as a parsing issue: The use of the word "parsed" shows they understood the architecture — the model generates tokens, and then a parser needs to interpret them. The parser was missing or not configured. This is a user who understands the system architecture deeply. They're not reporting a vague "it doesn't work" — they're providing a precise delta between expected behavior and observed behavior, with enough detail for the assistant to immediately identify the fix.

The Output Knowledge Created

This message created actionable knowledge that directly shaped the next phase of work:

1. The systemd service needed updating: The ExecStart command in /etc/systemd/system/sglang-kimi.service needed --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 flags added.
2. The production documentation needed updating: /root/production_v2.md needed to document these parser flags and explain their purpose.
3. The verification procedure needed expanding: Future server tests should include a tool-call-eliciting prompt (e.g., "What files are in the current directory?") to verify parser functionality, not just a simple text generation test.
4. The restart cost was now known: Any configuration change would incur a ~9.5 minute cold start penalty, making it important to batch changes and minimize restarts.
5. A broader pattern was revealed: The gap between "server is running" and "server is usable" is a recurring theme in LLM deployment, and future work should explicitly verify the full API contract, not just basic generation.

The Broader Significance

This message is a microcosm of a challenge that pervades LLM deployment in production: the gap between model-level correctness and serving-level correctness. A model can generate perfect tokens. A server can load those tokens and stream them over HTTP. But the semantic layer — the interpretation of those tokens as structured actions, reasoning chains, or tool invocations — requires explicit configuration that is easy to forget.

In the rush to get a model serving in production, it's tempting to declare victory when the health check passes and tokens flow. This message is a reminder that the real test is whether the API contract is fulfilled — whether the consumer of the API (whether a human or an agentic application) gets data in the format they expect. The Kimi-K2.5 model speaks in <|tool_calls_section_begin|> and <|tool_call_begin|>. The OpenAI-compatible API speaks in JSON arrays of tool call objects. The server's job is to translate between these two languages — and without the parser flags, that translation never happens.

The user's message, precise and diagnostic, closed the gap between "the server works" and "the server works correctly." It's a textbook example of a user providing exactly the information needed to identify and fix a configuration bug, and it set the stage for the next round of production hardening that would follow.