The Final Polish: Adding Tool Call and Reasoning Parsers to a Production SGLang Deployment

In the lifecycle of deploying a large language model to production, the journey rarely ends when the server starts returning tokens. The real work begins when you discover that those tokens are not being structured correctly for the downstream consumers. Message 5699 in this opencode session captures exactly such a moment — a seemingly small, two-flag addition to a systemd service file that transforms a raw-token-spewing inference server into a properly parsed, API-compliant production endpoint for the Kimi-K2.5 INT4 model.

The Context: A Hardened Deployment Takes Shape

By the time we reach message 5699, the assistant has already accomplished an enormous amount. The Kimi-K2.5 INT4 model — a 547 GB behemoth — has been deployed across 8 RTX PRO 6000 Blackwell GPUs using SGLang with EAGLE-3 speculative decoding. The optimal configuration was painstakingly discovered through dozens of benchmarks: topk=1 speculation with spec_v2 overlap scheduling, flashinfer attention backend with allreduce fusion, and CUDA 13 stack with Blackwell-specific optimizations. The deployment had been codified into a systemd service (sglang-kimi.service) that starts on boot, auto-restarts on failure, and binds to 0.0.0.0:30000 for external access.

But then came the user's report in message 5690. When they called the model from a separate session, the output contained raw, unparsed special tokens:

"... let me check if there are any existing files in the workspace. response <|tool_calls_section_begin|> <|tool_call_begin|> functions.glob:0 <|tool_call_argument_begin|> {"pattern": "*/"} <|tool_call_end|> <|tool_calls_section_end|>"

The tokens with <|... were correct for Kimi-K2.5's chat template, but SGLang was not parsing them into the structured API response fields that clients expect. The reasoning_content field was null, the thinking was dumped raw into content, and tool_calls was empty — the structured function call information was embedded as raw text. This is a critical failure for any model intended to be used as a function-calling agent, which is precisely what Kimi-K2.5 is designed for.

The Investigation: Finding the Right Flags

The assistant's response in message 5691 correctly diagnosed the issue as a "tokenizer/chat template issue — SGLang isn't parsing Kimi's special tokens for tool calls and thinking." It then launched a subagent task (message 5696) to search the SGLang codebase for available parser options. The task returned a comprehensive summary: SGLang v0.5.9 supports a ToolCallParserEnum registry with entries for various model families including deepseekv3, deepseekv31, deepseekv32, and crucially, kimi_k2. Similarly, the reasoning parser had a kimi_k2 option.

The subagent also discovered that the Kimi-K2.5 model ships with its own deployment guide that explicitly documents these flags. The answer was straightforward: --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2.

Before applying the fix, the assistant wisely reproduced the issue in message 5697 by sending a curl request to the running server with a tool-use prompt. The response confirmed the problem: the model output thinking tokens and tool call tokens embedded in the content field, with reasoning_content null and tool_calls absent. This reproduction was critical — it validated that the issue existed in the current deployment and provided a baseline to verify the fix against later.

The Fix: Rewriting the Systemd Service

Message 5699 is the application of that fix. The assistant uses a cat heredoc piped through SSH to completely overwrite the systemd service file on the remote machine. This is a deliberate and informed choice. Earlier, in messages 5682-5684, the assistant had attempted to use sed to add the --host 0.0.0.0 flag to the service file, and the result was a disaster — the flag was duplicated, appearing both before and after --mem-fraction-static 0.88. The assistant had to rewrite the entire file to fix it. That lesson was learned: for critical infrastructure files, a full rewrite via heredoc is safer than surgical sed patches, especially when the file is already being modified for other reasons.

The new service file is identical to the previous version except for two additional lines at the end of the ExecStart command:

  --tool-call-parser kimi_k2 \
  --reasoning-parser kimi_k2

These flags tell SGLang to use Kimi-K2-specific parsers that recognize the model's unique special tokens. The --tool-call-parser kimi_k2 flag activates a detector that scans generated text for Kimi's <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|>, and <|tool_call_end|> tokens, extracting structured function call data into the API's tool_calls field. The --reasoning-parser kimi_k2 flag similarly detects <|think|> and <|/think|> tokens (or Kimi's equivalent reasoning markers) and separates the reasoning content into the reasoning_content field, keeping the final answer clean in content.

The Thinking Process: What This Message Reveals

The assistant's reasoning in this message is compact but revealing. The todo list progression visible in the surrounding messages shows a clear three-step plan: investigate → configure → update and restart. By message 5699, the investigation is complete (the subagent found the flags, the reproduction confirmed the bug), and the configuration is in progress. The assistant is executing step two: updating the service file.

The decision to rewrite the entire file rather than patch it reflects a practical lesson in operational discipline. When you're managing a production service that takes 9.5 minutes to cold-start a 547 GB model, every restart is costly. Getting the service file right in one shot is essential. The heredoc approach guarantees correctness because the entire file is specified in one place, with no risk of regex edge cases or line-ordering surprises.

There is also an implicit assumption in this message: that the two flags are sufficient and that no other configuration changes are needed. This assumption is reasonable given the subagent's thorough investigation of the SGLang codebase, which confirmed that kimi_k2 parsers exist and are designed specifically for this model family. The assistant does not, however, verify the fix by restarting the server and re-running the curl test within this message — that comes in the subsequent messages (not shown in the subject). The message is purely about updating the configuration file.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of context. First, familiarity with systemd service files — the [Unit], [Service], and [Install] sections, the ExecStart directive, environment variables, and restart policies. Second, knowledge of SGLang's server architecture: the --tool-call-parser and --reasoning-parser flags are SGLang-specific and not part of the OpenAI API specification — they are SGLang's mechanism for adapting model-specific chat templates to the standardized API format. Third, understanding of Kimi-K2.5's token structure: the model uses custom delimiter tokens for tool calls and reasoning that differ from other models like DeepSeek or Llama. Fourth, awareness of the earlier sed mishap (messages 5682-5684) that motivated the full-file rewrite approach.

Output Knowledge Created

This message produces a corrected systemd service file that will, upon restart, enable proper parsing of Kimi-K2.5's tool calls and reasoning content. More broadly, it creates a documented pattern for fixing similar issues: (1) identify the model family's special tokens, (2) search the SGLang codebase for corresponding parser implementations, (3) add the appropriate --tool-call-parser and --reasoning-parser flags, and (4) use a full-file rewrite to avoid sed-induced corruption.

The message also implicitly documents that Kimi-K2.5 requires these two specific flags — knowledge that is not obvious from SGLang's default configuration. Without them, the model's function-calling capability is effectively broken from the API consumer's perspective, even though the raw tokens are correct.

A Lesson in Production Polish

Message 5699 is a reminder that deploying a model is not the same as deploying a model correctly. The server was running, generating tokens, passing health checks, and responding to requests — but it was failing at the crucial task of structuring its output for programmatic consumption. The two flags added in this message are the difference between a server that returns raw, client-side-unparseable text and one that returns clean, API-compliant JSON with properly separated reasoning_content and tool_calls fields.

For any production LLM deployment, especially one intended for agentic use cases where tool calling is the primary interface, this kind of parsing configuration is not optional. It is the difference between a model that can be integrated into an automated pipeline and one that requires a human to manually extract function calls from raw text. The assistant's careful, methodical approach — investigate, reproduce, fix, verify — is a model of how to handle such issues in a production environment where every restart costs nearly ten minutes of cold-start time.