The Pivot Point: Enabling Tool Calling on a Custom vLLM Deployment

In the sprawling arc of a months-long coding session to deploy large language models on a cluster of eight RTX PRO 6000 Blackwell GPUs, few messages carry the quiet weight of a simple user request. After dozens of rounds spent wrestling with flash-attn compilation failures, patching vLLM's GGUF loader for the obscure glm_moe_dsa architecture, debugging incoherent model output caused by tensor parallelism sharding mismatches, and finally achieving a stable ~55 tok/s single-request throughput with the GLM-5 744B GGUF model, the user issues a three-word command: "Enable tool calling on vllm" ([msg 2063]). The assistant's response — message [msg 2064] — is the first step in what will become a major pivot, one that ultimately leads away from GLM-5 entirely and toward a different model, a different quantization format, and a different set of engineering challenges.

The Message in Full

The subject message is deceptively brief. It contains two tool calls executed in parallel: an Exa web search for vLLM's tool calling documentation, and a bash command that greps the vLLM CLI help for tool-related flags. The assistant's preamble states the intent plainly:

Let me check what tool calling configuration vLLM supports and what's needed for GLM-5.

The bash command returns the critical output — the list of available --tool-call-parser options, which includes glm45, glm47, kimi_k2, and dozens of other model-specific parsers. This single line of output confirms that vLLM ships with built-in support for GLM-family tool calling, and that enabling it is a matter of passing the right flags rather than writing custom parsing logic.

Why This Message Matters

To understand the significance of this message, one must appreciate the journey that preceded it. The session had spent the better part of a week deploying GLM-5 — a 744B-parameter Mixture-of-Experts model — in GGUF format with a custom Q4_K_XL quantization. The deployment required patching vLLM's gguf_loader.py to support the glm_moe_dsa architecture, fixing a latent bug in DeepSeek V2/V3 GGUF support along the way, building llama-gguf-split from source to merge ten split files into a single 402GB weight file, implementing a custom Triton MLA sparse attention backend for Blackwell SM120 GPUs, debugging weight loading KeyErrors by force-dequantizing indexer weights, and fixing two separate bugs in the Triton MLA attention backend and GGUF dequantization shard ordering that had been producing garbage output. The final working service achieved ~55 tok/s — impressive for a 744B-parameter model on PCIe-only GPUs, but still limited by the allreduce bottleneck that consumed 65-70% of each decode step.

The user's request to enable tool calling arrives at a moment of apparent stability. The service is running, producing coherent output, and the assistant has just declared "All done" in [msg 2062]. But tool calling is not a trivial add-on — it requires the model to support function-calling syntax in its chat template, and vLLM to have a compatible parser that can extract tool calls from the model's generated text and inject tool definitions into the prompt.

The Two-Pronged Research Strategy

The assistant's response reveals a methodical research approach. Rather than blindly adding flags to the systemd service file, it simultaneously consults two sources of truth: the official vLLM documentation (via Exa search) and the concrete CLI implementation (via --help). This dual approach is characteristic of the assistant's working style throughout the session — it never assumes documentation matches implementation, and it always verifies against the actual installed software.

The Exa search query — "vLLM tool calling function calling configuration --tool-call-parser --enable-auto-tool-choice" — is carefully constructed to surface the exact documentation page for the CLI flags that will need to be added. The bash command — ssh root@10.1.230.174 '/root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --help 2>&1 | grep -i -A2 "tool"' — runs against the live server environment, ensuring the output reflects the actual installed vLLM version (a nightly build, not a stable release).

What the Output Reveals

The grep output is revelatory. It shows that vLLM supports --enable-auto-tool-choice (with its negating counterpart --no-enable-auto-tool-choice) and --tool-call-parser with a long list of model-specific parsers. Among them are glm45 and glm47 — two parsers for the GLM-4.5 and GLM-4.7 model families respectively. The presence of these parsers suggests that GLM-5, being a successor in the same lineage, likely uses a compatible tool-calling format. Also present is kimi_k2, which will become relevant later when the session pivots to the Kimi-K2.5-NVFP4 model.

The output also reveals --exclude-tools-when-tool-choice-none, a flag that controls whether tool definitions are stripped from the system prompt when tool_choice is set to "none". This is important for production deployments where the same endpoint serves both tool-using and non-tool-using clients.

The Assumptions Embedded in the Message

The assistant makes several assumptions in this message, all of them reasonable but worth examining. First, it assumes that GLM-5 is still the model being served — the pivot to Kimi-K2.5-NVFP4 has not yet occurred, and the systemd service is still named vllm-glm5. Second, it assumes that one of the existing GLM parsers (glm45 or glm47) will be compatible with GLM-5, an assumption that will be validated in subsequent messages by inspecting the chat template. Third, it assumes that the vLLM nightly build includes the tool-calling feature, which the CLI output confirms.

There is one notable gap: the assistant does not yet check whether the GLM-5 model's tokenizer has a chat template that supports tool calling. This investigation happens in the very next message ([msg 2065]), where the assistant loads the tokenizer from HuggingFace and inspects its chat_template. The template indeed contains tool-calling XML syntax with <tool_call>, <arg_key>, and <arg_value> tags, confirming compatibility.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the structure of the message. It begins with a declarative intent statement — "Let me check what tool calling configuration vLLM supports and what's needed for GLM-5" — which frames the investigation as a two-part problem: understanding vLLM's capabilities and understanding the model's requirements. The parallel execution of the Exa search and the bash command shows that the assistant treats documentation and implementation as complementary sources that should be consulted simultaneously.

The choice to grep for "tool" in the help output rather than reading the full help text is a deliberate efficiency: the --help output for vLLM's API server is hundreds of lines long, and the assistant knows that the relevant flags all contain the word "tool." This pattern of targeted information extraction — using grep, head, and other Unix tools to focus on relevant output — is a hallmark of the assistant's approach throughout the session.

The Broader Significance

Message [msg 2064] sits at a inflection point in the conversation. It is the last message that deals with GLM-5 before the session pivots to the Kimi-K2.5-NVFP4 model — a 1T-parameter MoE model using NVIDIA's NVFP4 quantization, downloaded as 119 safetensor shards totaling 540GB. The tool-calling investigation initiated here will carry forward to the new model, and the kimi_k2 parser visible in the CLI output will become directly relevant.

In a deeper sense, this message exemplifies the assistant's engineering philosophy: never guess, always verify. When faced with a configuration task, it does not blindly append flags to a service file based on memory or assumption. It consults the documentation, checks the actual CLI, and only then proceeds to implementation. This discipline, maintained across hundreds of messages and dozens of tool calls, is what allowed the session to successfully deploy not one but two massive language models on hardware that was never designed for them — PCIe-connected consumer-grade GPUs running server-class inference workloads.

The message is also a testament to the power of small, focused investigations. In two tool calls and a single sentence of intent, the assistant maps the entire landscape of vLLM's tool-calling support, identifies the relevant parsers for the model family, and sets the stage for the configuration changes that follow. It is a reminder that in complex engineering work, the most valuable output is often not code but understanding — and that understanding begins with asking the right questions in the right order.