The Art of Discovery: Investigating Tool Calling Support for GLM-5 in vLLM

In the middle of a complex deployment session spanning multiple model architectures, GPU configurations, and quantization formats, a seemingly simple user request arrives: "Enable tool calling on vllm" ([msg 2063]). The assistant's response ([msg 2065]) is a brief investigative message that, on its surface, appears straightforward — two parallel commands, one observation, and a plan. But this message encapsulates a critical moment of architectural discovery, where the assistant must bridge the gap between what the user wants and what the underlying software stack can actually deliver.

The Context: A Long Journey of Model Deployment

To understand why this message matters, we must appreciate the context. The session has been a marathon of ML infrastructure engineering: deploying the GLM-5 model (a 744B-parameter MoE architecture) across 8 RTX PRO 6000 Blackwell GPUs, wrestling with flash-attn compilation, patching vLLM's GGUF loader for the glm_moe_dsa architecture, debugging Triton MLA attention backends, and finally achieving ~55 tok/s throughput. The GLM-5 service is now running as a systemd service, producing coherent output. The user's request to enable tool calling is a natural next step — moving from basic text generation to functional interaction where the model can call external tools and APIs.

The Reasoning: Why This Investigation Was Necessary

The assistant's reasoning, visible in the message's opening line, reveals a sophisticated chain of inference: "I can see that vLLM has a glm4_moe_tool_parser module and the CLI lists glm45 and glm47 parsers. GLM-5 likely uses one of these." This is not a guess — it's a reasoned deduction based on the vLLM codebase structure. The assistant had previously discovered (in [msg 2064]) that vLLM's --tool-call-parser CLI argument accepts values including glm45 and glm47. The existence of glm4_moe_tool_parser.py in the vLLM source confirms that the GLM model family has established tool parsing support.

The critical insight here is the model lineage: GLM-5 is the fifth generation of the GLM (General Language Model) series. If GLM-4 has a tool parser (glm4_moe_tool_parser), and vLLM explicitly lists glm45 and glm47 as supported parsers, then GLM-5 likely maps to one of these. The "45" in glm45 likely refers to GLM-4 version 5 (or GLM-4.5), while "47" refers to GLM-4 version 7. GLM-5, being a distinct generation, might use the glm45 parser (if it's considered a GLM-4.5 evolution) or might need a different approach entirely.

The Investigation: Two Parallel Reconnaissance Commands

The assistant dispatches two commands simultaneously — a pattern that reveals its understanding of the parallel execution model. Both commands are independent probes that don't depend on each other's output, so they can run concurrently.

The first command lists the contents of vLLM's tool_parsers directory:

ssh root@10.1.230.174 'ls /root/ml-env/lib/python3.12/site-packages/vllm/tool_parsers/ 2>&1'

This is a reconnaissance operation — cataloging what's available. The output reveals a rich ecosystem of tool parsers: glm4_moe_tool_parser.py, glm47_moe_tool_parser.py, kimi_k2_tool_parser.py, deepseekv3_tool_parser.py, and many more. The presence of both glm4_moe_tool_parser and glm47_moe_tool_parser is significant — it tells us that the GLM family has at least two generations of tool parsing support, and that glm47 is a subclass of glm4 (as we later learn from the file header in [msg 2066]).

The second command attempts to inspect the GLM-5 chat template directly from the model's tokenizer:

ssh root@10.1.230.174 'python3 -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained(\"zai-org/GLM-5\", trust_remote_code=True); print(t.chat_template[:3000] if t.chat_template else \"No chat template\")" 2>&1'

This command fails with ModuleNotFoundError: No module named 'transformers'. This is a telling failure — it reveals that the Python environment (/root/ml-env) does not have the transformers library installed. This is somewhat surprising given that vLLM typically depends on transformers for tokenization and model configuration. The failure could be because the environment was set up with a minimal vLLM installation, or because transformers was installed in a different path. The error message itself becomes valuable information — it tells the assistant that it cannot inspect the model's chat template through the standard transformers API, and must find another way.

Assumptions and Their Implications

The message operates on several key assumptions:

Assumption 1: GLM-5 uses one of the existing GLM tool parsers. This is reasonable — the GLM family shares architectural patterns, and vLLM's parser architecture is designed to be model-specific. However, it's not guaranteed. GLM-5 might use a completely different tool format that doesn't map to any existing parser. The assistant is operating on the principle of maximum reuse, which is sound engineering practice.

Assumption 2: The chat template is the authoritative source for tool calling format. This is correct in principle — the chat template defines how tools are formatted in the conversation history. But it's not the only source. The tool parser also needs to understand the model's output format for tool calls, which is defined by the model's training data and fine-tuning, not just the template.

Assumption 3: The glm45 parser in the CLI corresponds to a file in the tool_parsers directory. This turns out to be slightly off — the CLI lists glm45 as a parser name, but the directory contains glm4_moe_tool_parser.py and glm47_moe_tool_parser.py, not a file named glm45. This suggests that glm45 might be an alias or that the parser registration happens through a different mechanism (perhaps a mapping dictionary in the entrypoints code).

The Mistake: An Unprepared Environment

The most concrete "mistake" in this message is the assumption that transformers would be available in the Python environment. The command fails immediately with ModuleNotFoundError. This isn't a logical error — it's an environmental mismatch. The assistant is running commands in a remote SSH session on a machine where the Python environment was carefully constructed for vLLM inference, not for general model inspection. The transformers library might be installed in a different location, or might have been excluded from the minimal inference environment.

This failure is actually productive. It forces the assistant to find alternative approaches — and indeed, in the next message ([msg 2066]), the assistant successfully retrieves the chat template by explicitly invoking the venv's Python interpreter (/root/ml-env/bin/python3) rather than the system Python. The error also reveals that the default python3 on the remote machine is not the venv's Python, which is useful system knowledge.

Input Knowledge Required

To fully understand this message, one needs:

  1. vLLM's tool calling architecture: That vLLM supports model-specific tool parsers, that they're registered via the --tool-call-parser CLI argument, and that each parser implements logic for extracting tool calls from model output.
  2. GLM model family lineage: That GLM-4, GLM-4.5, GLM-4-7, and GLM-5 are related models sharing similar architectural patterns, and that tool parsing support for earlier generations might be applicable to later ones.
  3. The chat template concept: That HuggingFace tokenizers carry a chat_template property defining how conversations (including tool calls) are formatted into the model's input format.
  4. The deployment context: That the model is running as a systemd service on a remote machine, that the Python environment is at /root/ml-env, and that commands must be executed via SSH.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. The complete list of available tool parsers in this vLLM installation, confirming that GLM family parsers exist.
  2. The absence of transformers in the default Python path, which informs subsequent debugging and forces alternative approaches.
  3. The hypothesis that GLM-5 maps to either glm45 or glm47, which will be tested in subsequent messages.
  4. The structure of the vLLM tool parser directory, revealing that glm47_moe_tool_parser is a separate file from glm4_moe_tool_parser, suggesting an inheritance hierarchy (confirmed in [msg 2066] where we see Glm47MoeModelToolParser extends Glm4MoeModelToolParser).

The Thinking Process

The message reveals a clear investigative methodology. The assistant starts with an observation (the CLI lists GLM parsers), forms a hypothesis (GLM-5 likely uses one of these), and then designs two parallel experiments to test that hypothesis. The first experiment catalogs available resources; the second attempts to inspect the model's own specification. When the second experiment fails, the assistant doesn't treat it as a dead end — it treats it as information. The error message itself becomes data: "transformers is not installed in this Python environment."

This is characteristic of effective debugging: every outcome, including failures, produces useful knowledge. The assistant doesn't panic or retry the same command — it absorbs the information and will adjust its approach in the next round.

Conclusion

Message [msg 2065] is a brief but dense investigative step in a complex deployment journey. It demonstrates how a simple user request ("Enable tool calling") triggers a cascade of discovery: checking available infrastructure, probing model capabilities, and adapting to environmental constraints. The message's value lies not in what it accomplishes (the commands are dispatched, results are pending) but in how it structures the investigation — forming hypotheses, testing them in parallel, and treating every result as actionable information. In the broader narrative of the session, this message marks the transition from "is the model serving correctly?" to "can the model do something useful?" — a shift from infrastructure reliability to functional capability.