Validating Tool Calling on a 744B-Parameter Model: The Final Integration Test

Introduction

In the long arc of deploying a massive 744-billion-parameter language model (GLM-5) across eight RTX PRO 6000 Blackwell GPUs, there comes a moment when infrastructure, patches, and configuration converge into a single test. Message [msg 2073] captures that moment: the assistant sends a curl request to the freshly restarted vLLM API server, asking the model to use a tool — specifically, a get_weather function — and receives back a correct, structured tool call response. This message is not merely a status update; it is the culmination of an extraordinary engineering journey spanning driver installation, CUDA toolkit configuration, flash-attn compilation, GGUF patching, Triton kernel debugging, and systemd service orchestration. It is the moment when all the pieces prove they fit together.

The Message in Full

The assistant writes:

Server is up with Application startup complete. Let me test tool calling:

``bash ssh root@10.1.230.174 'curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '"'"'{ "model": "/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf", "messages": [ {"role": "user", "content": "What is the weather like in San Francisco today?"} ], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} }, "required": ["location"] } } } ], "tool_choice": "auto", "max_tokens": 256, "temperature": 0.1 }'"'"' 2>&1' ``

The response: ``json {"id":"chatcmpl-b94ccea37594ca14","object":"chat.completion","created":1771619653,"model":"/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf","choices":[{"index":0,"message":{"role":"assistant","content":"I'll check the current weather in San Francisco for you.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[{"id":"chatcmpl-tool-b88428162229c4d0","type":"function","function":{"name":"get_weather","arguments":"{\"location\": \"San Francisco, CA\"}"}}],"reasoning":null},"l... ``

Why This Message Was Written: The Motivation and Context

The immediate trigger for this message was the user's request at [msg 2063]: "Enable tool calling on vllm." This seemingly simple instruction set off a chain of investigation and configuration changes. The assistant had to determine which tool-call parser GLM-5 required, verify that the chat template matched the parser's expectations, update the systemd service file, restart the service, wait for the 402GB GGUF model to reload across all eight GPUs, and finally validate that everything worked end-to-end.

But the deeper motivation goes beyond the user's request. Throughout the session — segments 12 through 16 — the assistant had been fighting an uphill battle to get GLM-5 working at all. The model had been downloaded as a set of GGUF split files, merged into a single 402GB file, loaded through custom patches to vLLM's gguf_loader.py and weight_utils.py, and debugged through incoherent output caused by tensor parallelism sharding mismatches and Triton MLA attention backend bugs. By [msg 2062], the assistant had achieved a working deployment with ~55 tok/s throughput. Tool calling was the final capability to enable before the deployment could be considered production-ready.

This message therefore represents the capstone validation — the test that proves the model is not merely generating tokens, but is correctly interpreting structured function definitions, deciding when to invoke a tool, formatting the arguments according to the JSON schema, and returning them through vLLM's OpenAI-compatible API. Without this test, the deployment would be incomplete.

How Decisions Were Made

Several decisions precede this message and are reflected in its content.

Parser selection: When the user asked to enable tool calling, the assistant first researched what parsers vLLM offered ([msg 2064]). It discovered that vLLM's CLI listed glm45 and glm47 parsers. Rather than guessing, the assistant inspected the actual GLM-5 chat template by loading it from HuggingFace ([msg 2066]). The template revealed a <tool_call> XML format with <arg_key>/<arg_value> pairs — a format that matched the glm47 parser exactly. The assistant also read the parser source code ([msg 2067]) to confirm that Glm47MoeModelToolParser inherited from Glm4MoeModelToolParser and overrode the regex to handle the specific format. This was a data-driven decision, not an assumption.

Service file update: The assistant edited the systemd service file to add --enable-auto-tool-choice and --tool-call-parser glm47 to the vLLM command line ([msg 2069]). This required understanding that tool calling is not enabled by default in vLLM — it must be explicitly activated via CLI flags.

Test design: The test prompt — "What is the weather like in San Francisco today?" — was chosen because it is a classic tool-calling benchmark. It requires no real external data; the model simply needs to recognize that it should call get_weather with the location extracted from the query. The tool_choice: "auto" setting lets the model decide whether to call a tool, which tests the model's autonomous reasoning. The low temperature (0.1) ensures deterministic output for verification.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit.

The model would correctly invoke the tool: This is the central assumption being tested. The assistant assumed that GLM-5, with the glm47 parser, would recognize the get_weather function signature and format a correct tool call. This was a reasonable assumption given that GLM-5's chat template explicitly supports tool calling in the <tool_call> XML format, and the glm47 parser was designed for this exact format.

The service would be healthy after restart: The assistant waited approximately 6 minutes between triggering the restart ([msg 2070]) and running this test ([msg 2073]). It assumed the model would finish loading within that window. The assistant checked at [msg 2071] (after 60 seconds) that the service was active and loading was progressing, and at [msg 2072] (after 360 more seconds) that the API server routes were registered. The final check showed "Application startup complete," confirming the assumption was correct.

The curl quoting would work: The assistant used a complex quoting pattern ('"'"') to embed JSON within a remote SSH command. This assumes that the quoting would survive two levels of shell interpretation — the local shell and the remote SSH shell. This is a non-trivial shell quoting challenge, and the assistant got it right.

No regressions from the tool-calling flags: Adding --enable-auto-tool-choice and --tool-call-parser glm47 could theoretically break existing functionality (e.g., regular chat completions). The assistant implicitly assumed these flags would not interfere with the model's core generation capability.

Mistakes and Incorrect Assumptions

The message itself contains no visible mistakes — the test succeeded. However, examining the broader context reveals some noteworthy points.

The test is minimal: A single tool call test with one function and one parameter does not fully validate tool calling. It does not test multiple tools, required vs. optional parameters, streaming tool calls, parallel tool calls, or the tool_choice: "required" mode. The assistant could have designed a more comprehensive test suite but chose a minimal smoke test. This is a pragmatic trade-off — the service had already been restarted multiple times, and a quick validation was appropriate before declaring success.

The response contains no reasoning trace: The model returned "reasoning":null. GLM-5 is capable of chain-of-thought reasoning, but the test did not include a reasoning_parser flag or a system prompt asking for reasoning. This is not a mistake per se — tool calling and reasoning are separate features — but it means the test did not verify that tool calling works correctly when reasoning is also enabled, which is a common production scenario.

The model's "content" field is a filler response: The model generated "I'll check the current weather in San Francisco for you." as the content alongside the tool call. This is the model's natural-language acknowledgment before the tool call. Some deployments prefer to suppress this filler text and return only the tool call. Whether this is desirable depends on the application, but the assistant did not configure any such suppression.

Input Knowledge Required to Understand This Message

To fully grasp what this message means, a reader needs knowledge spanning several domains:

vLLM architecture: Understanding that vLLM serves models via an OpenAI-compatible API, that it supports tool calling through parser plugins, and that --enable-auto-tool-choice and --tool-call-parser are CLI flags that activate this feature.

GLM-5 model family: Knowing that GLM-5 is a 744B-parameter MoE model from Zhipu AI (zai-org), that it uses a specific chat template with <tool_call> XML format, and that its architecture (glm_moe_dsa) required custom patches to vLLM's GGUF loader.

GGUF format and deployment: Understanding that the model was deployed as a 402GB GGUF Q4_K_XL quantized file, split across 10 files that had to be merged using llama-gguf-split, and loaded through custom vLLM patches that handled tensor parallelism sharding and dequantization.

Infrastructure context: Knowing that the server runs on Ubuntu 24.04 with 8 RTX PRO 6000 Blackwell GPUs (SM120), that NCCL communication happens over PCIe (no NVLink), and that the service runs as a systemd unit with NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS environment variables.

Shell quoting: The '"'"' pattern is a Bourne shell trick to embed single quotes within single-quoted strings. Understanding this is necessary to parse the curl command correctly.

Output Knowledge Created by This Message

This message produces several important pieces of knowledge:

Confirmation that GLM-5 tool calling works with vLLM's glm47 parser: This was previously untested. The successful response proves that the parser correctly extracts the tool call from the model's generated text and formats it as a structured API response.

Confirmation that the service restart was successful: The model loaded, the API server started, and it accepted and processed a tool-calling request without errors.

A validated configuration: The combination of --enable-auto-tool-choice, --tool-call-parser glm47, and the GLM-5 GGUF model is now a known-good configuration. This can be reused for future deployments.

A benchmark baseline for tool calling: The response time is not measured here (no time prefix on curl), but the created timestamp (1771619653) and the model's output provide a point of comparison for future tests.

Evidence that the earlier coherence issues are resolved: Throughout segments 15 and 16, the assistant struggled with incoherent model output caused by tensor parallelism sharding bugs and Triton attention backend issues. The fact that the model correctly parsed the tool definition, understood the user's intent, extracted "San Francisco, CA" from the query, and formatted a valid JSON arguments string demonstrates that the model is now producing coherent, semantically correct output.

The Thinking Process Visible in the Reasoning

While this message does not contain explicit reasoning tags (the assistant's internal monologue is not shown), the structure of the message reveals the assistant's thought process:

Verify service health first: The assistant opens with "Server is up with Application startup complete." This is not just a status update — it is a prerequisite check. The assistant knows that testing tool calling on a partially loaded or failed service would produce misleading errors.
Design a minimal but informative test: The assistant chooses a single-function, single-parameter tool call. This is the simplest possible test that validates the entire tool-calling pipeline: model recognizes tool definition → model decides to invoke → model formats arguments → parser extracts tool call → API returns structured response. Any failure in this chain would be immediately visible.
Use the exact same model path: The model parameter in the curl request matches the GGUF file path used throughout the deployment (/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf). This consistency ensures the test uses the same model instance that was just loaded.
Set temperature low for determinism: temperature: 0.1 reduces randomness, making the test reproducible. If the test were run again, the model would likely produce the same tool call.
Quote carefully for remote execution: The '"'"' pattern is a deliberate choice. The assistant could have piped the JSON through stdin or used a heredoc, but chose inline quoting for simplicity. The fact that the response came back correctly proves the quoting worked.
Read the response immediately: The assistant does not wait for a separate check — the curl command is the test, and its output is the result. This is efficient: one command to test, one response to verify.

The Broader Significance

This message sits at the intersection of several engineering narratives that run through the entire session.

The patching narrative: The GLM-5 GGUF deployment required extensive patches to vLLM — the gguf_loader.py was modified to handle the glm_moe_dsa architecture, weight_utils.py was patched for force-dequantization of indexer weights, and the Triton MLA attention backend was rewritten for Blackwell SM120 GPUs. Tool calling was the one feature that did not require patching — it worked with stock vLLM code once the correct parser was selected. This is a quiet victory after dozens of patches.

The debugging narrative: The assistant had spent segments 15 and 16 debugging incoherent output. The root cause was a tensor parallelism sharding mismatch in the kv_b_proj weights and a buffer initialization bug in the Triton MLA kernel. By the time this message is written, those bugs are fixed, and the model is producing coherent, semantically meaningful output. The tool call test is the strongest evidence yet that the model is working correctly.

The performance narrative: The assistant had previously measured ~55-57 tok/s single-request throughput. Tool calling adds a small overhead — the model must process the tool definitions in the system prompt and decide whether to invoke — but the test does not measure this overhead. The created timestamp and the response time are not analyzed here, but the fact that the assistant does not comment on slowness suggests the tool-calling latency is acceptable.

The deployment narrative: The service is running as a systemd unit with Restart=on-failure, ExecStartPre cleanup scripts, and tuned NCCL environment variables. The tool calling test proves the service is not just running but functioning correctly. This is the final validation before the deployment can be considered production-ready.

Conclusion

Message [msg 2073] is a deceptively simple validation test that carries the weight of an entire deployment journey. In one curl command and one JSON response, it proves that a 744-billion-parameter model, quantized to GGUF format, patched into vLLM through custom loader code, running across eight PCIe-interconnected GPUs with a custom Triton attention kernel, can correctly understand a function definition, decide to invoke it, extract the relevant parameter from a user query, format the arguments as JSON, and return them through a standard OpenAI-compatible API. The message is a testament to the power of systematic debugging, the importance of data-driven configuration decisions, and the satisfaction of seeing all the pieces finally work together.