The Pivot Point: Researching GGUF Serving Options After Abandoning the NVFP4 Path

Introduction

In the long arc of optimizing GLM-5-NVFP4 inference on a cluster of RTX PRO 6000 Blackwell GPUs, message [msg 1483] marks a decisive inflection point. After weeks of deep kernel engineering—profiling FP4 GEMMs, patching FlashInfer attention backends, implementing gather-then-cast optimizations, and achieving a respectable 29% throughput improvement—the entire trajectory shifts. The user has decided to abandon the NVFP4 quantization path entirely, pointing instead toward unsloth's GGUF quantizations of GLM-5. Message [msg 1483] is the assistant's first response to this strategic pivot: two parallel web searches aimed at answering the two most urgent questions that the new direction raises. What serving engine can handle a multi-hundred-gigabyte GGUF model across eight GPUs? And how does llama.cpp, the reference GGUF runtime, perform with GLM-5 in a multi-GPU configuration?

This article examines that single message in depth: the reasoning that motivated it, the assumptions embedded in the search queries, the knowledge required to interpret the results, and the thinking process visible in the assistant's choice of research targets. It is a message that contains no code, no bash commands, no patches—only information gathering—yet it is one of the most consequential messages in the entire session, because the answers it seeks will determine the technical architecture of the deployment for the remainder of the project.

The Message Quoted

The subject message consists of two parallel web search tool calls, each with its results displayed inline:

[assistant] [exa_web_search_exa] {"numResults":5,"query":"llama.cpp GLM-5 GGUF multi-GPU tensor split performance benchmark 2026"}
Title: Run GLM-5 Locally For Agentic Coding - DataCamp
Published Date: 2026-02-13T22:59:19.352Z
URL: https://www.datacamp.com/tutorial/run-glm-5-locally
Text: Run GLM-5 Locally For Agentic Coding | DataCamp
[Skip to main content] 
# Run GLM-5 Locally For Agentic Coding
Run GLM-5, the best open-weight AI model, on a single GPU with llama.cpp, and connect it to Aider to turn it into a powerful local coding agent.
List
Feb 13, 2026 · 8 min read
Contents
## GroupTraining more people?
Get your team ...
[exa_web_search_exa] {"numResults":5,"query":"sglang GGUF support GLM-5 serving engine comparison vllm llama.cpp 2026"}
Title: [Feature] How to serve GGUF model?
Author: sgl-project
Published Date: 2024-11-06T00:00:00.000Z
URL: https://github.com/sgl-project/sglang/issues/1937
Text: [Feature] How to serve GGUF model? ·Issue #1937 ·sgl-project/sglang ·GitHub
[Skip to content] 
## Navigation Menu
Toggle navigation
[] 
[Sign in] 
Appearance settings
Search or jump to...
# Search code, resources, users, issues, pull requests...

The message is deceptively simple: two search queries, two results. But the strategic weight behind these queries is enormous, and understanding why these particular questions were asked at this particular moment requires reconstructing the full context of the session.

Context: The Road to the Pivot

To understand message [msg 1483], one must understand what came immediately before it. The assistant had just completed a grueling diagnostic exercise spanning multiple segments of the conversation. In segment 10, the assistant computed the theoretical maximum single-stream performance of the GLM-5-NVFP4 model on the Blackwell GPUs, performed a system audit, upgraded the kernel to 6.14.11, fixed post-reboot CUDA issues, and built diagnostic tools to identify the primary bottleneck. The conclusion was stark: FP4 GEMM kernel overhead was the dominant factor limiting throughput.

In segment 11, the assistant drilled deeper. Using a torch profiler trace on the live sglang server, the assistant discovered that 69% of decode time—64.6 milliseconds per step—was being consumed by aten::copy_ / unrolled_elementwise_kernel, the operation that casts the KV cache from FP8 to BF16 on every layer for the entire 495,000-token pool. This was the fundamental architectural bottleneck: the FlashInfer MLA attention backend, as implemented for GLM-5's architecture, stored the KV cache in FP8 but the attention computation required BF16, forcing a full-pool cast on every decode step. Moving approximately 857 MB per layer per step, this cast dwarfed all other sources of latency.

The assistant implemented a clever gather-then-cast patch that only cast the active KV entries for the current request rather than the entire pool. This achieved a 29% improvement (from 10.5 to 13.5 tok/s, TPOT from 95.6 ms to 74.1 ms). But the user, after seeing the diagnostic evidence and understanding the architectural limitation, made a strategic decision: abandon the NVFP4 quantization path entirely. In message [msg 1479], the user directed the assistant toward unsloth's GLM-5 GGUF quantizations on Hugging Face, listing options like IQ4_XS (403 GB), Q4_K_S (428 GB), UD-Q4_K_XL (431 GB), and others, asking which would be best for coding/agent tasks and fit the system constraints.

The assistant's response in [msg 1480] acknowledged the decision ("Good call") and began evaluating the options. But before committing to a specific quantization or download strategy, the assistant needed to answer a prerequisite question: what serving engine would be used to run the GGUF model? This is the question that message [msg 1483] seeks to answer.## The Two Research Questions: Why These Queries?

The assistant issues two parallel web searches. The first query is: "llama.cpp GLM-5 GGUF multi-GPU tensor split performance benchmark 2026". The second is: "sglang GGUF support GLM-5 serving engine comparison vllm llama.cpp 2026". These are not random queries; they reveal a precise set of assumptions and concerns.

Query 1: llama.cpp Multi-GPU Performance

The first query targets llama.cpp specifically. This makes sense: llama.cpp is the reference implementation for GGUF inference, the runtime that popularized the GGUF format. It supports multi-GPU inference through tensor splitting (the --tensor-split flag), distributing model layers across GPUs. For a model the size of GLM-5—even at 4-bit quantization, the files range from 403 GB (IQ4_XS) to 473 GB (Q4_1)—multi-GPU support is not optional; it is mandatory. No single GPU in the system has enough VRAM (96 GB per GPU × 8 GPUs = 768 GB total, but the model must fit across them with overhead for KV cache and activations).

The query includes the terms "multi-GPU tensor split" and "performance benchmark 2026," indicating the assistant is looking for real-world measurements, not just theoretical capability. The assistant wants to know: does llama.cpp scale well across multiple GPUs for GLM-5 specifically? Are there known issues with the model's Mixture-of-Experts architecture in llama.cpp's multi-GPU path? What throughput can be expected?

The search returns a DataCamp tutorial titled "Run GLM-5 Locally For Agentic Coding," published February 13, 2026—just days before this conversation. This is highly relevant and timely. The article describes running GLM-5 on a single GPU with llama.cpp and connecting it to Aider for coding agent tasks. However, the article focuses on single-GPU deployment, not the multi-GPU tensor-split configuration that this system requires. The assistant would need to dig deeper to find multi-GPU benchmarks.

Query 2: Serving Engine Comparison

The second query is broader and more strategic: "sglang GGUF support GLM-5 serving engine comparison vllm llama.cpp 2026". This reveals that the assistant is not assuming llama.cpp will be the serving engine. The user explicitly rejected llama.cpp in a subsequent message (not yet visible at this point in the conversation), calling it "not a proper inference engine." But at the time of message [msg 1483], the assistant is still evaluating options objectively.

The query targets three engines: sglang (the current serving framework), vLLM (a popular alternative with strong multi-GPU support), and llama.cpp (the reference GGUF runtime). The assistant is asking: which of these engines supports GGUF models? And specifically, which supports GLM-5 in GGUF format?

The search returns a GitHub issue from the sglang project (issue #1937, dated November 2024) titled "How to serve GGUF model?" This is a telling result. The fact that someone needed to ask "how to serve GGUF model" in the sglang issue tracker suggests that GGUF support in sglang is not native or well-documented. The issue is from 2024, and by early 2026 the situation may have changed, but the assistant now has a data point suggesting that sglang's GGUF support is experimental or requires workarounds.

Assumptions Embedded in the Search Strategy

The assistant's search strategy reveals several assumptions:

  1. GGUF is the right format to pursue. The assistant does not question the user's decision to pivot to GGUF. The searches are framed entirely around "how do we serve GGUF," not "should we serve GGUF." This is appropriate—the user made a clear directive—but it means the assistant implicitly accepts that GGUF's advantages (wider ecosystem, potentially better quantization quality via Unsloth Dynamic 2.0) outweigh the costs of abandoning the entire NVFP4 infrastructure built over weeks.
  2. Multi-GPU is non-negotiable. Both queries assume multi-GPU deployment. The first query explicitly includes "multi-GPU tensor split," and the second compares serving engines that all support multi-GPU. The assistant correctly recognizes that a 400+ GB model cannot run on a single 96 GB GPU.
  3. The serving engine matters as much as the quantization. The assistant does not simply search for "how to download GLM-5 GGUF." It immediately jumps to serving engine compatibility. This reflects a sophisticated understanding that the quantization format is only half the equation—the runtime's ability to efficiently schedule the model across GPUs, manage KV cache, and handle the MoE architecture is equally critical.
  4. Performance data exists. The assistant assumes that by early 2026, enough people have deployed GLM-5 via GGUF that benchmarks and tutorials exist. This is a reasonable assumption given the model's popularity, and the DataCamp tutorial confirms it.

What the Assistant Did Not Know (Yet)

At the time of message [msg 1483], the assistant does not yet know several critical facts that will emerge in subsequent messages:

The Thinking Process Visible in the Message

Although the message contains only tool calls and their results, the thinking process is visible in the query construction. The assistant is reasoning through a decision tree:

  1. Branch 1: Can we use llama.cpp? The first query tests this branch. If llama.cpp has good multi-GPU performance with GLM-5 GGUF, it becomes the default choice—it's the most mature GGUF runtime, has the widest quantization support, and is actively maintained. The DataCamp result is encouraging but doesn't answer the multi-GPU question.
  2. Branch 2: Can we use sglang? The second query tests whether the current serving stack can be repurposed. If sglang supports GGUF natively, the assistant can reuse the existing infrastructure (server configuration, benchmarking scripts, monitoring). The GitHub issue result is discouraging—it suggests GGUF support in sglang is not straightforward.
  3. Branch 3: Can we use vLLM? vLLM is included in the second query as a comparison point. It has strong multi-GPU support and a growing GGUF backend, but compatibility with GLM-5's MoE architecture in GGUF format is uncertain. The assistant is effectively running a feasibility study in parallel, using web search as the tool. The two queries cover the most likely serving engines, and the results will inform the next steps: which engine to test, what configuration to use, and whether additional patching or workarounds will be needed.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message [msg 1483], a reader needs:

Output Knowledge Created by This Message

Message [msg 1483] produces two concrete pieces of knowledge:

  1. A reference to a real-world GLM-5 deployment. The DataCamp tutorial confirms that GLM-5 can be run with llama.cpp, that it works for coding agent tasks (via Aider), and that the community is actively using this combination. This is positive signal for the llama.cpp branch of the decision tree.
  2. A data point about sglang's GGUF support. The GitHub issue #1937 suggests that GGUF support in sglang is not native and may require workarounds. This weakens the sglang branch of the decision tree, making it less likely that the existing serving infrastructure can be reused without significant modification. More importantly, the message creates negative knowledge: it rules in certain directions and rules out others. The assistant now knows that llama.cpp is a viable path (pending multi-GPU benchmarks), that sglang's GGUF support is questionable, and that further investigation into vLLM's GGUF compatibility is warranted. This shapes the entire subsequent conversation, which will involve downloading the GGUF, merging shards, testing vLLM loading, and ultimately deploying the model.

Conclusion

Message [msg 1483] is a quiet but pivotal moment in the GLM-5 optimization saga. It contains no dramatic code changes, no breakthrough performance numbers, no clever algorithmic insights. It is simply two web searches. But those searches represent the moment when the assistant accepted the user's strategic pivot and began the work of rebuilding the deployment from scratch around a new quantization format. The queries reveal the assistant's mental model: a decision tree with three branches (llama.cpp, sglang, vLLM), evaluated against the constraints of multi-GPU deployment and GLM-5's architecture. The results provide directional signal—positive for llama.cpp, negative for sglang—that will guide the next phase of the project.

In the broader narrative of the opencode session, this message is the bridge between two eras: the NVFP4 era of kernel-level optimization and the GGUF era of serving engine evaluation. It is the moment when the assistant stops asking "how do we make this quant faster?" and starts asking "how do we serve this new quant at all?" The answers to those questions will determine the final architecture of the GLM-5 deployment, making [msg 1483] one of the most strategically important messages in the entire conversation.