The Pivot Point: Assessing vLLM + GGUF Compatibility for GLM-5 on Blackwell GPUs

In the sprawling, multi-session journey to deploy GLM-5—a 744-billion-parameter Mixture-of-Experts model with 256 experts—on a cluster of 8 NVIDIA RTX PRO 6000 Blackwell GPUs, few messages capture the tension of a strategic pivot as vividly as [msg 1506]. This message sits at a critical inflection point in the conversation: the team has just abandoned the NVFP4 quantization path after discovering that 69.3% of decode time was wasted on a full-pool KV cache cast from FP8 to BF16 (see [msg 1497]). The user has made the decisive call to switch to GGUF UD-Q4_K_XL quantization running on vLLM. But before the assistant can even begin downloading the 431 GB of GGUF split files, it must answer a fundamental question: will this new path actually work?

Message 1506 is the moment where the assistant confronts that question head-on, and it reveals the careful, research-driven thinking that characterizes the entire session.

The Context of the Pivot

To understand why [msg 1506] was written, one must appreciate the magnitude of what came before. The NVFP4 path had consumed dozens of messages across multiple segments. The team had profiled the model with PyTorch's profiler, identified the KV cache cast bottleneck, implemented a gather-then-cast patch that improved single-stream throughput by 29% (from 10.5 to 13.5 tok/s), tested alternative attention backends (trtllm_mla and cutlass_mla, both incompatible with GLM-5), explored expert parallelism (EP8 crashed under load), and tried numerous other optimizations (MSCCLPP, piecewise CUDA graphs, OEA routing). The user's patience had run out, and the directive was clear: pivot to GGUF on vLLM.

But pivoting to a new inference engine and quantization format is not a simple matter of swapping flags. The assistant had to verify that vLLM's GGUF support actually works for GLM-5's architecture—a DeepSeekV3-derived design with Multi-head Latent Attention (MLA), 256 experts, and a complex tensor layout. And this verification is precisely what drives [msg 1506].

What the Message Actually Does

The message begins with a reflection on the previous web search results:

Important findings. There are known SM120 issues with vLLM, particularly with DeepSeek-style MoE models. Let me check the current state more carefully — the GGUF path may avoid some of the FP4/FP8 kernel issues since GGUF uses different quantization kernels.

This opening reveals the assistant's reasoning process. It has just read a GitHub issue about NVFP4 MoE kernels failing on SM120 (Blackwell) GPUs (from [msg 1505]). The assistant is connecting dots: if the FP4/FP8 kernels have SM120 bugs, switching to GGUF—which uses a fundamentally different quantization scheme (K-quants rather than NVFP4)—might sidestep those issues entirely. This is a reasonable hypothesis, but the assistant doesn't stop there. It immediately identifies a second, potentially more dangerous risk: even if the kernels work, does vLLM's GGUF loader even recognize GLM-5's architecture?

The assistant then takes two concrete actions. First, it runs an Exa web search with the query:

vLLM GLM-5 GGUF support DeepSeekV3ForCausalLM 744B MoE tensor parallel

This search is carefully crafted. It includes "DeepSeekV3ForCausalLM" because GLM-5 is derived from DeepSeek V3's architecture. It includes "744B" to narrow results to the correct model scale. It includes "MoE" and "tensor parallel" because those are the key deployment challenges. The assistant is not just asking "does GLM-5 work with vLLM"—it's asking a more precise question about architecture compatibility.

Second, the assistant runs a bash command to check the GPU state:

ssh root@10.1.230.174 'nvidia-smi --query-gpu=gpu_name,memory.free,memory.total --format=csv,noheader 2>/dev/null | head -2'

This returns:

NVIDIA RTX PRO 6000 Blackwell Server Edition, 97250 MiB, 97887 MiB NVIDIA RTX PRO 6000 Blackwell Server Edition, 97250 MiB, 97887 MiB

The GPU check is almost perfunctory—it confirms the two GPUs are available with ~97 GB free memory each. But its inclusion in this message is telling. The assistant is grounding its research in the physical reality of the hardware. No matter what the web searches reveal about software compatibility, the deployment must ultimately run on these specific Blackwell GPUs with their SM120 compute capability.

The Reasoning and Assumptions on Display

Message 1506 is a masterclass in structured uncertainty. The assistant knows it doesn't know whether GGUF will work, but it has a clear methodology for finding out. The reasoning proceeds through several layers:

Layer 1: Kernel-level risk assessment. The assistant has just learned that NVFP4 MoE kernels have SM120 bugs. It hypothesizes that GGUF quantization might avoid these because GGUF uses different kernels. This is a reasonable assumption—GGUF models use K-quant or IQ-quant kernels that are implemented at a lower level of abstraction than NVFP4-specific kernels. However, it's an assumption that could prove wrong if the GGUF loader itself has SM120 issues in its memory management or tensor operations.

Layer 2: Architecture-level risk assessment. The assistant recognizes that even if the kernels work, the GGUF loader must recognize GLM-5's architecture. This is a separate concern. The deepseek2 architecture (which GLM-5 is based on) was known to be unsupported in earlier versions of transformers' GGUF loader, with errors like ValueError: GGUF model with architecture deepseek2 is not supported yet. The assistant needs to verify whether this has been fixed.

Layer 3: The "current state" check. The phrase "Let me check the current state more carefully" reveals the assistant's awareness that the situation may have changed since the GitHub issues were filed. The issues about deepseek2 GGUF support were from early-to-mid 2025. By the time of this message (likely early 2026), transformers has reached version 5.2.0, and vLLM has nightly builds with GLM-5 recipe support. The assistant is careful not to assume that old issues reflect current reality.

Input Knowledge Required

To fully understand [msg 1506], the reader needs substantial background knowledge:

  1. The SM120 architecture: Blackwell GPUs use compute capability 12.0 (SM120), which is architecturally distinct from both the previous Hopper generation (SM90) and the data-center-focused Blackwell SM100. SM120 lacks TMEM, has only 100 KB shared memory per SM, and uses Ampere-era mma.sync instructions rather than SM100's tcgen05.mma. These differences mean that CUDA kernels optimized for SM90 or SM100 may fail or underperform on SM120.
  2. GGUF quantization: GGUF is a file format for quantized LLMs that uses group-wise quantization (K-quants, IQ-quants). Unlike NVFP4 (which uses NVIDIA's native FP4 format), GGUF quantization is implemented at the software level and doesn't depend on hardware FP4 support. This is why the assistant hopes GGUF might avoid SM120 kernel bugs.
  3. The DeepSeek V3 / GLM-5 architecture: GLM-5 uses Multi-head Latent Attention (MLA), 256 experts with 8 activated per token, and a complex tensor layout that includes fused gate_up_proj for experts, separate KV bias tensors, and e_score_correction_bias. The GGUF loader must correctly map all these tensors.
  4. vLLM's GGUF loading pipeline: vLLM doesn't parse GGUF files directly—it delegates architecture detection to HuggingFace's transformers library. The transformers GGUF config mapping must include the architecture name (e.g., glm-dsa or deepseek2) for vLLM to proceed. If transformers doesn't recognize the architecture, vLLM fails with an error before even attempting to load weights.

Output Knowledge Created

Message 1506 produces several pieces of knowledge:

  1. Confirmation of GPU state: The two Blackwell GPUs are healthy with 97 GB free memory each. This confirms the hardware is ready for the GGUF download and deployment.
  2. A refined risk model: The assistant has articulated two distinct risks—kernel-level SM120 compatibility and architecture-level GGUF loader support—and has a plan to investigate both.
  3. A specific search query for the next investigation: The Exa search will return results that reveal whether vLLM's GGUF loader supports the DeepSeek V3 architecture. This will directly inform the next steps.
  4. A decision framework: The message implicitly establishes a decision tree: if the GGUF loader supports GLM-5's architecture, proceed with download and deployment; if not, the team may need to patch vLLM's GGUF loader or reconsider the approach entirely.

The Broader Significance

What makes [msg 1506] noteworthy is not the information it contains—the GPU state check is trivial, and the web search hasn't returned yet—but the thinking process it reveals. The assistant is operating in a high-stakes environment where a wrong assumption could waste hours or days. Downloading 431 GB of GGUF split files only to discover that vLLM can't load them would be catastrophic. The assistant's methodical approach—checking kernel compatibility, architecture support, and hardware state before committing to the download—is precisely the kind of risk management that distinguishes effective technical work from guesswork.

The message also reveals the assistant's willingness to challenge its own assumptions. The hypothesis that "GGUF may avoid FP4/FP8 kernel issues" is immediately tempered by the recognition that "vLLM's GLM-5 support status" is a separate, equally important concern. This dual-track thinking—holding multiple hypotheses simultaneously and testing each independently—is a hallmark of rigorous engineering.

In the messages that follow [msg 1506], the assistant's fears are confirmed. The web search reveals that deepseek2 architecture GGUF is indeed not supported by vLLM's transformers-based GGUF loader, and MLA is explicitly incompatible with GGUF quantization. This leads to a major escalation: the user rejects all alternative paths (reverting to sglang, llama.cpp, or FP8) and directs the assistant to add GGUF support to vLLM directly—a task that consumes the remainder of the segment and involves writing a comprehensive patch for vLLM's gguf_loader.py.

Message 1506, then, is the canary in the coal mine. It's the moment when the assistant's careful research begins to reveal the cracks in the GGUF plan, setting the stage for the deep code modification work that follows. Without this message's disciplined questioning, the team might have blindly downloaded 431 GB of unusable model files and only discovered the incompatibility hours later. The message is a testament to the value of pausing to think before acting—even when the user has already made a strategic decision and the pressure is on to execute.