The GGUF Frontier: Engineering vLLM Support for GLM-5 from First Principles

Introduction

In the sprawling, multi-week campaign to deploy the GLM-5 744-billion-parameter Mixture-of-Experts model on 8 NVIDIA RTX PRO 6000 Blackwell GPUs, Segment 12 represents a decisive inflection point. After exhaustive profiling revealed that the NVFP4 quantization path on sglang was fundamentally bottlenecked — with 69.3% of decode time consumed by an unavoidable FP8-to-BF16 KV cache cast — the user made a strategic pivot to GGUF UD-Q4_K_XL quantization on vLLM. But that pivot immediately encountered a wall: neither the transformers library nor the gguf-py package supported the glm-dsa architecture used by GLM-5. The user's response to this dead end was unambiguous and transformative: "E. add this gguf support to vllm" ([msg 1518]). With those five words, the effort shifted from a deployment exercise into a software development project — one that would require writing a comprehensive patch for vLLM's gguf_loader.py, installing bleeding-edge versions of transformers and gguf-py from source, and enabling a deployment configuration that had never been successfully achieved before.

This article synthesizes the work captured across the messages in Segment 12 — a journey through strategic decision-making, parallel codebase reconnaissance, dependency resolution, tensor name mapping, and patch architecture design. It represents a masterclass in AI-assisted systems engineering at the frontier of what existing tools support.

The Strategic Pivot: Abandoning NVFP4 for GGUF

The pivot to GGUF was not a simple switch. It was the culmination of weeks of profiling and optimization work that had revealed fundamental architectural limitations in the NVFP4 path. The assistant had already freed disk space by deleting the NVFP4 model files ([msg 1515]), installed vLLM nightly (0.16.0rc2.dev313), and begun downloading the 431 GB GGUF split files from Hugging Face ([msg 1527]). But before the download could complete, the assistant discovered the critical blocker: vLLM's GGUF support depends on transformers to parse the GGUF file's metadata header into a HuggingFace model configuration, and transformers simply did not know about the deepseek2 or glm-dsa architectures.

The assistant's initial investigation ([msg 1517]) revealed a devastating picture. The GGUF_CONFIG_MAPPING dictionary in transformers.integrations.ggml.py — which maps GGUF architecture names to HuggingFace config classes — contained entries for architectures like bloom, falcon, gemma, llama, mistral, phi3, qwen2, and qwen2_moe, but nothing for any DeepSeek-derived architecture. The deepseek2 GGUF architecture simply did not exist in the mapping. Multiple GitHub issues confirmed the pattern: every attempt to load a DeepSeek GGUF model on vLLM resulted in the same error: ValueError: GGUF model with architecture deepseek2 is not supported yet.

But the research also revealed a critical nuance that would shape the entire patch strategy: the blocker was solely in transformers. vLLM's own GGUFModelLoader already had manual weight mappings for DeepSeek architectures, including expert weight sideloading logic for fused gate_up_proj tensors. The GlmMoeDsaForCausalLM model class already existed as a stub in vLLM, inheriting from DeepseekV2ForCausalLM. The missing piece was the transformers config mapping — without it, the GGUF file couldn't even be opened. This discovery transformed the problem from "build a complete GGUF loading pipeline from scratch" to "add a single architecture entry to the config mapping and wire up the tensor name translations."

The Five-Word Decision That Changed Everything

Message [msg 1518] is one of the most consequential messages in the entire session. The user's five-word response — "E. add this gguf support to vllm" — rejected four alternative paths and endorsed a massive engineering undertaking. The assistant had presented five options:

• Option A: vLLM + FP8 — risked VRAM exhaustion at 860 GB needed vs. 768 GB available
• Option B: Return to sglang + NVFP4 — would revisit the previously working path but still faced the KV cache cast bottleneck
• Option C: sglang + FP8 — shared the same VRAM concerns as Option A
• Option D: llama.cpp GGUF — the simplest path (it already worked and was what Unsloth recommended), but the user had already ruled it out
• Option E: Patch transformers/vLLM to add DeepSeek GGUF support — the most ambitious option The user's choice reveals several important assumptions. First, the user valued GGUF's memory efficiency above all other considerations — the UD-Q4_K_XL quantization at 431 GB left significant headroom for KV cache compared to the 860+ GB required by FP8. Second, the user had confidence in the assistant's ability to modify vLLM's source code, having already demonstrated deep expertise in CUDA kernel optimization and sglang internals across the preceding segments. Third, the user's phrasing — "add this gguf support to vllm" — suggested a belief that the gap was smaller than the assistant estimated. The word "this" refers to the specific GLM-5 GGUF model, not a general DeepSeek GGUF support framework. This turned out to be partially correct: the assistant later discovered that vLLM already had manual weight mappings for DeepSeek architectures, and the gguf-py library from llama.cpp already defined LLM_ARCH_GLM_DSA with a complete tensor name map. The blocker was primarily in transformers, not in vLLM itself.

Parallel Reconnaissance: Understanding Three Codebases Simultaneously

With the user's directive in hand, the assistant launched a three-pronged parallel research effort ([msg 1520]) that would illuminate the entire GGUF loading pipeline. Three subagent tasks were dispatched simultaneously, each exploring a different codebase:

Task 1: Study transformers GGUF mapping. This task investigated how the HuggingFace transformers library handles GGUF files, specifically the GGUF_CONFIG_MAPPING dictionary in ggml.py that maps GGUF metadata keys to HuggingFace config parameters, and the GGUF_CONFIG_DEFAULTS_MAPPING that provides default values. The goal was to identify exactly what code paths needed to be modified to add support for the deepseek2 or glm_moe_dsa GGUF architecture. The task discovered that the mapping pipeline has three stages: config mapping (GGUF metadata keys to HF config params), tensor name mapping (GGUF tensor names to HF tensor names), and weight loading (actual tensor data transfer). The config mapping stage was the blocker — without an entry for deepseek2 or glm-dsa in GGUF_CONFIG_MAPPING, the entire pipeline fails before reaching the tensor name mapping stage.

Task 2: Study vLLM GGUF loader code. This task examined vLLM's gguf_loader.py, specifically the GGUFModelLoader class and its _get_gguf_weights_map() method. The goal was to determine whether vLLM already had the infrastructure to handle DeepSeek/GLM model types, or whether new weight mappings would need to be written. The task discovered that vLLM's GGUF loader does NOT use transformers' ggml.py — it has its own _get_gguf_weights_map() method that builds a dictionary mapping GGUF tensor names to HuggingFace tensor names. When using --hf-config-path zai-org/GLM-5 (external config) and --tokenizer zai-org/GLM-5 (external tokenizer), the transformers GGUF config mapping is bypassed entirely. vLLM loads the HF config directly, creates a dummy HF model via AutoModelForCausalLM.from_config(), and builds its own GGUF-to-HF tensor name mapping. This was the critical insight: the blocker was not in vLLM's code but in its upstream dependency.

Task 3: Study GLM-5 GGUF tensor names. This task explored the actual tensor names inside the GLM-5 GGUF file using the installed gguf-py Python package. The goal was to produce a complete mapping from GGUF tensor names (like blk.0.attn_k_b and blk.0.attn_v_b) to HuggingFace tensor names (like model.layers.0.kv_b_proj), accounting for any structural differences introduced during the quantization process. The task discovered that the glm-dsa architecture in gguf-py defines 38 tensor types, including specialized entries like INDEXER_ATTN_K, INDEXER_ATTN_Q_B, NEXTN_EH_PROJ, and NEXTN_EMBED_TOKENS that are unique to the DSA (Dynamic Sparse Attention) mechanism.

The research yielded several critical discoveries. The most important was that the blocker was solely in transformers — vLLM already had manual weight mappings for DeepSeek architectures, and GlmMoeDsaForCausalLM already existed as a stub inheriting from DeepseekV2ForCausalLM. The GGUF architecture name was glm-dsa, not deepseek2, meaning a simple deepseek2 entry in transformers' config mapping wouldn't suffice — the patch needed to handle the glm-dsa architecture specifically. The KV projection tensors were split in GGUF format: kv_b_proj in the HuggingFace model became separate attn_k_b and attn_v_b tensors that must be reassembled during loading. Expert weights used fused gate_up_proj format, and the e_score_correction_bias tensor required manual mapping.

Building the Foundation: Installing Bleeding-Edge Dependencies

With the research complete, the assistant began the implementation phase by resolving the dependency chain from the ground up. The first step was installing gguf-py from llama.cpp source ([msg 1539]). The PyPI version (0.17.1) lacked the glm-dsa architecture entirely, but the llama.cpp git HEAD had added it as architecture key 73. The assistant installed directly from source:

uv pip install --python ~/ml-env/bin/python3 \
  "git+https://github.com/ggml-org/llama.cpp.git#subdirectory=gguf-py"

This was a deliberate move to bypass the PyPI release cycle. The gguf-py package on PyPI is a snapshot of the llama.cpp repository at some point in time. If the glm-dsa architecture was added to llama.cpp after that snapshot, the PyPI package would not include it. By installing directly from the git repository's HEAD, the assistant got the absolute latest definitions — including the glm-dsa architecture with its complete tensor name map.

The next step was verifying that the installed gguf-py could correctly map every GLM-5 tensor name ([msg 1542]). The assistant wrote a systematic test that covered all 26 tensor categories: attention projections (q_a, q_b, kv_a, kv_b, o), attention norms, layer norms, MLP projections, router weights, expert weights (with fused gate_up_proj), shared experts, expert score correction bias, DSA indexer tensors (wq_b, wk, k_norm, weights_proj), and final layers (embed_tokens, norm, lm_head). Every single HF tensor name mapped correctly to its GGUF counterpart.

The test also confirmed a critical detail: kv_b_proj maps to blk.0.attn_kv_b in the tensor name map, but the actual GGUF file stores it as two separate tensors (attn_k_b and attn_v_b). This meant the vLLM loader would need to look up both attn_k_b and attn_v_b in the GGUF file and concatenate them to reconstruct the HF kv_b_proj weight.

The Tensor Mapping Puzzle: Decoding GLM-5's GGUF Structure

One of the most technically challenging aspects of the patch was understanding the complete tensor name mapping for GLM-5's GGUF representation. The glm-dsa architecture in gguf-py defined 38 tensor types ([msg 1543]), including specialized entries like INDEXER_ATTN_K, INDEXER_ATTN_Q_B, NEXTN_EH_PROJ, and NEXTN_EMBED_TOKENS that are unique to the DSA (Dynamic Sparse Attention) mechanism.

The assistant systematically verified each tensor category. The standard attention projections mapped cleanly: q_a_proj -> blk.N.attn_q_a, q_b_proj -> blk.N.attn_q_b, kv_a_proj_with_mqa -> blk.N.attn_kv_a_mqa, kv_b_proj -> blk.N.attn_kv_b, o_proj -> blk.N.attn_output. The layer norms mapped to blk.N.attn_norm and blk.N.ffn_norm. The dense MLP components mapped to blk.N.ffn_gate, blk.N.ffn_up, blk.N.ffn_down.

But the MoE components revealed the complexity. The router (gate) mapped to blk.N.ffn_moe_gate. Expert weights mapped to blk.N.ffn_moe_expt.N.ffn_gate/up/down. Shared experts mapped to blk.N.ffn_moe_shared_gate/down. The expert score correction bias mapped to blk.N.ffn_moe_gate.ffn_moe_e_score_corr. And the DSA indexer tensors mapped to blk.N.attn_indexer_wq_b, blk.N.attn_indexer_wk, blk.N.attn_indexer_k_norm, blk.N.attn_indexer_weights_proj.

The assistant discovered that 150 parameters were unmapped by gguf-py's automatic name mapping — exactly 75 MoE layers × 2 params each (experts.gate_up_proj and gate.e_score_correction_bias). These would need manual mapping in the vLLM loader, following the pattern already established for DeepSeek architectures.

The Critical Discovery: KV Projection Split

The most technically subtle discovery was the kv_b_proj split. In the HuggingFace reference model, the key-value projection exists as a single combined weight matrix: model.layers.N.self_attn.kv_b_proj.weight. But during GGUF conversion via llama.cpp's convert_hf_to_gguf.py, this tensor is split into two separate tensors: attn_k_b (the key portion) and attn_v_b (the value portion). The GGUF file on disk contains these split tensors, not the combined one.

The assistant traced the exact transformation in the llama.cpp conversion script ([msg 1559], [msg 1560]). The conversion performs a complex sequence: reshape [n_head*(qk_nope+v_head), kv_lora_rank] → [n_head, qk_nope+v_head, kv_lora_rank]; split into k_b [n_head, qk_nope, kv_lora_rank] and v_b [n_head, v_head, kv_lora_rank]; k_b is transposed: [n_head, kv_lora_rank, qk_nope]; v_b permuted: [kv_lora_rank, v_head, n_head]. The reversal of these operations is non-trivial and depends on the exact sequence of reshape, split, transpose, and permute operations applied during conversion.

This discovery directly informed the patch. The vLLM loader would need to buffer the attn_k_b and attn_v_b tensors, concatenate them in the correct dimension, and yield a single combined kv_b_proj weight. The assistant initially considered a simple torch.cat([k_b, v_b], dim=0) approach, but the complex transformation chain meant the reassembly might require more careful handling. The dequantized GGUF tensors come back in their stored shapes — after dequantization they should be 2D: k_b [n_head*qk_nope, kv_lora_rank] (12288, 512) and v_b [n_head*v_head, kv_lora_rank] (16384, 512). A simple torch.cat([k_b, v_b], dim=0) gives [28672, 512] which matches kv_b_proj.weight shape [n_head*(qk_nope+v_head), kv_lora_rank]. But the transpose and permute during conversion mean the simple concatenation may not produce the correct interleaving — the k and v portions within each head need to be interleaved correctly. This required careful verification.

The Patch Architecture: Designing the Solution

With the tensor mappings verified, the assistant turned to designing the actual patch for vLLM's gguf_loader.py ([msg 1557]). The patch strategy had two main components:

Part 1: _get_gguf_weights_map() — Add glm_moe_dsa model_type handling with:

• Map model_type to "glm-dsa" for gguf-py architecture identification
• Add manual expert weight mappings (following the pattern established for deepseek_v2/deepseek_v3)
• Add manual e_score_correction_bias mappings
• Add manual kv_b_proj split handling (map both attn_k_b and attn_v_b to kv_b_proj)
• Add sideload patterns for expert parameters Part 2: load_weights() or _get_weights_iterator() — Handle the kv_b_proj reassembly from attn_k_b + attn_v_b The critical challenge was the kv_b_proj reassembly. The GGUF weight iterator yields one tensor at a time, but the patch needed to combine two tensors into one. The assistant's reasoning evolved in real time: initially considering overriding the iterator to buffer tensors, then pivoting to a simpler shard-based approach where each split tensor maps to the same HF parameter name as a "shard," using the existing stacked_params_mapping mechanism. This mirrored how vLLM already handled gate_up_proj being composed of gate_proj and up_proj for DeepSeek architectures. The assistant also identified an auto-mapping conflict that needed resolution. The gguf-py name map maps kv_b_proj → attn_kv_b, but the actual GGUF file has attn_k_b and attn_v_b (NOT attn_kv_b). The auto-mapping would create an entry blk.N.attn_kv_b.weight → model.layers.N.self_attn.kv_b_proj.weight, but this GGUF tensor doesn't exist in the file. The solution was elegant: manual mappings are added FIRST to gguf_to_hf_name_map, then auto-mapping adds entries that DON'T already have their HF name as a value. So the manual attn_k_b→kv_b_proj and attn_v_b→kv_b_proj mappings would be in the map, and the auto-mapping's attn_kv_b→kv_b_proj entry would simply not match any GGUF tensor in the file. No weight would be yielded for it, and the real weights would come from attn_k_b and attn_v_b.

The Silent Pivot: Consolidation and Planning

At a critical juncture in the investigation, the user sent an empty message ([msg 1561]) — a conversation_data tag with nothing between its markers. This empty message, whether intentionally blank or simply not captured, served as a checkpoint request. The assistant responded with one of the most comprehensive summary documents in the entire session ([msg 1562]), covering:

• Goal: Deploy GLM-5 GGUF on vLLM
• Discoveries: Hardware specs, software versions, the critical finding about why vLLM+GGUF doesn't work out of box, the architecture of vLLM's GGUF loading pipeline, GLM-5 GGUF specifics including the exact tensor name mapping
• Accomplished: A checklist of completed work (vLLM nightly installed, transformers upgraded, gguf-py installed from source, tensor mappings verified, dummy model created)
• In Progress: The 431 GB GGUF download running in background
• Not Yet Done: Six remaining tasks including writing the gguf_loader.py patch, handling the kv_b_proj split reassembly, mapping expert weights, and testing the loaded model
• Patch Plan Details: A complete, line-by-line specification of what needs to be added to gguf_loader.py, including the exact Python code for expert weight mappings, e_score_correction_bias handling, kv_b_proj split reassembly, and sideload patterns This summary served as the blueprint for the next phase of work. The assistant had transitioned from research to execution, from understanding the problem to building the solution.

The Dependency Tightrope: Version Management

Throughout this effort, the assistant walked a tightrope of dependency management. The vLLM nightly build (0.16.0rc2.dev313) had its own pinned dependencies, including transformers v4.57.6. But the assistant needed transformers v5.3.0.dev0 from git HEAD to get the latest GGUF integration code. And the gguf-py library needed to come from llama.cpp source rather than PyPI.

The assistant resolved these conflicts by installing each component from its bleeding-edge source while ensuring compatibility. The transformers upgrade from v4.57.6 to v5.3.0.dev0 was necessary because the GGUF config mapping code in ggml.py had been significantly reorganized between versions. The gguf-py installation from llama.cpp source was necessary because the PyPI version (0.17.1) predated the glm-dsa architecture addition. The vLLM nightly build was necessary because earlier versions lacked the GlmMoeDsaForCausalLM stub class.

This dependency resolution was not without risk. Each bleeding-edge component could introduce regressions or API incompatibilities. But the assistant's systematic verification approach — testing each component independently before integrating — mitigated these risks. The tensor mapping verification ([msg 1542]) was particularly important because it confirmed that the bleeding-edge gguf-py actually worked correctly before the assistant committed to using it.

The GGUF Download: A 431 GB Commitment

While the assistant worked on research and patching, the 431 GB GGUF download continued in the background ([msg 1527]). The download was launched with nohup to survive SSH disconnections, using HF_HUB_ENABLE_HF_TRANSFER=1 for fast parallel transfers, and logging to /tmp/gguf_download.log for monitoring.

The download served as both a practical necessity and a strategic commitment. By starting the download before the patches were written, the assistant ensured that when the patches were ready, the data would be too. The alternative — patch first, then download — would leave the assistant waiting at the end. This parallelization minimized total wall-clock time.

But the download also represented a bet on the feasibility of the patch work. 431 GB of disk space, bandwidth, and time would be wasted if the patches turned out to be impossible. The assistant's confidence in the patch strategy — based on the research findings that the blocker was solely in transformers and that vLLM already had the infrastructure for DeepSeek architectures — made this a calculated risk.

Conclusion: From Research to Implementation Readiness

The work captured in Segment 12 represents a complete arc from strategic pivot to implementation readiness. The assistant:

1. Discovered the blocker: Neither transformers nor gguf-py supported the glm-dsa architecture
2. Received the directive: The user chose Option E — "add this gguf support to vllm"
3. Conducted parallel research: Three subagent tasks explored transformers, vLLM, and gguf-py simultaneously
4. Resolved dependencies: Installed vLLM nightly, transformers from git HEAD, and gguf-py from llama.cpp source
5. Verified tensor mappings: Confirmed all 26+ tensor categories map correctly between HF and GGUF naming conventions
6. Discovered critical details: The kv_b_proj split, fused expert weights, and 150 unmapped parameters
7. Designed the patch strategy: A two-part plan for modifying gguf_loader.py with expert weight sideloading, KV split reassembly, and indexer tensor mapping
8. Started the download: 431 GB of GGUF split files downloading in the background The assistant is now positioned to apply and test the patch once the download completes. The foundation is solid: the tensor mappings are verified, the dependency chain is resolved, and the patch architecture is designed. What remains is the implementation — writing the actual Python code that will enable vLLM to load GLM-5 in its GGUF-quantized form for the first time. This segment exemplifies the power of structured research, parallel exploration, and deep understanding of software architecture in AI-assisted development. The assistant didn't just write code — it understood the full dependency graph of the problem, verified each assumption with concrete evidence, and designed a surgical patch that fits within the existing vLLM architecture. It's a masterclass in how to approach a complex systems integration problem when the existing tools simply don't support what you're trying to do. The broader significance of this work extends beyond GLM-5. The patterns established here — the parallel codebase reconnaissance, the bottom-up dependency resolution, the systematic tensor mapping verification, the strategic parallelization of download and development — constitute a template for any situation where a cutting-edge model needs to be deployed on infrastructure that the existing software ecosystem doesn't yet support. As AI models continue to evolve faster than the tools that serve them, this kind of engineering will become increasingly essential.

References

[1] The Great GGUF Pivot: Engineering vLLM Support for GLM-5 from Scratch — Chunk article covering the parallel research, dependency resolution, and patch design work in Segment 12.