Mapping the Unmappable: Diagnosing GGUF Tensor Name Gaps for GLM-5 on vLLM

In the sprawling, multi-week effort to deploy the 431 GB GLM-5 model on an 8-GPU machine, message 1552 represents a quiet but pivotal diagnostic moment. The assistant had just pivoted from the failed NVFP4 quantization path toward a GGUF UD-Q4_K_XL deployment on vLLM, only to discover that neither transformers nor the installed gguf-py library supported the glm-dsa architecture used by GLM-5. After the user explicitly rejected all alternative paths—reverting to sglang, llama.cpp, or FP8—the assistant was directed to add GGUF support to vLLM directly. Message 1552 captures the precise moment when the assistant, having already installed the latest gguf-py from llama.cpp HEAD and verified that the glm-dsa architecture was present, drilled down into the remaining gaps between the HuggingFace model's state dictionary and the GGUF tensor name map. These gaps would determine exactly what code needed to be written in the vLLM patch.

The 150 Unmapped Parameters: Pattern Recognition in Action

The message opens with a concise analytical statement: "So there are exactly 150 unmapped params: 75 MoE layers × 2 params each (experts.gate_up_proj and gate.e_score_correction_bias). The experts.down_proj is mapped automatically." This conclusion is the result of the previous message's investigation (msg 1551), which had created a dummy HF model on meta device and iterated all 1,629 state dict keys, checking each against the glm-dsa tensor name map. The output had shown exactly 150 unmapped keys, all following a clear pattern: every MoE layer (layers 3 through 77) contributed two unmapped parameters.

The assistant's reasoning here is exemplary of systematic debugging. Rather than being alarmed by 150 unmapped parameters out of 1,629—roughly 9% of the total—the assistant immediately recognizes the pattern. GLM-5 has 78 layers total, with the first 3 being dense (no MoE) and layers 3 through 77 being MoE layers with 64 experts each. That's 75 MoE layers. Each MoE layer has two unmapped params: experts.gate_up_proj (the fused gate and up projection for all experts) and gate.e_score_correction_bias (a bias term for the router's score correction). The experts.down_proj is already mapped automatically by gguf-py, which means only the fused expert weights and the router bias need manual handling.

This pattern recognition is crucial because it tells the assistant exactly what the vLLM patch needs to handle for expert weights. The existing deepseek_v2 and deepseek_v3 model types in vLLM's GGUF loader already have manual mappings for expert weights—they sideload individual expert tensors from the GGUF file and assemble them into fused weights. The assistant can now see that the glm-dsa case will follow the same pattern, just with slightly different tensor names.

The kv_b_proj Split: A Subtler Discovery

Having confirmed the expert weight situation, the assistant turns to a second potential mismatch: the KV projection split. The GGUF format for glm-dsa defines both ATTN_KV_B (a single fused tensor) and ATTN_K_B/ATTN_V_B (split tensors). The question is: does the actual GGUF file use the fused format or the split format?

The assistant runs a targeted Python script to probe the gguf-py name map's reverse mapping. The results are revealing:

Input Knowledge Required to Understand This Message

To fully grasp what is happening in message 1552, a reader needs substantial context. First, one must understand the GGUF model format—a quantized model format used primarily by llama.cpp that stores tensors with architecture-specific names. Second, familiarity with the vLLM GGUF loader architecture is essential: vLLM creates a dummy HuggingFace model (via AutoModelForCausalLM.from_config()) and then maps GGUF tensor names to HF state dict keys using the gguf-py library's TensorNameMap. Third, knowledge of the GLM-5 model architecture is needed: it uses Multi-Head Latent Attention (MLA) with a DeepSeek-style MoE, including 64 experts per MoE layer, a shared expert group, and a DSA (Dynamic Sparse Attention) indexer mechanism. Fourth, one must understand the distinction between fused expert weights (gate_up_proj) and individual expert weights (gate_proj/up_proj)—the GGUF file stores individual expert tensors that must be fused during loading.

Output Knowledge Created

This message produces several critical pieces of knowledge that directly inform the vLLM patch. First, it confirms that the expert weight handling must follow the same pattern as deepseek_v2/deepseek_v3: manual sideloading of individual expert tensors from the GGUF file, with the gate_up_proj fusion happening during the load process. Second, it reveals the kv_b_proj split situation: the GGUF file stores separate attn_k_b and attn_v_b tensors that must be concatenated into the single kv_b_proj weight expected by the HF model. Third, it confirms that the e_score_correction_bias tensor (the router's score correction bias) is not automatically mapped and must be handled manually. Fourth, it validates that the experts.down_proj tensor IS automatically mapped, reducing the manual mapping burden.

The Thinking Process: Systematic, Data-Driven, and Incremental

The assistant's thinking process in this message is a masterclass in systematic debugging. Each step builds on the previous one, and each question answered generates new questions. The flow is:

  1. Observe: 150 params are unmapped (from msg 1551 output).
  2. Pattern-match: 75 MoE layers × 2 params each = 150. The unmapped params are gate_up_proj and e_score_correction_bias.
  3. Hypothesize: The kv_b_proj might also have a mapping issue—the GGUF might use split K/V tensors.
  4. Test: Write a Python script that probes the name map for both the fused and split KV tensor names.
  5. Analyze results: The name map supports both fused and split mappings, but the HF model only has the fused version. The GGUF file likely stores split tensors.
  6. Implication: The patch needs to handle reassembly of split K/V tensors. What's notable is what the assistant does NOT do. It does not jump to conclusions about the 150 unmapped params being a problem—instead, it recognizes them as a known pattern that matches existing vLLM code. It does not assume the GGUF file uses the fused attn_kv_b format—it explicitly checks for the split format. This cautious, evidence-based approach minimizes the risk of writing incorrect patch code.

Broader Significance

Message 1552 exemplifies the kind of deep, multi-layered diagnostic work that makes the difference between a brittle hack and a robust patch. The assistant is not just copying code from the deepseek_v2 handler and changing names—it is systematically verifying every tensor mapping, every potential split/merge operation, and every edge case. The discovery that kv_b_proj must be reassembled from split K/V tensors is the kind of detail that, if missed, would cause silent corruption or crashes at load time.

This message also illustrates the power of the "dummy model on meta device" technique. By creating a lightweight HF model without allocating actual memory (using torch.device("meta")), the assistant can iterate the full state dict and compare every key against the gguf-py name map. This technique is fast, memory-efficient, and reveals exactly which mappings need manual intervention. It is a pattern that could be applied to any new model architecture being added to vLLM's GGUF support.

In the broader narrative of the GLM-5 deployment effort, message 1552 is the moment when the patch's requirements become fully clear. The assistant now knows exactly what code needs to be written: expert weight sideloading (following the deepseek_v2 pattern), KV split reassembly, and e_score_correction_bias manual mapping. The next step—writing and testing the actual patch—can proceed with confidence.