The Tensor Mapping Puzzle: Deciphering GLM-5's GGUF Structure for vLLM Integration

Introduction

In any machine learning deployment pipeline, the moment when theory meets reality is often the most instructive. Message [msg 1553] captures precisely such a moment in the ongoing effort to deploy the GLM-5 model on vLLM using a GGUF quantized format. After weeks of struggling with the NVFP4 quantization path — battling flash-attn compilation issues, diagnosing KV cache cast bottlenecks, and ultimately abandoning the approach — the team had pivoted to GGUF UD-Q4_K_XL quantization. But this new path presented its own formidable challenge: neither transformers nor gguf-py supported the glm-dsa architecture used by GLM-5. The user had explicitly rejected all alternative deployment strategies (reverting to sglang, llama.cpp, or FP8), directing the assistant to "add GGUF support to vLLM" instead.

Message [msg 1553] is the culmination of a deep-dive research session spanning multiple codebases. In it, the assistant synthesizes findings from probing the gguf-py tensor name map, the Hugging Face model's state dictionary, and vLLM's GGUF loader architecture. The message reveals a critical tensor mapping mismatch that would otherwise cause the GGUF loader to silently fail, and it transitions from diagnosis to action by presenting a completed research checklist and signaling readiness to write the patch.

The Discovery: A Split Tensor Nobody Expected

The central revelation of [msg 1553] is deceptively simple but architecturally significant:

Interesting! So gguf-py maps: - kv_b_proj → attn_kv_b (the combined one) - k_b_proj → attn_k_b (separate K) - v_b_proj → attn_v_b (separate V)

>

But the HF model only has kv_b_proj, not k_b_proj/v_b_proj. And the GGUF file will have attn_k_b and attn_v_b (split), NOT attn_kv_b.

This is a classic impedance mismatch between two serialization formats. In the Hugging Face (HF) model format, the key-value projection in the MLA (Multi-head Latent Attention) layer is a single fused linear layer called kv_b_proj. But the GGUF file, as produced by llama.cpp's conversion script, splits this into two separate tensors: attn_k_b for the key projection and attn_v_b for the value projection. The gguf-py library's tensor name map knows about both representations — it has entries mapping kv_b_proj to attn_kv_b, k_b_proj to attn_k_b, and v_b_proj to attn_v_b. But the Hugging Face model only defines the fused version, while the GGUF file only contains the split versions.

The consequence is stark: the automatic name-mapping path in vLLM's GGUF loader would look for blk.N.attn_kv_b.weight in the GGUF file (because the HF dummy model says kv_b_proj maps there), but that tensor simply does not exist. The actual tensors in the file are blk.N.attn_k_b.weight and blk.N.attn_v_b.weight. The loader would fail with a missing-tensor error, and the root cause would be deeply non-obvious to anyone unfamiliar with both the HF and GGUF tensor naming conventions.

The Reasoning Process: How the Assistant Arrived at This Insight

This discovery did not happen in isolation. It was the product of a systematic, multi-step investigation that spanned messages [msg 1540] through [msg 1553]. Understanding the reasoning chain is essential to appreciating what [msg 1553] accomplishes.

Step 1: Verifying architecture support. In [msg 1540], the assistant first confirmed that gguf-py (installed from the latest llama.cpp source) includes glm-dsa as architecture key 73. This was the foundational check — without architecture support in the tensor name map, no amount of patching would work.

Step 2: Probing the tensor name map. In [msg 1541] and [msg 1542], the assistant retrieved the tensor name map for glm-dsa and tested it against a list of known HF parameter names from the GLM-5 architecture. The map correctly resolved attention parameters (q_a_proj, q_b_proj, kv_a_proj_with_mqa, o_proj), layer norms, indexer tensors, and embedding layers. But it returned None for per-expert weights and e_score_correction_bias.

Step 3: Discovering the split tensor structure. In [msg 1543] and [msg 1544], the assistant enumerated the 38 tensor types registered for the glm-dsa architecture. Crucially, it discovered that the architecture defines both ATTN_KV_B (the combined tensor, enum value 123) and ATTN_K_B (124) / ATTN_V_B (125) as separate entries. This dual registration is the telltale sign of a format mismatch: the GGUF specification supports both representations, but a given file will only contain one or the other.

Step 4: Forming the initial patch plan. In [msg 1545], the assistant synthesized these findings into a comprehensive patch plan covering transformers ggml.py, vLLM's gguf_loader.py, and the handling of expert weights, KV split reassembly, indexer tensors, and nextn tensors. However, this plan was based on an assumption that vLLM's GGUF loader depended on transformers for the actual weight mapping.

Step 5: Correcting the assumption. In messages [msg 1546] through [msg 1548], the assistant traced the actual code path in vLLM's config/model.py and gguf_utils.py. It discovered that vLLM's GGUF loader uses AutoModelForCausalLM.from_config(config) to create a dummy Hugging Face model on the meta device, then iterates its state_dict() to determine which parameters need to be loaded. The transformers ggml.py is not used — vLLM has its own _get_gguf_weights_map() method. This meant the transformers patch was unnecessary, and the entire effort could focus on vLLM's gguf_loader.py.

Step 6: Validating the dummy model. In [msg 1549] and [msg 1550], the assistant verified that the dummy HF model could be created successfully and inspected its state dictionary. This revealed another important detail: the expert weights use a fused format (experts.gate_up_proj instead of separate experts.0.gate_proj and experts.0.up_proj), matching the FusedMoE layer structure.

Step 7: Quantifying the unmapped parameters. In [msg 1551], the assistant ran a systematic comparison between the HF model's state dict keys and the gguf-py name map, finding exactly 150 unmapped parameters — 75 MoE layers × 2 params each (experts.gate_up_proj and gate.e_score_correction_bias). The experts.down_proj was mapped automatically, confirming that only the fused gate+up projection and the score correction bias needed manual handling.

Step 8: The final piece. In [msg 1552], the assistant probed the kv_b_proj mapping specifically, confirming that kv_b_proj maps to attn_kv_b (the combined tensor that doesn't exist in the GGUF file), while k_b_proj and v_b_proj map to the split tensors that do exist. This set the stage for the synthesis in [msg 1553].

Input Knowledge Required

To fully understand [msg 1553], one needs familiarity with several distinct knowledge domains:

GGUF format internals. The GGUF (GGML Universal Format) is a binary format for storing quantized model weights, originally developed for llama.cpp. It uses a tensor name convention based on block indices (e.g., blk.0.attn_k_b) rather than the Hugging Face convention (e.g., model.layers.0.self_attn.kv_b_proj). The gguf-py library provides a TensorNameMap that bridges these naming conventions, but the mapping is not always one-to-one — as this case demonstrates.

MLA (Multi-head Latent Attention) architecture. GLM-5 uses the DeepSeek-style MLA, where the key and value projections share a common latent representation. The kv_a_proj_with_mqa projects the input into a shared latent space, and kv_b_proj projects it back to the full key/value dimensions. In the GGUF conversion, this single kv_b_proj is split into separate key and value projections (attn_k_b and attn_v_b), likely because llama.cpp's internal attention implementation treats them separately.

vLLM's GGUF loader architecture. Understanding that vLLM creates a dummy Hugging Face model on the meta device, iterates its state dict, and uses the gguf-py name map to find corresponding tensors in the GGUF file is essential. The loader does not use transformers' own GGUF loading code (ggml.py); it has its own parallel implementation in _get_gguf_weights_map().

The GLM-5 model structure. GLM-5 has 78 transformer layers, each containing an MLA attention layer with indexer (DSA) components, a dense MLP, and a Mixture-of-Experts (MoE) layer with 160 experts per layer (fused into gate_up_proj and down_proj), shared experts, and a routing gate with e_score_correction_bias.

Assumptions and Potential Mistakes

The assistant made several assumptions during this investigation, most of which were validated but one of which was initially incorrect:

The incorrect assumption: transformers ggml.py is required. In [msg 1545], the assistant's initial patch plan included modifying transformers ggml.py to add a glm-dsa config mapping. This was based on the assumption that vLLM's GGUF loader depended on transformers for architecture metadata parsing. The assistant then caught this error by tracing the actual code path in messages [msg 1546]–[msg 1548], discovering that vLLM uses --hf-config-path to load the config directly from Hugging Face, bypassing the transformers GGUF code entirely. This self-correction is a good example of the assistant's rigorous approach — it doesn't assume the patch plan is correct until it has verified the actual code paths.

The validated assumption: gguf-py from llama.cpp HEAD supports glm-dsa. The assistant assumed that installing gguf-py from the latest llama.cpp source would include the glm-dsa architecture. This was validated in [msg 1540] where the architecture list showed 73: glm-dsa. However, this assumption carried risk — the architecture support might have been incomplete or buggy, as it was likely very new code.

The validated assumption: the HF dummy model can be created on meta device. The assistant assumed that AutoModelForCausalLM.from_config(config, trust_remote_code=True) would work with torch.device("meta") for the GLM-5 config. This was validated in [msg 1549], but it depended on the GLM-5 model code being compatible with meta device initialization — not all Hugging Face models support this.

The implicit assumption: the GGUF file follows the same tensor naming as gguf-py expects. This is the most critical assumption underlying the entire patch effort. The assistant is working backwards from the gguf-py tensor name map to infer what the GGUF file will contain. If the actual GGUF file (still downloading at 431 GB) uses different tensor names or a different split strategy, the patch will need adjustments. The assistant acknowledges this uncertainty implicitly by noting "the GGUF file will have attn_k_b and attn_v_b (split)" — this is an inference from the gguf-py architecture definition, not a direct inspection of the file.

Output Knowledge Created

Message [msg 1553] creates several pieces of valuable knowledge:

1. A precise tensor mapping specification. The message documents exactly which GGUF tensors need to be combined to reconstruct the HF model's kv_b_proj: blk.N.attn_k_b.weight and blk.N.attn_v_b.weight must be concatenated (presumably along the appropriate dimension) to form the single kv_b_proj.weight. This is the kind of detail that is almost never documented and must be reverse-engineered from the code.

2. A validated patch scope. By confirming that only 150 parameters are unmapped (the fused expert weights and score correction biases) and that the KV split is the only structural mismatch, the message establishes a clear and bounded scope for the vLLM patch. The assistant knows exactly what needs to be added to _get_gguf_weights_map(): a new branch for model_type == "glm_moe_dsa" that mirrors the existing deepseek_v2/deepseek_v3 handling but with the additional KV split reassembly logic.

3. A completed research checklist. The message includes a todo list showing all research items as completed:

The Thinking Process: Synthesis and Decision

What makes [msg 1553] particularly interesting is the way it synthesizes multiple threads of investigation into a single coherent insight. The message is structured as a direct continuation of the preceding investigation — it begins with "Interesting!" as if the assistant is reacting to its own discovery in real time. This is a hallmark of the assistant's reasoning style: it works through problems incrementally, documenting each finding and building on it.

The key reasoning step in this message is the realization that the gguf-py name map's dual registration of ATTN_KV_B, ATTN_K_B, and ATTN_V_B is not just a convenience feature but a signal of an actual format mismatch. The assistant connects three facts:

  1. The HF model has kv_b_proj (fused)
  2. The gguf-py map maps kv_b_proj to attn_kv_b
  3. But the GGUF file will contain attn_k_b and attn_v_b (split) The conclusion — "This means kv_b_proj will be unmapped by the automatic path, and we need manual handling to combine attn_k_b + attn_v_b → kv_b_proj" — is a direct logical consequence of these three facts. It's a beautiful example of deductive reasoning applied to a systems integration problem. The message also demonstrates a pragmatic engineering mindset. Rather than getting lost in the complexity of the full patch (which involves expert weights, score correction biases, indexer tensors, nextn tensors, and the KV split), the assistant focuses on the single most critical issue: the KV split mismatch that would cause the loader to fail outright. The expert weight handling and other mappings are important but would produce graceful degradation (missing weights) rather than a hard crash. The KV split is the showstopper.

Conclusion

Message [msg 1553] represents the culmination of a deep and systematic investigation into the tensor naming conventions of three different systems: the Hugging Face model format, the GGUF serialization format, and vLLM's weight loading architecture. It identifies a critical mismatch — the split attn_k_b/attn_v_b tensors in GGUF versus the fused kv_b_proj in HF — that would otherwise cause the deployment attempt to fail with an opaque error. By documenting this discovery, validating the patch scope, and completing the research phase, the message sets the stage for the implementation work that follows. It is a textbook example of how careful, multi-step reverse engineering can bridge the gap between two incompatible systems, turning a seemingly insurmountable blocker into a well-defined engineering task.