The Critical Patch: Reassembling Split Attention Weights for GLM-5 GGUF in vLLM

"Now add the kv_b reassembly in the _get_weights_iterator method and store the config on the instance"

This single line, issued by the assistant in message 1610, represents the culmination of an extensive forensic investigation into the internals of vLLM's GGUF loading pipeline. On its surface, it is a brief instruction to apply a code edit. But beneath that terseness lies a story of architectural discovery, latent bug hunting, and the careful design of a patch that would enable a 744-billion-parameter model to run on a framework that had never been designed to support it.

The Problem: A Weight Split Across Two Tensors

The GLM-5 model, developed by Zhipu AI, uses Multi-Head Latent Attention (MLA), a sophisticated attention mechanism that compresses the key-value cache into a low-rank latent space. In the Hugging Face (HF) reference implementation, this compression is governed by a single weight matrix called kv_b_proj with shape [28672, 512] — 64 key-value heads times 448 combined dimensions, projected into a 512-dimensional latent space.

When the llama.cpp conversion scripts translate this model into the GGUF format, they perform a structural transformation. The kv_b_proj weight is split into two separate tensors: attn_k_b (the key projection, shape [1, 512, 192] in its stored 3D form) and attn_v_b (the value projection, shape [1, 256, 512]). This split is part of a broader conversion to a Multi-Query Attention (MQA) representation, where n_head_kv is set to 1 in the GGUF metadata, and the C++ inference engine handles the decompression internally.

The problem is that vLLM's model code — specifically the DeepseekV2ForCausalLM class, from which GlmMoeDsaForCausalLM inherits — expects the original HF format: a single kv_b_proj.weight tensor with shape [28672, 512]. The GGUF file contains neither a tensor named attn_kv_b (which the auto-mapping expects) nor a single combined tensor. It contains two separate tensors that must be identified, buffered, transposed, concatenated, and yielded as a single weight.

The Discovery: A Latent Bug in DeepSeek Support

The assistant's investigation, spanning messages 1587 through 1609, revealed something more troubling: this was not just a GLM-5 problem. The same kv_b_proj split exists in every DeepSeek V2 and V3 GGUF file produced by llama.cpp's conversion scripts. Yet vLLM's gguf_loader.py contained zero handling for kv_b_proj, attn_k_b, or attn_v_b — a fact confirmed by a grep that returned empty results ([msg 1596]).

The assistant traced the issue through the entire loading pipeline. In gguf_loader.py, the _get_gguf_weights_map() method builds a bidirectional mapping between GGUF tensor names and HF parameter names. For the deepseek2 architecture, gguf-py's get_tensor_name_map() maps kv_b_proj → attn_kv_b, but the actual GGUF file contains attn_k_b and attn_v_b — not attn_kv_b. This means the auto-mapped entry points to a tensor that doesn't exist, and the two tensors that do exist are silently skipped because they have no entry in the name map.

The result is catastrophic: kv_b_proj remains uninitialized. A GitHub issue confirmed the symptom: ValueError: Attempted to use an uninitialized parameter in vllm._fused_mul_mat_gguf ([msg 1604]). This was an open, known bug in vLLM's DeepSeek V2/V3 GGUF support — one that the assistant's GLM-5 patch would incidentally fix.

The Design: Sentinel Suffixes and Buffered Reassembly

The assistant's chosen approach, articulated in message 1609, is elegant in its simplicity. Rather than adding complex state-tracking to the weight iterator, it exploits the existing mapping infrastructure:

Map both split tensors to the same HF name with sentinel suffixes: attn_k_b → kv_b_proj.weight__k_b and attn_v_b → kv_b_proj.weight__v_b. The __k_b and __v_b suffixes are not real parameter names — they are markers that the weight iterator will recognize.
Intercept in the weight iterator: When the iterator encounters a name ending in __k_b or __v_b, it strips the suffix and buffers the tensor in a per-layer dictionary. When both the k_b and v_b components for a given layer have arrived, it performs the reassembly: transposing k_b back from its stored shape, concatenating along the appropriate dimension, and yielding the combined kv_b_proj.weight.
Store config on the instance: The model configuration (containing n_head_kv, kv_lora_rank, qk_nope_head_dim, v_head_dim, and other architectural parameters) is stored on the loader instance so the reassembly logic can compute the correct output shape. This approach has several virtues. It avoids modifying the core weight-yielding loop, it handles layers independently (important for models with heterogeneous layer structures like GLM-5's first 3 dense layers), and it cleanly separates the mapping concern from the reassembly concern.

The Thinking Process: From Shape Analysis to Implementation

The assistant's reasoning, visible across the preceding messages, reveals a meticulous approach to the shape problem. In message 1594, it manually traces the tensor transformations:

"k_b: view as [1, 192, 512] then transpose → [1, 512, 192], stored as 3D in GGUF" "v_b: view as [1, 256, 512], stored as 3D in GGUF"

It then works through the reverse transformation step by step, computing the intermediate shapes and verifying that the final concatenation produces the expected [28672, 512] output. This is not abstract reasoning — it is concrete tensor arithmetic, performed by reading the llama.cpp conversion source code and mentally simulating the forward and reverse passes.

The assistant also confronts a subtle question: does the GGUF store the MQA representation (with n_head_kv=1) or the original multi-head representation (with n_head_kv=64)? The answer determines whether the reassembly needs to "un-MQA" the weights or simply transpose and concatenate. By examining the GGUF metadata written by set_gguf_parameters() — which sets n_head_kv=1 — and the actual tensor shapes in the file, the assistant deduces that the GGUF was produced by an older converter that preserved the original n_head_kv=64 shape representation. This discovery, made later in the session, would force a revision of the reassembly logic — but at message 1610, the initial implementation proceeds with the MQA-aware approach.

Assumptions and Risks

The patch makes several assumptions that warrant scrutiny. First, it assumes that the transpose-and-concatenate operation exactly reverses the split performed by llama.cpp's conversion — any off-by-one in dimension ordering would produce silently incorrect weights. Second, it assumes that all layers use the same kv_lora_rank, qk_nope_head_dim, and v_head_dim parameters, which is true for GLM-5 but may not hold for future architectures. Third, it assumes that the sentinel suffix approach does not conflict with any other weight name in the model — a reasonable assumption given that __ is not a standard separator in Hugging Face parameter names.

The most significant risk is the latent DeepSeek bug. By fixing kv_b_proj loading for GLM-5, the assistant also fixes it for DeepSeek V2/V3 — but this fix has never been tested with those models. If the shape arithmetic differs between DeepSeek and GLM-5 (different kv_lora_rank, different head dimensions), the fix could produce incorrect results for DeepSeek users. The assistant acknowledges this implicitly by treating the fix as incidental rather than intentional.

Conclusion

Message 1610 is a moment of synthesis — the point where days of research, debugging, and architectural analysis crystallize into a single code edit. The kv_b reassembly logic is the linchpin of the entire GLM-5 GGUF deployment: without it, the model's attention mechanism cannot initialize, and every subsequent step (model loading, inference, benchmarking) is impossible. By fixing this one piece, the assistant unlocks the entire pipeline. And by incidentally fixing the same bug for DeepSeek V2/V3, it contributes a fix to a known open issue in the vLLM project — a bonus that emerged not from design but from the thoroughness of the investigation.