The Shape Mismatch: Unraveling GGUF's MLA Weight Transformation for GLM-5

In the sprawling, multi-day effort to deploy the GLM-5 model on a production-grade GPU server, one message stands out as a moment of crystallizing insight. Message 1594 is not where code is written or commands are executed—it is where the assistant thinks aloud, tracing the exact tensor shapes through the pipeline from HuggingFace checkpoint to GGUF quantized file, and discovering a fundamental shape mismatch that threatens to break the entire deployment. This message is the fulcrum upon which the rest of the integration pivots.

The Context: A High-Stakes Model Deployment

To understand why message 1594 matters, we must first understand the stakes. The assistant is deploying GLM-5, a 78-layer Mixture-of-Experts model with Multi-head Latent Attention (MLA)—a sophisticated attention mechanism that compresses the key-value cache into a low-rank latent space. The user has chosen to deploy using GGUF quantization (the UD-Q4_K_XL variant from unsloth) on vLLM, a high-performance inference engine. This path requires bridging two incompatible representations: the original HuggingFace checkpoint format (which vLLM natively understands) and the GGUF format (which is designed for llama.cpp's inference engine).

The GGUF conversion process, implemented in llama.cpp's convert_hf_to_gguf.py, does not simply re-encode weights—it transforms them. For MLA architectures like DeepSeek V2/V3 and GLM-5, the conversion splits the combined kv_b_proj weight into separate attn_k_b and attn_v_b tensors, and critically, it forces num_key_value_heads = 1—converting the model to a Multi-Query Attention (MQA) representation. This is an optimization for llama.cpp's inference path, but it creates a nightmare for anyone trying to load these GGUF files back into a framework that expects the original multi-head format.

The Message: Tracing the Shape Transformation

Message 1594 opens with a declaration of understanding: "Now I have the full picture." The assistant has spent the preceding messages (1579–1593) meticulously studying the llama.cpp conversion code, reading the vLLM gguf_loader.py, inspecting the actual HuggingFace model weights, and tracing the inheritance chain from GlmMoeDsaModel through DeepseekV2Model. This message is the synthesis of all that research.

The assistant recaps the critical finding: DeepseekV2Model.set_gguf_parameters() sets self.hparams["num_key_value_heads"] = 1 at line 7941 of the conversion script. Then modify_tensors() reads this value (now 1) and uses it to reshape the kv_b_proj weight. The stored GGUF shapes are therefore:

k_b = k_b.transpose(1, 2)  # → [1, 192, 512]
kv_b = torch.cat([k_b, v_b], dim=1)  # → [1, 448, 512]
kv_b_proj = kv_b.reshape(n_head_kv * (qk_nope_head_dim + v_head_dim), kv_lora_rank)  # → [448, 512]

But then comes the critical realization—bolded in the assistant's own reasoning: "BUT the original HF shape is [28672, 512] = [64 * 448, 512] with n_head_kv=64. The GGUF stores it with n_head_kv=1 which means the data layout is different!"

This is the moment of insight. The simple reshape from [1, 448, 512] to [448, 512] would produce a weight of shape [448, 512], but the original model expects [28672, 512]. The factor of 64—the original number of key-value heads—is missing. The data has been fundamentally rearranged.

The Assumptions and Their Consequences

The assistant makes several assumptions in this message, some explicit and some implicit:

Assumption 1: The GGUF conversion faithfully preserves data, just reshapes it. This is correct—GGUF quantization (especially the Q4_K_XL variant used here) preserves the underlying values while applying block-wise quantization. The information is all there; it's just arranged differently.

Assumption 2: The reversal must produce the exact HF shape. This is the core of the problem. The assistant assumes that vLLM expects the exact same shape as the original HuggingFace checkpoint. But this assumption is worth questioning: could vLLM's DeepSeek V2/V3 loader handle the MQA-shaped weights natively? The message ends with the assistant checking this very question: "Let me check how vLLM handles DeepSeek V2/V3 GGUF — it must have solved this same problem."

Assumption 3: The n_head_kv=1 transformation is a complete MQA conversion. The assistant correctly identifies that llama.cpp forces MQA representation, but the exact nature of this transformation matters enormously. In a true MQA conversion, the key-value heads are pooled into a single head—information is lost. But in the DeepSeek MLA case, the "conversion to MQA" is more subtle: the kv_b_proj weight is simply reshaped from [n_head * (qk_nope + v_head), kv_lora_rank] to [1, (qk_nope + v_head), kv_lora_rank] and then split. The per-head information is still present in the weight values; it's just been rearranged into a 3D tensor with a leading dimension of 1.

The Mistake That Wasn't

The assistant's initial reversal code produces a weight of shape [448, 512], which is clearly wrong for the expected [28672, 512]. But this isn't a mistake—it's a discovery. The assistant is working through the math in real-time, and the mismatch reveals the deeper problem. The correct reversal requires understanding that the GGUF's n_head_kv=1 is a view transformation, not a pooling transformation. The 64 heads' worth of data is still there, but it's been compressed along the head dimension.

The actual reversal needs to account for the fact that the original weight has shape [n_head * (qk_nope + v_head), kv_lora_rank] = [64 * 448, 512] = [28672, 512]. The GGUF stores it as [1, qk_nope, kv_lora_rank] for k_b and [1, v_head, kv_lora_rank] for v_b. The reversal must therefore:

  1. Remove the leading dimension of 1 (squeeze)
  2. Transpose k_b to align dimensions
  3. Concatenate along the feature dimension
  4. Repeat or expand the result to account for the 64 heads Wait—that last step is wrong too. The data isn't repeated; the 64 heads' worth of information is encoded in the weight values themselves. The correct approach is to realize that the GGUF's n_head_kv=1 is a lie—or rather, a convention. The weight was originally [28672, 512] and was reshaped to [64, 448, 512] before being split and transposed. The reversal must go back through these steps in reverse order.

Input Knowledge Required

To fully understand message 1594, the reader needs:

  1. Understanding of MLA (Multi-head Latent Attention): The GLM-5 model uses MLA, which compresses the KV cache into a low-rank latent space. The kv_b_proj weight projects from this latent space back to the full key-value representation. Its shape is [n_head * (qk_nope + v_head), kv_lora_rank].
  2. Knowledge of GGUF format: GGUF is a binary format for storing quantized model weights, designed for llama.cpp. It supports various quantization schemes and metadata fields like n_head_kv.
  3. Understanding of MQA vs MHA: Multi-Query Attention uses a single key-value head shared across all query heads, while Multi-Head Attention has separate key-value heads per query head. The GGUF conversion forces MQA representation even for models that were originally MHA.
  4. Familiarity with the llama.cpp conversion pipeline: The convert_hf_to_gguf.py script transforms HuggingFace checkpoints into GGUF format, applying various model-specific transformations including weight splitting and reshaping.
  5. Knowledge of PyTorch tensor operations: The message uses transpose, view, reshape, and cat operations that require understanding of tensor dimension semantics.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. The exact GGUF storage format for GLM-5's attention weights: attn_k_b is stored as [1, 512, 192] (3D) and attn_v_b as [1, 256, 512] (3D).
  2. The shape mismatch diagnosis: The GGUF stores weights with n_head_kv=1 but vLLM expects n_head_kv=64, creating a factor-of-64 discrepancy in the leading dimension.
  3. The inheritance chain: GlmMoeDsaModel → DeepseekV2Model in llama.cpp's conversion code, meaning the same kv_b split logic applies to both architectures.
  4. The research direction: The assistant identifies that vLLM must have already solved this problem for DeepSeek V2/V3 GGUF support, pointing to the next investigation target.
  5. The reversal algorithm (tentative): A first-pass algorithm for reassembling kv_b_proj from the split tensors, which will need refinement based on the n_head_kv discrepancy.

The Thinking Process: A Window into Debugging

What makes message 1594 remarkable is the visible thinking process. The assistant doesn't just state conclusions—it walks through the math step by step, checking each assumption, and catches its own error in real-time.

The progression is:

  1. Recap: State the known facts about the GGUF conversion
  2. Propose: Write the reversal code based on those facts
  3. Verify: Check against the known HF shape
  4. Discover: Find the mismatch (448 vs 28672)
  5. Diagnose: Identify n_head_kv as the root cause
  6. Question: Ask how vLLM handles this for similar architectures
  7. Investigate: Begin searching for the answer in vLLM's DeepSeek code This is textbook debugging methodology: understand the transformation, propose a reverse transformation, verify against ground truth, and when a discrepancy appears, trace it back to its source. The assistant also demonstrates good research practices by: - Citing specific line numbers (7941, 8055) from the llama.cpp code - Using exact tensor shapes from both the HF checkpoint and the GGUF metadata - Recognizing when an assumption needs verification (the n_head_kv question) - Looking for existing solutions in analogous code paths (DeepSeek V2/V3 in vLLM)

Broader Implications

This message has implications beyond just this deployment. The discovery that the GGUF conversion forces n_head_kv=1 for MLA architectures means that any framework attempting to load DeepSeek V2/V3 or GLM-5 GGUF files must implement this reversal. The assistant's investigation reveals that vLLM's existing DeepSeek V2/V3 GGUF support (if it exists) must already handle this transformation—but if it doesn't, then the bug affects multiple model architectures.

Furthermore, the message highlights a fundamental tension in the ML ecosystem: model formats designed for one inference engine (llama.cpp's GGUF) make assumptions about the target runtime that may not hold for other engines (vLLM). The "MQA conversion" is an optimization for llama.cpp's C++ kernels, but it becomes a compatibility burden for any other framework trying to consume these files.

Conclusion

Message 1594 is the intellectual heart of the GLM-5 GGUF deployment effort. It is where raw information—line numbers, tensor shapes, inheritance chains—is synthesized into genuine understanding. The assistant identifies a shape mismatch that would have caused silent corruption or outright crashes if not caught. By tracing the exact transformation from HF to GGUF and back, the message provides the foundation for the correct reversal algorithm that will be implemented in subsequent messages.

This message also exemplifies a crucial skill in ML engineering: the ability to reason about tensor shapes across format boundaries. In a world where models are trained in one framework, quantized in another, and deployed in a third, this kind of cross-format reasoning is essential. The assistant's methodical approach—state the facts, propose the solution, verify against ground truth, and investigate discrepancies—is a model for how to debug complex model deployment pipelines.