Decoding the GGUF Tensor Shapes: A Detective Story in Model Deployment

Introduction

In the complex world of large language model deployment, few tasks are as intricate as adapting a model's weights from one format to another. This is especially true when the model architecture is novel and the tooling ecosystem hasn't fully caught up. Message [msg 1663] captures a pivotal moment in such an adaptation: the assistant, having just merged ten split GGUF files into a single 402GB model file for the GLM-5 architecture, discovers that the tensor shapes inside the GGUF file tell a different story than expected. This discovery forces a fundamental revision of the weight reassembly logic, and in doing so, reveals a deeper truth about how the model was originally converted.

This message is a masterclass in reverse-engineering through first principles. It demonstrates how careful inspection of data shapes, combined with knowledge of the conversion pipeline, can uncover assumptions that were baked into code at every level — assumptions that, if left uncorrected, would have caused the model to fail silently at load time.

The Context: A Long Road to GGUF Deployment

To understand the significance of this message, we must first appreciate the journey that led to it. The session began with an ambitious goal: deploy the GLM-5 model using NVFP4 quantization on SGLang, running across eight NVIDIA RTX PRO 6000 Blackwell GPUs. After extensive performance analysis revealed that the NVFP4 path was hitting fundamental bottlenecks in FP4 GEMM kernels on the SM120 architecture, the user pivoted to an alternative approach: GGUF quantization using unsloth's UD-Q4_K_XL format, deployed on vLLM.

This pivot required the assistant to write a comprehensive patch for vLLM's GGUF loader, adding support for the glm_moe_dsa architecture — a novel architecture that combines elements of DeepSeek's MoE design with GLM's unique attention mechanism. The patch had to handle several challenges:

  1. Expert weight mapping: The GGUF format stores expert weights differently than vLLM expects, requiring manual remapping.
  2. Split attention weights: The kv_b_proj weight (a combined key/value bias projection) is split into separate attn_k_b and attn_v_b tensors in GGUF format, requiring reassembly.
  3. Architecture detection: vLLM's weight loader needed to recognize the glm_moe_dsa architecture and apply the correct mapping rules. The download itself was a saga: the 431GB model was split into 10 shards, and the assistant had to restart a failed download, build the llama-gguf-split tool from source, and merge the shards into a single 402GB file. By the time we reach message [msg 1663], the merge has completed successfully, and the assistant is inspecting the merged file to verify that the patch will work correctly.

The Message: Reading the Tensor Shapes

The message opens with the assistant examining the output of a Python script that inspected the GGUF file's tensor metadata. Two critical tensors are under scrutiny:

blk.0.attn_k_b.weight: type=Q8_0 shape=[np.uint64(192), np.uint64(512), np.uint64(64)] n_elements=6291456
blk.0.attn_v_b.weight: type=Q8_0 shape=[np.uint64(512), np.uint64(256), np.uint64(64)] n_elements=8388608

These are the split attention key and value bias tensors. In the original HuggingFace model, these are combined into a single kv_b_proj weight of shape [28672, 512] (where 28672 = 64 heads × 448 dimensions per head, and 512 is the kv_lora_rank). The GGUF conversion process splits this into k_b (the first 192 dimensions per head, representing the non-rotary portion of the key) and v_b (the remaining 256 dimensions per head, representing the value).

The assistant immediately recognizes a critical discrepancy. The shapes are three-dimensional, with the third dimension being 64. This is n_head_kv — the number of key-value heads. The assistant had previously assumed that the GGUF was produced with n_head_kv=1, a multi-query attention optimization that the latest llama.cpp conversion scripts apply to DeepSeek-derived architectures. But the data clearly shows n_head_kv=64, the original model dimension.

The Reasoning Process: Working Backwards from First Principles

What follows is a beautiful example of deductive reasoning. The assistant walks through the shape interpretation step by step:

Step 1: Understanding GGUF's dimension convention. GGUF stores tensor shapes in reverse order compared to the data layout. So a GGUF shape of [192, 512, 64] means the actual data tensor has shape [64, 512, 192]. This is a critical detail that many developers get wrong.

Step 2: Verifying element counts. The assistant checks that 64 × 512 × 192 = 6,291,456, which matches the reported n_elements. This confirms the shape interpretation is correct.

Step 3: Tracing the conversion logic. The assistant reconstructs how the original kv_b_proj weight was split during GGUF conversion:

  1. kv_b = data_torch.view(64, 448, 512) — reshape to [n_head_kv, qk_nope_head_dim + v_head_dim, kv_lora_rank]
  2. k_b = kv_b[:, :192, :] → [64, 192, 512] — take the first 192 dimensions (non-rotary key)
  3. k_b.transpose(1, 2) → [64, 512, 192] — transpose for GGUF storage
  4. v_b = kv_b[:, 192:, :] → [64, 256, 512] — take the remaining 256 dimensions (value) Step 4: Deriving the reverse operation. With n_head_kv=64, the reassembly is straightforward:
  5. k_b = k_b.transpose(1, 2) → [64, 192, 512] — undo the transpose
  6. kv_b = torch.cat([k_b, v_b], dim=1) → [64, 448, 512] — concatenate along the head dimension
  7. kv_b_proj = kv_b.reshape(64 * 448, 512) → [28672, 512] — flatten to match the original HF shape The assistant notes with satisfaction: "This is much simpler — the shapes work out perfectly."

The Broader Significance

This discovery has implications beyond just the GLM-5 model. The assistant had previously noted that the existing DeepSeek V2/V3 GGUF support in vLLM was also broken due to the same kv_b_proj mapping issue. The patch being developed fixes a latent bug for those architectures as well.

More importantly, this message reveals a fundamental tension in the GGUF ecosystem. The llama.cpp conversion scripts have evolved over time, with different versions applying different optimizations. The n_head_kv=1 optimization (collapsing multiple key-value heads into a single head for more efficient storage) was added relatively recently. Models converted with older versions of the script — or with custom forks like unsloth's — may not have this optimization. A robust loader must handle both cases.

The assistant's approach — inspecting the actual tensor shapes and deriving the correct logic from first principles — is far more reliable than blindly following the conversion code's assumptions. It's a lesson in defensive programming: never assume your input data conforms to the latest version of the spec.

Assumptions Made and Corrected

Several assumptions were implicit in the assistant's earlier work:

  1. The GGUF was produced with n_head_kv=1: This was the assumption baked into the initial patch. The tensor inspection proved it wrong.
  2. The conversion followed the latest llama.cpp conventions: The unsloth fork may use a different conversion pipeline.
  3. The shape dimensions would match the expected layout: The assistant had to carefully verify the GGUF dimension ordering convention. The key mistake was not in the logic itself, but in the assumption about which version of the conversion script produced the file. This is an easy trap to fall into when working with rapidly evolving tooling.

Input and Output Knowledge

Input knowledge required to understand this message includes:

Conclusion

Message [msg 1663] is a small but crucial moment in a much larger deployment effort. It demonstrates that successful model deployment is not just about running commands — it's about understanding the data at every stage of the pipeline, questioning assumptions, and being willing to revise your understanding when the data tells a different story.

The assistant's methodical approach — reading the raw tensor shapes, interpreting them correctly in the GGUF convention, tracing the conversion logic step by step, and deriving the reverse operation — is a template for how to debug data format issues in machine learning systems. It's slow, careful work, but it's the only way to ensure that a 402GB model file loads correctly across all eight GPUs.

The message also highlights a broader truth about the ML ecosystem: formats and conventions are constantly evolving, and code that works today may break tomorrow when a new version of a conversion script changes an optimization. The only defense is to verify, verify, verify — and to build your understanding from first principles rather than from assumptions about what the data "should" look like.