Unraveling the GGUF Tensor Name Map: A Critical Investigation into vLLM's DeepSeek V2/V3 GGUF Support

Introduction

In the course of deploying the massive GLM-5 model (a 78-layer, 402GB GGUF-quantized architecture) on vLLM, the assistant encountered a fundamental architectural puzzle. The GGUF conversion process, as implemented in llama.cpp, splits the attention key-value bias tensor (kv_b_proj) into two separate tensors (attn_k_b and attn_v_b). To load this model into vLLM, the assistant needed to reverse this split — but the exact mechanism depended on understanding how gguf-py's tensor name mapping system worked. Message [msg 1600] captures the precise moment when the assistant ran a diagnostic Python script to inspect this mapping, and the results proved far more consequential than a simple lookup.

The Message

The assistant executed the following command on the remote server:

ssh root@10.1.230.174 '/root/ml-env/bin/python3 << '\''PYEOF'\''
import gguf
import torch
from transformers import AutoConfig, AutoModelForCausalLM

# Check what gguf-py maps for deepseek2 architecture
arch = None
for key, value in gguf.MODEL_ARCH_NAMES.items():
    if value == "deepseek2":
        arch = key
        break

print(f"Architecture: {arch}")
name_map = gguf.get_tensor_name_map(arch, 3)  # 3 layers for testing

# Check kv_b_proj mapping
for suffix in ["kv_b_proj", "k_b_proj", "v_b_proj"]:
    result = name_map.get_name(f"model.layers.0.self_attn.{suffix}")
    print(f"  model.layers.0.self_attn.{suffix} -> {result}")
PYEOF
'

The output was:

Architecture: 69
  model.layers.0.self_attn.kv_b_proj -> blk.0.attn_kv_b
  model.layers.0.self_attn.k_b_proj -> blk.0.attn_k_b
  model.layers.0.self_attn.v_b_proj -> blk.0.attn_v_b

Context: The GGUF Deployment Challenge

To understand why this message was written, we must trace the preceding chain of reasoning. The assistant had been working for hours to deploy the GLM-5 model using GGUF quantization. The model, originally in NVFP4 format, had been abandoned after extensive profiling revealed that KV cache FP8-to-BF16 cast overhead consumed 69% of inference time. The user pivoted to unsloth's UD-Q4_K_XL GGUF quantization, which promised better performance through a different quantization strategy.

However, GLM-5 uses the glm_moe_dsa architecture, which is a variant of DeepSeek V2's MLA (Multi-head Latent Attention) mechanism. The llama.cpp conversion script, in its DeepseekV2Model.modify_tensors() method, performs a critical transformation: it splits the kv_b_proj.weight tensor into attn_k_b and attn_v_b. This is done because the GGUF format and the ggml inference library expect the MLA key-value bias to be stored as two separate tensors, reflecting the decomposed structure of MLA where the key and value projections have different dimensionalities.

The problem is that vLLM's model code expects the original merged kv_b_proj tensor. The assistant had already written a patch to vLLM's gguf_loader.py to reassemble these split tensors, but the exact reassembly logic depended on understanding the tensor shapes and the naming convention used in the GGUF file.

The Investigation Leading Up to This Message

The immediate predecessor to this message was a series of investigations into the GGUF conversion code. The assistant had:

  1. Discovered that n_head_kv is forced to 1 during GGUF conversion (line 7941 of convert_hf_to_gguf.py: self.hparams[&#34;num_key_value_heads&#34;] = 1). This is the MQA (Multi-Query Attention) representation, where the key-value heads are collapsed into a single head for efficiency.
  2. Checked the original HF model shapes and found that kv_b_proj.weight has shape [28672, 512] = [64 * (192 + 256), 512], meaning the original model has 64 key-value heads.
  3. Realized a fundamental shape mismatch: If the GGUF stores the tensors with n_head_kv=1, the stored shapes would be [1, 512, 192] for k_b and [1, 256, 512] for v_b, but the original HF shape is [28672, 512]. This raised the question: how can you reconstruct a [28672, 512] tensor from [1, 512, 192] and [1, 256, 512] tensors?
  4. Checked whether vLLM already handles DeepSeek V2/V3 GGUF and found that gguf_loader.py has no kv_b handling at all. This was a startling discovery — it meant that either DeepSeek V2/V3 GGUF support in vLLM was completely untested/broken, or the kv_b split was handled somewhere else entirely.
  5. Found a GitHub issue (vllm-project/vllm #30641) reporting that DeepSeek V3 GGUF doesn't work in vLLM, confirming the suspicion that this was a known problem.

The Purpose of This Diagnostic Script

The assistant's reasoning at this point was: "Before I can write the reassembly logic, I need to understand exactly how gguf-py names these tensors. Does the name map already know about both the merged attn_kv_b name and the split attn_k_b/attn_v_b names? If so, the GGUF file could contain either representation, and the loader needs to handle both."

The script queries gguf.get_tensor_name_map() for the deepseek2 architecture. This function returns a bidirectional mapping between HF-style parameter names (like model.layers.0.self_attn.kv_b_proj) and GGUF-style tensor names (like blk.0.attn_kv_b). The assistant tests three name suffixes: kv_b_proj (the merged name), k_b_proj (the key bias split), and v_b_proj (the value bias split).

The Critical Output and Its Implications

The output revealed something crucial:

Assumptions and Their Validation

The assistant made several assumptions in this investigation:

  1. Assumption: The gguf-py name map would reveal how the tensors are stored. This was validated — the name map clearly shows both merged and split naming conventions.
  2. Assumption: The deepseek2 architecture in gguf-py would have entries for both kv_b_proj and the split names. This was confirmed, and it was a non-obvious result. The fact that both naming conventions exist in the map suggests that the GGUF format has evolved over time, with older converters producing merged tensors and newer converters (like the current llama.cpp) producing split tensors.
  3. Assumption: The architecture key for deepseek2 is discoverable through gguf.MODEL_ARCH_NAMES. The script found it successfully (architecture key 69).
  4. Implicit assumption: The name map is consistent across all layers. By testing with 3 layers, the assistant assumed the mapping is uniform. This is a reasonable assumption for transformer models where all layers share the same structure.

Input Knowledge Required

To understand this message, one needs:

  1. Knowledge of the GGUF format: GGUF is a binary format for storing quantized models, developed by the llama.cpp project. It uses a tensor name mapping system where HF-style names are translated to GGUF-style names (e.g., model.layers.0.self_attn.kv_b_proj → blk.0.attn_kv_b).
  2. Knowledge of MLA (Multi-head Latent Attention): DeepSeek V2/V3 and GLM-5 use MLA, which decomposes the key-value projection into separate components. The kv_b_proj tensor in the HF format is a concatenation of the key bias and value bias projections.
  3. Knowledge of the llama.cpp conversion pipeline: The conversion script (convert_hf_to_gguf.py) transforms HF model weights into GGUF format. For DeepSeek V2 models, it splits kv_b_proj into attn_k_b and attn_v_b and forces n_head_kv=1 (MQA representation).
  4. Knowledge of vLLM's weight loading architecture: vLLM uses a gguf_loader.py module that iterates over GGUF tensors and maps them to the model's parameter names using the gguf-py name map.
  5. Python and the gguf library: The script uses gguf.MODEL_ARCH_NAMES (a dict mapping architecture IDs to names) and gguf.get_tensor_name_map() (which returns a name mapping object).

Output Knowledge Created

This message produced several pieces of actionable knowledge:

  1. The gguf-py name map supports both merged and split kv_b naming. This means the GGUF loader must handle both cases: if the GGUF file contains attn_kv_b, load it directly; if it contains attn_k_b and attn_v_b, reassemble them.
  2. The architecture key for deepseek2 is 69. This is a minor but useful detail for debugging.
  3. The existing DeepSeek V2/V3 GGUF support in vLLM is incomplete. The absence of any kv_b handling in gguf_loader.py means that any DeepSeek V2 or V3 model converted to GGUF would fail to load the attention bias weights, likely causing silent weight corruption or runtime errors.
  4. The reassembly approach must be generalized. Since the name map already knows about both naming conventions, the patch should handle both cases generically, rather than hardcoding the split for GLM-5 only.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible through the sequence of tool calls leading up to this message, shows a methodical investigative process:

  1. Start with the concrete problem: The GGUF download is running in the background. While waiting, investigate the tensor mapping.
  2. Trace the conversion code: Read the exact llama.cpp code that performs the split, understand the reshape and transpose operations.
  3. Identify the shape mismatch: The original HF shape is [28672, 512] but the GGUF stores with n_head_kv=1, implying shapes like [1, 512, 192]. This seems contradictory.
  4. Check if vLLM already handles this: Search vLLM's codebase for any kv_b handling in GGUF loading. Find nothing.
  5. Verify the suspicion: Check GitHub for bug reports about DeepSeek V3 GGUF not working. Find confirmation.
  6. Query the gguf-py name map: Run a diagnostic script to see exactly how the tensor names map. This is message [msg 1600].
  7. Interpret the results: The name map supports both naming conventions. This means the GGUF file could contain either format, and the loader must handle both. The beauty of this investigation is that it started with a specific problem (loading GLM-5 GGUF) and uncovered a general bug affecting an entire family of models (DeepSeek V2/V3 GGUF). The assistant didn't just fix the GLM-5 issue — it identified and prepared to fix a latent bug in vLLM's GGUF support that would affect any user trying to run DeepSeek V2 or V3 models in GGUF format.

Conclusion

Message [msg 1600] represents a pivotal moment in the GLM-5 deployment effort. What began as a routine check of tensor name mappings became the key insight that unlocked the entire GGUF loading pipeline. By confirming that gguf-py's name map supports both merged and split kv_b tensor naming conventions, the assistant gained the confidence to write a generalized reassembly patch that would handle both cases. The discovery that vLLM's existing DeepSeek V2/V3 GGUF support was broken added urgency and broader impact to the work. This message exemplifies the kind of deep, investigative debugging that characterizes serious ML engineering — tracing through multiple layers of abstraction (llama.cpp conversion, gguf-py name mapping, vLLM weight loading) to understand exactly how data flows from one format to another, and where the gaps lie.