The Latent Bug: How Debugging GLM-5 GGUF Loading Revealed a Broken DeepSeek V2/V3 Path in vLLM

Introduction

In the complex ecosystem of large language model deployment, few tasks are as intricate as adapting a model architecture that the inference engine was never designed to support. This was the challenge facing the assistant in Segment 13 of an opencode coding session, where the goal was to deploy the GLM-5 model using GGUF quantization on vLLM. The GLM-5 model uses a novel architecture called glm_moe_dsa (DeepSpeed Attention), which is closely related to DeepSeek V2/V3's MLA (Multi-head Latent Attention) mechanism but with critical differences in how expert weights and attention projections are organized.

Message 1602 represents a pivotal moment of discovery in this journey. In this single message, the assistant connects several threads of investigation and arrives at a startling conclusion: the existing GGUF support for DeepSeek V2/V3 in vLLM is broken, suffering from a latent bug where the critical kv_b_proj weight goes unloaded. This discovery fundamentally changes the assistant's understanding of the problem and has direct implications for the patch it is writing.

The Context: A Chain of Investigations

To understand message 1602, we must trace the chain of reasoning that led to it. The assistant had been working through a multi-step process:

First, it had discovered that llama.cpp's convert_hf_to_gguf.py splits the kv_b_proj weight into two separate tensors during GGUF conversion: attn_k_b and attn_v_b. This is done as part of the MLA decomposition, where the key and value projections are separated for efficient inference. The conversion code forces n_head_kv = 1 (a single key-value head, i.e., MQA representation) regardless of the original model's configuration.

Second, the assistant had verified the actual shapes of the GLM-5 model's weights. The original kv_b_proj.weight has shape [28672, 512] — which equals [64 * (192 + 256), 512], where 64 is the number of key-value heads, 192 is qk_nope_head_dim, and 256 is v_head_dim. But the GGUF conversion with n_head_kv=1 would reshape this to a completely different layout.

Third, the assistant had been examining vLLM's GGUF loading infrastructure — specifically gguf_loader.py and weight_utils.py — to understand how it maps GGUF tensor names back to HuggingFace model parameter names. This is where the critical insight emerged.

The Discovery: Tracing the Missing Weight

Message 1602 begins with the assistant reading the get_gguf_extra_tensor_names function from vLLM's weight_utils.py:

So get_gguf_extra_tensor_names returns the HF names whose GGUF counterparts are NOT in the file. For DeepSeek, attn_kv_b won't be in the actual GGUF file, so kv_b_proj.weight will be returned as "extra" — meaning it gets treated as a weight to be provided by other means (like tie_word_embeddings), or it just gets skipped.

This is the moment of synthesis. The assistant had previously discovered (in message 1600) that gguf-py's tensor name map associates kv_b_proj with attn_kv_b. But the actual GGUF file produced by llama.cpp does not contain an attn_kv_b tensor — it contains attn_k_b and attn_v_b separately. Therefore, when vLLM's GGUF loader iterates over the tensors in the file, it never finds a match for kv_b_proj. The weight is classified as "extra" — a tensor expected by the model but missing from the GGUF file.

The assistant then articulates the consequence with striking clarity:

But wait — kv_b_proj is a real weight that MUST be loaded. If it's "extra" (not in GGUF), the parameter will be left uninitialized! This means DeepSeek V2/V3 GGUF loading in vLLM is currently broken for kv_b_proj. The model will have uninitialized kv_b_proj weights.

The bold emphasis is the assistant's own, and it marks a watershed moment. What began as an investigation into supporting a new architecture (GLM-5's glm_moe_dsa) has uncovered a pre-existing bug in vLLM's handling of the closely related DeepSeek V2/V3 architecture.

The Reasoning Process: Connecting the Dots

The thinking visible in message 1602 is a masterclass in systematic debugging. The assistant proceeds through several logical steps:

  1. Understand the data flow: It traces how get_gguf_extra_tensor_names works — it computes the set difference between expected GGUF keys (from the architecture's tensor name map) and actual GGUF keys (from the file). Any expected key not found in the file is returned as "extra."
  2. Apply the understanding to DeepSeek: The assistant knows that the gguf-py name map maps kv_b_proj → attn_kv_b. It also knows from its earlier research (messages 1582-1594) that llama.cpp's converter produces attn_k_b and attn_v_b instead of attn_kv_b. Therefore, attn_kv_b is in the expected set but not in the actual file — so kv_b_proj.weight is classified as "extra."
  3. Reason about consequences: The assistant considers what happens to "extra" weights. It notes that they might be handled specially (like tie_word_embeddings) or simply skipped. Given that kv_b_proj is a critical attention projection weight, being skipped would leave the parameter uninitialized, causing silent corruption of model outputs.
  4. Formulate the hypothesis: The assistant explicitly states that DeepSeek V2/V3 GGUF loading is broken for kv_b_proj. It then hedges slightly — "Unless... let me check if there's a special handling in the model's load_weights" — and proceeds to investigate further by reading more of the weight_utils code. This hedging is important. The assistant has formed a strong hypothesis but recognizes that there might be another code path that handles the split tensors. It does not commit to the conclusion but instead uses it as motivation for further investigation.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message:

  1. That attn_kv_b is indeed the expected GGUF key for kv_b_proj: This was verified in message 1600 by querying gguf-py's tensor name map. The assumption is well-grounded.
  2. That the GGUF file genuinely lacks attn_kv_b: This follows from the earlier research into llama.cpp's conversion code, which clearly shows the split into attn_k_b and attn_v_b. However, the assistant has not yet inspected the actual GGUF file to confirm this — it's relying on its understanding of the conversion code.
  3. That "extra" weights are simply skipped: The assistant acknowledges this might not be the full story, noting that some extra weights (like tie_word_embeddings) might be handled through other mechanisms. It's appropriately cautious.
  4. That the bug manifests as uninitialized weights: This is a reasonable inference, but the actual behavior depends on how vLLM's weight loading handles the mismatch. It could also manifest as a crash, a warning, or silent correctness degradation. One potential mistake in the reasoning is the assumption that the DeepSeek V2/V3 GGUF path has been tested and is known to work. The assistant seems surprised to find it broken, suggesting it assumed the existing support was functional. In reality, the GGUF support for DeepSeek architectures in vLLM may have been added but never properly tested end-to-end, or it may work only for specific configurations that don't use the MLA decomposition.

Input Knowledge Required

To fully understand message 1602, the reader needs knowledge of:

Output Knowledge Created

Message 1602 produces several valuable insights:

  1. A confirmed bug in vLLM: The assistant has identified that DeepSeek V2/V3 GGUF loading likely fails to load the kv_b_proj weight, which would cause incorrect model behavior.
  2. A roadmap for the patch: The discovery directly informs what the assistant needs to do next. Rather than just adding glm_moe_dsa to the architecture list, the patch must also handle the attn_k_b/attn_v_b → kv_b_proj reassembly that the existing code doesn't handle.
  3. A broader impact assessment: The bug affects not just GLM-5 but potentially all DeepSeek V2/V3 GGUF models loaded through vLLM. This is significant because DeepSeek V3 is a widely used open-weight model.
  4. A clear next step: The assistant ends the message by deciding to investigate further — specifically checking if there's special handling in the model's load_weights method that might compensate for the missing tensor. This demonstrates a methodical approach to hypothesis testing.

The Broader Significance

Message 1602 exemplifies a common pattern in systems debugging: the discovery of a latent bug while attempting to add new functionality. The assistant's investigation into GLM-5 GGUF support naturally led it to examine the existing DeepSeek V2/V3 path, and the same logical gap that would break GLM-5 loading was found to already break DeepSeek loading.

This is a powerful reminder that in complex software systems, "supported" does not always mean "working." The DeepSeek V2/V3 GGUF support in vLLM may have been added based on the tensor name map without end-to-end testing, or it may have worked at one point and regressed due to changes in llama.cpp's conversion format.

The assistant's systematic approach — tracing the data flow from GGUF file through tensor name mapping to weight loading — is precisely the methodology needed to uncover such bugs. Rather than assuming the existing infrastructure is correct, the assistant verifies each step and discovers the disconnect.

Conclusion

Message 1602 is a turning point in the GLM-5 deployment effort. What began as a straightforward task of adding a new architecture to vLLM's GGUF loader has evolved into a deeper investigation of the entire GGUF loading pipeline, revealing a bug that affects not just the target model but also the closely related DeepSeek V2/V3 models.

The assistant's reasoning demonstrates the importance of understanding the full data flow in a system, from model conversion through storage to loading. By connecting its knowledge of llama.cpp's conversion code with vLLM's loading infrastructure, the assistant identified a mismatch that would otherwise have led to silent model corruption.

This message also highlights the value of explicit hypothesis formation and cautious hedging. The assistant states its conclusion boldly but immediately qualifies it with "Unless..." and proceeds to gather more evidence. This intellectual honesty — the willingness to be wrong and the commitment to verification — is what makes debugging productive.

The next steps, as the assistant indicates, will involve checking the model's load_weights method for special handling, and ultimately implementing the attn_k_b/attn_v_b → kv_b_proj reassembly logic that is missing from both the DeepSeek V2/V3 and GLM-5 paths. The discovery in message 1602 has transformed the scope of the work: from adding a new architecture to fixing a fundamental gap in the GGUF loading infrastructure.