The Final Piece of the Puzzle: Verifying vLLM's Weight Naming for the GLM-5 GGUF Patch

In the middle of a complex, multi-session effort to deploy the GLM-5 model on a high-performance GPU server, the assistant reached a critical inflection point. After abandoning the NVFP4 quantization path and pivoting to GGUF UD-Q4_K_XL deployment on vLLM, the assistant had discovered a major blocker: neither transformers nor gguf-py supported the glm-dsa architecture used by GLM-5. The user had explicitly rejected all alternative paths—reverting to sglang, llama.cpp, or FP8—and instead directed the assistant to add GGUF support to vLLM directly. Message [msg 1554] represents the final research step before writing that patch, a moment where the assistant verified one last critical assumption about how vLLM's model expects its weights to be named.

Context: The GGUF Pivot

The session had been a long and winding road. Starting with NVIDIA driver installation on Ubuntu 24.04, through CUDA toolkit setup, flash-attn build troubleshooting, and extensive performance tuning of the NVFP4 model on SGLang, the assistant had spent dozens of rounds optimizing throughput. But the user ultimately decided to abandon NVFP4 and switch to GGUF quantization using unsloth's UD-Q4_K_XL method, deploying on vLLM instead. This pivot required a fundamentally different approach: rather than tuning server parameters, the assistant now needed to write code to make vLLM understand a model architecture it had never seen before.

The research phase had been thorough. The assistant had installed vLLM nightly (version 0.16.0rc2.dev313), upgraded transformers to 5.3.0.dev0 from git HEAD, and installed gguf-py from llama.cpp source—which, crucially, already defined LLM_ARCH_GLM_DSA with a complete tensor name map. Through a series of probing commands, the assistant had discovered several key facts:

  1. The blocker is in transformers, not vLLM: vLLM already has manual weight mappings for DeepSeek architectures, and the GGUF loader creates a dummy HF model via AutoModelForCausalLM.from_config() to establish the name mapping. The HF config for GLM-5 loads fine with model_type: "glm_moe_dsa".
  2. The GGUF file splits kv_b_proj: In the GGUF format, the key-value projection for MLA (Multi-head Latent Attention) is stored as two separate tensors: attn_k_b and attn_v_b. But the HF model expects a single kv_b_proj tensor.
  3. Expert weights use fused format: The MoE experts are stored as fused gate_up_proj tensors (combining gate and up projections), and there's an e_score_correction_bias tensor that needs manual mapping.
  4. 150 parameters are unmapped: Out of 1629 total parameters in the HF dummy model, exactly 150 were not automatically mapped by gguf-py—all of them being experts.gate_up_proj and gate.e_score_correction_bias across the 75 MoE layers.

The Subject Message: A Targeted Verification

Message [msg 1554] is deceptively simple. It contains a single tool call—a grep command searching vLLM's DeepSeekV2 model implementation for specific weight names:

ssh root@10.1.230.174 'grep -n "kv_b_proj\|k_b_proj\|v_b_proj\|fused_qkv\|mla_params" ~/ml-env/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py 2>/dev/null | head -30'

The output shows six lines from the DeepSeekV2 model file, revealing that kv_b_proj is implemented as a ColumnParallelLinear layer (line 469), used in the forward pass (line 536), and also present in a fused variant (DeepSeekV2FusedQkvAProj, line 851). The fused_qkv_a_proj and kv_b_proj are listed together in a packed modules mapping (line 917).

This is the moment where the assistant confirms the final piece of the puzzle: vLLM's DeepSeekV2 model expects the weight under the name kv_b_proj, not as separate k_b_proj and v_b_proj tensors. This means the GGUF loader must reassemble the split attn_k_b and attn_v_b tensors from the GGUF file back into a single kv_b_proj weight before passing it to vLLM's load_weights() method.

Why This Verification Matters

The kv_b_proj tensor is the key-value up-projection in MLA (Multi-head Latent Attention), a core component of the GLM-5 architecture. In MLA, the attention mechanism projects a latent representation into separate key and value spaces. The GGUF quantization process splits this projection into two separate tensors (attn_k_b and attn_v_b) for storage efficiency, but vLLM's model implementation expects them combined as a single kv_b_proj weight.

If the assistant had gotten this wrong—if it had assumed vLLM expected separate k_b_proj and v_b_proj weights, or if it had mapped them differently—the weight loading would fail silently or produce incorrect results. The model might load without errors but produce garbage outputs, a notoriously difficult bug to diagnose.

The grep also reveals something subtler: the fused_qkv_a_proj variant (line 818) and the packed_modules_mapping (line 1262) suggest that vLLM has an alternative fused path for the query-key-value projections. This is important context because the GLM-5 GGUF file might use a different tensor layout than what the standard DeepSeekV2 model expects. The assistant needed to understand both the standard and fused paths to write a robust patch.

The Thinking Process Visible in the Message

This message sits at the end of a long chain of reasoning. The assistant's thinking process, visible in the preceding messages, shows a methodical approach:

  1. Start with the gguf-py library: Verify that the glm-dsa architecture is defined and has a complete tensor name map. This was confirmed in [msg 1541] and [msg 1542].
  2. Verify the HF model structure: Create a dummy HF model on meta device and inspect its state dict keys. This revealed the fused expert format (experts.gate_up_proj instead of per-expert gate_proj/up_proj) in [msg 1549] and [msg 1550].
  3. Identify unmapped parameters: Compare the HF model's state dict keys against the gguf-py name map to find tensors that need manual handling. This identified the 150 unmapped parameters in [msg 1551].
  4. Check the kv_b_proj split: Verify that gguf-py maps kv_b_proj to attn_kv_b but the GGUF file has separate attn_k_b and attn_v_b. This was confirmed in [msg 1552].
  5. Verify vLLM's expected naming: Finally, in [msg 1554], check how vLLM's DeepSeekV2 model expects the weight to be named in load_weights(). This confirms the reassembly requirement. This chain of reasoning demonstrates a critical software engineering skill: never assume—always verify. The assistant could have assumed that because gguf-py maps kv_b_proj to attn_kv_b, the GGUF file would contain attn_kv_b. But it checked empirically and found the split. Similarly, it could have assumed that vLLM would accept the weight under any name, but it verified the exact expected parameter name.

Assumptions and Potential Pitfalls

The message makes one significant assumption: that GLM-5's architecture is sufficiently similar to DeepSeekV2 that vLLM's existing DeepSeekV2 model implementation can serve as a reference for the weight naming convention. This is a reasonable assumption—both models use MLA (Multi-head Latent Attention) with similar projection structure—but it's not guaranteed. The GLM-5 model might have subtle differences in how it names or structures its attention projections.

Another assumption is that the kv_b_proj in the GGUF file is exactly the concatenation of attn_k_b and attn_v_b. The assistant hasn't actually inspected the GGUF file's tensor shapes to confirm this. It's possible that the split is not a simple concatenation but involves some transformation (e.g., interleaving or different quantization parameters). The patch will need to handle this carefully.

The assistant also assumes that the DeepSeekV2 model's load_weights() method can handle the fused gate_up_proj format used by GLM-5. The grep output shows that DeepSeekV2 uses stacked_params_mapping for gate_up_proj → gate_proj/up_proj splitting (visible in the subsequent message [msg 1555]), which suggests this pattern is already supported. But the GLM-5's expert structure might differ in ways that break this mapping.

Input Knowledge Required

To understand this message, the reader needs knowledge of:

Output Knowledge Created

This message produces one critical piece of knowledge: vLLM's DeepSeekV2 model expects kv_b_proj as a single ColumnParallelLinear layer. This directly informs the GGUF loader patch:

The Broader Significance

Message [msg 1554] represents the culmination of a systematic research effort. The assistant has now verified every layer of the stack: the GGUF format support in gguf-py, the HF model structure, the unmapped parameters, the kv_b_proj split, and vLLM's expected naming. With all these facts confirmed, the assistant can proceed to write the patch with confidence.

This kind of multi-layered verification is essential when working with complex ML infrastructure. A single mismatch between any two layers—GGUF file format, gguf-py name mapping, HF model structure, or vLLM weight loading—would cause the deployment to fail. By methodically checking each layer and its interfaces, the assistant minimizes the risk of subtle bugs that are difficult to diagnose.

The message also illustrates a key pattern in opencode sessions: the assistant uses the grep tool not just to find code but to understand the intent of the code. By searching for specific weight names in the DeepSeekV2 model, the assistant is reverse-engineering the expected interface that the GGUF loader must satisfy. This is a form of "reading the code as documentation," a skill that becomes increasingly important when working with rapidly evolving frameworks like vLLM where official documentation may lag behind the implementation.

Conclusion

Message [msg 1554] is a small but crucial step in a larger journey. It represents the final verification before a significant code change—adding GGUF support for a new model architecture to vLLM. The assistant's methodical approach, building knowledge layer by layer and verifying each assumption empirically, demonstrates the kind of systematic thinking required to deploy cutting-edge ML models on complex infrastructure. After this message, the assistant has all the information needed to write the patch, and the subsequent messages show the actual implementation and testing of the GLM-5 GGUF support in vLLM.