The Unmapped 150: Deciphering vLLM's GGUF Weight Pipeline for GLM-5

In the sprawling landscape of deploying large language models, few moments are as decisive as the one captured in message 1551 of this opencode session. After weeks of battling FP4 quantization performance issues on NVIDIA RTX PRO 6000 Blackwell GPUs, the user had made a hard pivot: abandon the NVFP4 path entirely and deploy GLM-5 via GGUF UD-Q4_K_XL quantization on vLLM. But this pivot immediately hit a wall. Neither Hugging Face transformers nor the gguf-py library recognized the glm-dsa architecture used by GLM-5. The user's directive was unambiguous — add GGUF support to vLLM, or find no alternative.

Message 1551 represents the culmination of a deep-dive investigation into three separate codebases — transformers, gguf-py (from llama.cpp), and vLLM's own gguf_loader.py — to understand exactly how GGUF weight loading works and what needs to be patched. It is the moment when all the pieces click into place, when the assistant transitions from confusion to clarity, and when the precise scope of the required patchwork becomes known. This article examines that message in detail: the reasoning that drove it, the assumptions it tested, the knowledge it consumed and produced, and the thinking process visible in its analysis.

The Context: A Pivot Under Pressure

To understand message 1551, one must appreciate the pressure that preceded it. The session had been a long, grueling optimization campaign for the GLM-5-NVFP4 model on SGLang. The assistant had diagnosed compute-bound performance, analyzed FP4 GEMM kernel efficiency on the SM120 architecture, tuned server parameters for a 28% throughput improvement, explored expert parallelism, piecewise CUDA graphs, and MSCCLPP allreduce optimization — all documented across eleven improvement documents. But the core bottleneck remained: small per-expert GEMMs on SM120 were fundamentally inefficient.

The user's decision to abandon NVFP4 and switch to GGUF quantization via unsloth's UD-Q4_K_XL was a strategic retreat. GGUF offers mature quantization support, broad tooling compatibility, and the promise of better performance through more conventional means. But the retreat encountered a new front: vLLM's GGUF support depends on transformers for architecture metadata parsing, and neither transformers v5.2.0 nor the installed gguf-py 0.17.1 included the glm-dsa architecture used by GLM-5. Multiple GitHub issues confirmed that DeepSeek and GLM GGUF models consistently fail with ValueError: GGUF model with architecture deepseek2 is not supported yet.

The user had explicitly rejected all alternative paths — reverting to SGLang, llama.cpp, or FP8 — and directed the assistant to add GGUF support to vLLM. This was the mandate that drove the research culminating in message 1551.

The Discovery: Fused Experts and the Weight Mapping Pipeline

Message 1551 opens with a critical observation that reshapes the assistant's entire understanding of the problem:

So the HF model has fused expert params: experts.gate_up_proj (no .weight suffix, no per-expert index) and experts.down_proj. These are from the FusedMoE layer in vLLM. But wait — this is the transformers model, not vLLM's model.

This "but wait" is the hinge point. The assistant had been operating under an implicit assumption: that the dummy model created for GGUF weight mapping would use the same parameter structure as the vLLM model it was ultimately loading into. The discovery that the dummy model uses transformers' FusedMoE layer — with fused gate_up_proj tensors and no per-expert indexing — fundamentally changes the mapping strategy.

The assistant then articulates the full weight mapping pipeline with remarkable clarity:

HF transformers model state_dict keys → gguf tensor names (via gguf-py's TensorNameMap)
GGUF tensor names → HF transformers model keys (reversed mapping)
When loading into vLLM model, the GGUF loader yields (hf_name, tensor) pairs, and vLLM's model.load_weights() handles the HF→vLLM name conversion This three-step pipeline is the key architectural insight. The GGUF loader doesn't directly map GGUF tensors to vLLM model parameters. Instead, it uses the Hugging Face transformers model as an intermediary — a "Rosetta Stone" that translates between the GGUF naming convention and whatever naming convention vLLM's internal models use. The gguf-py library handles step 1 (HF names → GGUF names), and vLLM's existing weight loading infrastructure handles step 3 (HF names → vLLM names). The GGUF loader only needs to handle step 2 — the actual tensor extraction and any special cases that don't map cleanly through gguf-py.

The 150 Unmapped Parameters

The assistant then runs a diagnostic script that crystallizes the problem into a concrete number: 150 of 1629 total parameters are unmapped by the gguf-py tensor name map. Every single one is either experts.gate_up_proj (the fused gate+up projection for MoE experts) or e_score_correction_bias (a routing bias correction tensor used in the DeepSeek-style gating mechanism).

This is a remarkably clean result. The gguf-py glm-dsa tensor map handles everything else — attention projections (including the complex MLA decomposition with q_a_proj, q_b_proj, kv_a_proj_with_mqa, and the split kv_b_proj), layer norms, embeddings, the DSA indexer tensors (wq_b, wk, k_norm, weights_proj), and even the nextn prediction head tensors. Only the expert weights and the gating bias fall through the cracks.

The fact that exactly 150 parameters are unmapped — and that they fall into exactly two categories — tells the assistant exactly what needs to be patched. The fused gate_up_proj format is a transformers-specific convention; in the GGUF file, these tensors are stored per-expert (or in some other arrangement). The e_score_correction_bias is a routing mechanism specific to the DeepSeek/GLM architecture that doesn't have a standard GGUF tensor name.

The Thinking Process: From Confusion to Certainty

What makes message 1551 particularly interesting is the visible reasoning process. The assistant doesn't just run a script and report results. It walks through a chain of deductions:

First, it recognizes the fused expert format. The earlier message 1550 had shown that the HF model's state dict contains model.layers.3.mlp.experts.gate_up_proj — not separate experts.0.gate_proj and experts.0.up_proj. The assistant initially attributes this to "the FusedMoE layer in vLLM," then corrects itself: "But wait — this is the transformers model, not vLLM's model." This self-correction is crucial because it reveals a deeper truth about the architecture.

Second, the assistant synthesizes the three-step pipeline. This isn't stated in any documentation the assistant has read; it's inferred from examining the code flow. The assistant had previously read vLLM's gguf_loader.py (message 1532) and traced how _get_gguf_weights_map() creates a dummy HF model. The insight that the GGUF loader uses HF as an intermediary — not as a direct source of weights — is the key that unlocks the entire patching strategy.

Third, the assistant connects this insight to the existing DeepSeek support in vLLM. The DeepSeekV2/V3 models in vLLM already have manual expert weight mappings in _get_gguf_weights_map(). The assistant realizes that the GLM-5 patch should follow the same pattern: add manual mappings for the expert weights and e_score_correction_bias, while letting gguf-py handle everything else.

Fourth, the assistant runs the unmapped-parameter diagnostic to validate this understanding. The result — 150 unmapped parameters, all experts and bias — confirms the hypothesis perfectly.

Assumptions Made and Validated

Message 1551 operates on several assumptions, some explicit and some implicit:

Assumption 1: The dummy model created by AutoModelForCausalLM.from_config() produces the correct parameter names. The assistant assumes that the transformers model instantiated on the meta device faithfully represents the parameter naming convention that vLLM's weight loader expects. This is validated by the diagnostic: the unmapped parameters are exactly the ones the assistant expected (experts and bias), and all other parameters map cleanly.

Assumption 2: The gguf-py glm-dsa tensor name map is complete and correct for all non-expert tensors. The assistant assumes that the 38 tensor types defined for the glm-dsa architecture in gguf-py (discovered in message 1544) cover all the attention, norm, embedding, indexer, and nextn tensors. The diagnostic validates this: only expert and bias tensors are unmapped.

Assumption 3: vLLM's existing DeepSeek expert weight mapping code can be adapted for GLM-5. The assistant assumes that the manual mapping pattern used for deepseek_v2 and deepseek_v3 in vLLM's GGUF loader — where expert weights are sideloaded from the GGUF file by iterating over expert indices — will work for the GLM-5 architecture. This is a reasonable assumption given that GLM-5 inherits from DeepSeekV2Model in the llama.cpp conversion script, but it remains untested at this point.

Assumption 4: The kv_b_proj split (into attn_k_b and attn_v_b in GGUF) needs to be reassembled. The assistant had discovered in message 1544 that the glm-dsa architecture defines both ATTN_KV_B (the fused key-value projection) and separate ATTN_K_B/ATTN_V_B tensors. The assumption is that the GGUF file stores them separately and they must be concatenated to match the HF model's single kv_b_proj.weight. This is a critical detail that will need careful handling in the patch.

Potential Mistakes and Oversights

While message 1551 is analytically sound, several potential issues deserve scrutiny:

The fused expert format may not be what the GGUF file actually contains. The assistant assumes that because the HF dummy model has experts.gate_up_proj (fused), the GGUF file must store experts differently. But the diagnostic only checks the HF model's naming, not the GGUF file's actual tensor layout. The GGUF file might store experts in yet another format — per-expert with separate gate/up projections, or fused with a different naming convention. The assistant will need to inspect the actual GGUF tensor names once the download completes.

The assumption that model.load_weights() handles HF→vLLM name conversion may be optimistic. While vLLM does have weight loading infrastructure, the GLM-5 model may use parameter names that don't match any existing vLLM model. The assistant hasn't verified that vLLM's GlmMoeDsaForCausalLM model class (or whatever vLLM calls it) has a load_weights() method that correctly maps from HF naming conventions.

The e_score_correction_bias handling is uncertain. This tensor is specific to the DeepSeek-style routing mechanism. The assistant assumes it needs manual mapping, but hasn't determined whether it should be mapped to a specific vLLM parameter name or handled differently. The DeepSeekV2/V3 support in vLLM may or may not have a corresponding parameter.

The .weight suffix issue. The assistant's diagnostic strips .weight and .bias suffixes before checking the gguf-py name map. This is correct for the name map lookup, but the actual GGUF tensor names in the file may or may not include these suffixes. The assistant will need to verify the actual GGUF tensor naming convention.

Input Knowledge Required

To fully understand message 1551, one needs:

Knowledge of the GGUF format: Understanding that GGUF files store tensors with specific names (like blk.0.attn_q_a, blk.0.attn_kv_b) and that the gguf-py library provides a TensorNameMap for translating between Hugging Face naming and GGUF naming.
Knowledge of vLLM's GGUF loader architecture: Understanding that vLLM's GGUFModelLoader creates a dummy Hugging Face model on the meta device, uses gguf-py to map HF parameter names to GGUF tensor names, extracts the tensors from the GGUF file, and yields (hf_name, tensor) pairs that vLLM's existing weight loading infrastructure consumes.
Knowledge of the GLM-5 / DeepSeek architecture: Understanding the Mixture-of-Experts (MoE) structure with fused gate_up_proj, the MLA (Multi-head Latent Attention) decomposition with q_a_proj, q_b_proj, kv_a_proj_with_mqa, and kv_b_proj, the DSA (Dynamic Sparse Attention) indexer mechanism, and the e_score_correction_bias routing correction.
Knowledge of the conversation history: Understanding that this message follows weeks of optimization work on the NVFP4 path, a user-directed pivot to GGUF, the discovery that glm-dsa architecture is unsupported, and the user's explicit rejection of alternative deployment strategies.

Output Knowledge Created

Message 1551 produces several concrete outputs:

A precise count and categorization of unmapped parameters: 150 of 1629 total, all experts.gate_up_proj and e_score_correction_bias.
A validated understanding of the weight mapping pipeline: The three-step HF→GGUF→vLLM translation chain, with the GGUF loader as the intermediary.
A clear patch scope: The vLLM gguf_loader.py needs manual mappings for expert weights (following the DeepSeek pattern) and e_score_correction_bias. The gguf-py library from llama.cpp HEAD already has the glm-dsa architecture support — it just needs to be installed from source.
A confirmed approach: The assistant will write a comprehensive patch for gguf_loader.py that handles glm_moe_dsa model_type with expert weight sideloading, KV split reassembly, and indexer/nextn tensor mapping.
A diagnostic script that can be reused to verify the patch: the unmapped-parameter analysis using the HF dummy model and gguf-py's TensorNameMap.

Conclusion

Message 1551 is a turning point in the opencode session. It represents the moment when a complex, multi-codebase problem becomes tractable — when the assistant moves from "what needs to be done?" to "here is exactly what needs to be patched, and here is how." The reasoning is methodical: observe a discrepancy (fused experts), trace the architecture (the three-step pipeline), connect to existing patterns (DeepSeek manual mappings), and validate with data (150 unmapped parameters). The message demonstrates how effective LLM-assisted coding can be when the assistant combines code reading, experimentation, and architectural reasoning to produce a precise, actionable plan.