Reading the Blueprint: How One Bash Command Unlocked vLLM's Weight Loading Architecture for GLM-5 GGUF Support

The Message

[assistant] [bash] ssh root@10.1.230.174 'sed -n "1340,1440p" ~/ml-env/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py' num_experts=self.config.n_routed_experts, num_redundant_experts=0, )

>

def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: rocm_aiter_moe_shared_expert_enabled = ( rocm_aiter_ops.is_fusion_moe_shared_experts_enabled() ) stacked_params_mapping = [ # (param_name, shard_name, shard_id) ("gate_up_proj", "gate_proj", 0), ("gate_up_proj", "up_proj", 1), ] ...

At first glance, this message appears to be a routine code inspection — a developer peeking at a specific function in a large codebase. But in the context of the broader GLM-5 deployment saga, this single sed command represents a critical turning point. It is the moment when the assistant, having already diagnosed the failure of the NVFP4 quantization path and pivoted to GGUF deployment on vLLM, digs into the deepest layer of the problem: understanding exactly how vLLM's model loading infrastructure expects weights to be named and structured. Without this knowledge, the GGUF loader patch would be built on guesswork. With it, the assistant can craft a precise, targeted modification to bridge the gap between GLM-5's GGUF tensor layout and vLLM's internal weight consumption model.

Context: The GGUF Pivot and the Research Phase

To understand why this message matters, one must appreciate the journey that led to it. The assistant had spent the better part of a day battling the NVFP4 quantization path for the GLM-5 model — a 79-layer, 8-expert Mixture-of-Experts architecture running on dual RTX PRO 6000 Blackwell GPUs. After exhaustive profiling, kernel analysis, and optimization attempts (documented across segments 7 through 11), the user made the decisive call to abandon NVFP4 and pivot to GGUF quantization using Unsloth's UD-Q4_K_XL format, deployed on vLLM.

But the pivot hit an immediate wall. When the assistant tried to set up vLLM with a GGUF model, it discovered that neither transformers (v5.2.0) nor the installed gguf-py (v0.17.1) included the glm-dsa architecture used by GLM-5. Multiple GitHub issues confirmed that DeepSeek and GLM GGUF models consistently failed with ValueError: GGUF model with architecture deepseek2 is not supported yet. The user explicitly rejected all alternative paths — reverting to sglang, llama.cpp, or FP8 — and directed the assistant to add GGUF support to vLLM directly.

This launched an intensive multi-front research effort. The assistant needed to understand three separate codebases simultaneously: the transformers GGUF config mapping system, vLLM's GGUFModelLoader, and the GLM-5 GGUF tensor structure as defined by the latest gguf-py from llama.cpp HEAD. Messages 1542 through 1554 show this research in real time: probing the tensor name map, discovering that kv_b_proj is split into separate attn_k_b and attn_v_b in GGUF format, verifying that expert weights use fused gate_up_proj format, and confirming that the e_score_correction_bias tensor needs manual mapping.

By message 1554, the assistant had established the key facts:

What This Message Reveals

The sed command extracts lines 1340 through 1440 of deepseek_v2.py, the file implementing the DeepSeekV2 architecture in vLLM. The output reveals the tail end of a model constructor (showing num_experts and num_redundant_experts configuration) and the beginning of the load_weights method.

The critical discovery is the stacked_params_mapping list:

stacked_params_mapping = [
    # (param_name, shard_name, shard_id)
    ("gate_up_proj", "gate_proj", 0),
    ("gate_up_proj", "up_proj", 1),
]

This is the pattern that tells vLLM how to handle fused MoE parameters. When the GGUF loader presents a weight tensor named gate_up_proj (the fused format), vLLM knows to split it into two shards: shard 0 goes to gate_proj and shard 1 goes to up_proj. This is exactly the pattern used by GLM-5's GGUF file, where expert weights are stored as fused gate_up_proj tensors.

The output also confirms that load_weights returns a set[str] — the set of parameter names that were actually loaded — which is used by vLLM's weight loading infrastructure to verify that all expected parameters were accounted for.

Why This Knowledge Was Essential

Without seeing this stacked_params_mapping, the assistant would have been operating with an incomplete mental model of how vLLM consumes weights. The GGUF loader's job is to produce (hf_name, tensor) pairs, but those pairs must match exactly what the model's load_weights() expects. If the assistant had assumed that vLLM expects separate gate_proj and up_proj tensors (as the HF model's state dict might suggest), the patch would have attempted to split the fused GGUF tensor during the GGUF loading phase — duplicating logic that vLLM already handles internally.

The message also reveals the structure of load_weights itself: it takes an Iterable[tuple[str, torch.Tensor]] and returns a set[str]. This tells the assistant that the weight loading is streaming — weights arrive as an iterable of name-tensor pairs — and the return value is used for validation. This is important because the GGUF loader must produce exactly the names that load_weights expects, no more and no less.

Assumptions and Potential Pitfalls

The assistant made several implicit assumptions in this investigation. First, it assumed that the GLM-5 model in vLLM would follow the same load_weights pattern as DeepSeekV2. This is a reasonable assumption — GLM-5 uses the same MLA (Multi-head Latent Attention) architecture and similar MoE structure — but it's not guaranteed. The GLM-5 model might have its own load_weights implementation with different parameter name expectations.

Second, the assistant assumed that the stacked_params_mapping is the only mechanism for handling fused weights. In practice, vLLM also uses packed_modules_mapping and other weight transformation patterns that might need to be considered.

Third, the assistant assumed that the DeepSeekV2 model file was the right reference. GLM-5 uses model_type: "glm_moe_dsa", which might map to a different model class in vLLM. However, given that GLM-5's architecture is derived from DeepSeek's MLA design, this is the most relevant reference point available.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of vLLM's architecture: Understanding that vLLM separates weight loading into two phases — the GGUF loader maps GGUF tensor names to HF parameter names, and then the model's load_weights() method consumes those parameters. This two-phase design means the GGUF loader doesn't need to know the internal structure of the model; it just needs to produce the right names.
  2. Knowledge of MoE fused weight formats: Understanding that Mixture-of-Experts models often fuse gate_proj and up_proj into a single gate_up_proj tensor for efficiency. The fused tensor has shape [2 * intermediate_size, hidden_size] where the first half is the gate projection and the second half is the up projection.
  3. Knowledge of the sed command: The specific sed -n "1340,1440p" invocation prints lines 1340 through 1440 of the file. This is a precise surgical extraction, not a broad search — the assistant already knew approximately where the load_weights method started.
  4. Knowledge of the broader context: Understanding that this is part of a GGUF loader patch for GLM-5, and that the assistant has already mapped out the GGUF tensor structure and identified the unmapped parameters.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The stacked_params_mapping pattern: Confirmed that vLLM's DeepSeekV2 model uses ("gate_up_proj", "gate_proj", 0) and ("gate_up_proj", "up_proj", 1) to split fused MoE weights. This means the GGUF loader should present the fused gate_up_proj tensor as-is, and let vLLM's load_weights() handle the splitting.
  2. The load_weights interface: Confirmed that the method takes Iterable[tuple[str, torch.Tensor]] and returns set[str]. This tells the assistant that the GGUF loader needs to produce an iterable of name-tensor pairs, and that the return value is used for validation.
  3. The model configuration parameters: The output shows num_experts=self.config.n_routed_experts and num_redundant_experts=0, confirming that the model config carries expert count information that the GGUF loader might need to access.
  4. Confirmation that the approach is viable: By seeing that vLLM already handles fused→split weight transformation internally, the assistant can simplify the GGUF loader patch — it doesn't need to split the fused tensors during GGUF loading.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is a textbook example of systematic reverse engineering. The thought process goes something like this:

"I know the GGUF file has fused gate_up_proj tensors for expert weights. I know the HF model's state dict also shows experts.gate_up_proj (fused). But how does vLLM's model actually consume these? Does it expect the fused tensor and split it internally, or does it expect separate tensors? Let me look at the DeepSeekV2 model's load_weights() method, since GLM-5 is architecturally similar."

The choice of sed with a specific line range (1340-1440) is itself revealing. The assistant didn't search for load_weights with grep — it already knew the approximate location. This suggests the assistant had previously scanned the file structure, perhaps during an earlier investigation, and remembered that load_weights was in that range. Alternatively, the assistant might have used grep -n "def load_weights" in a previous (unshown) step to find the line number, then used sed to extract a generous window around it.

The output is deliberately truncated — the ... at the end indicates the assistant only needed to see the beginning of the method to confirm the stacked_params_mapping pattern. The full method extends for many more lines, handling expert weight loading, shared experts, and other parameters. But the assistant only needed this one critical pattern to validate its approach.

Impact on the Broader Effort

This message directly informed the GGUF loader patch that the assistant was about to write. With the stacked_params_mapping confirmed, the assistant could design the patch to:

  1. Map GGUF tensor names to HF parameter names using the gguf-py name map for standard parameters
  2. Manually map expert weights (experts.gate_up_proj, experts.down_proj, gate.e_score_correction_bias) using the same pattern as the existing DeepSeekV2 handling
  3. Handle the kv_b_proj split by reassembling attn_k_b and attn_v_b from the GGUF file into a single kv_b_proj tensor
  4. Pass the fused gate_up_proj tensor directly to load_weights(), trusting vLLM's internal splitting mechanism The message also validated the assistant's overall approach: instead of trying to modify the GGUF file or create an entirely new model class, the assistant could work within vLLM's existing infrastructure, extending the GGUF loader to handle the glm_moe_dsa model type while reusing the DeepSeekV2 weight loading logic. In the next message ([msg 1556]), the assistant immediately follows up by checking how kv_b_proj is loaded, confirming that vLLM expects it as a single tensor named kv_b_proj.weight — which means the GGUF loader must concatenate the split attn_k_b and attn_v_b tensors before presenting them to the model. This completes the picture and enables the assistant to write a comprehensive, correct patch.

Conclusion

Message 1555 is a masterclass in targeted code investigation. In a single, precisely aimed sed command, the assistant extracted the exact piece of information needed to bridge the gap between GGUF tensor structure and vLLM weight consumption. It's a reminder that in complex system integration work, the most valuable insights often come not from broad searches or AI-generated guesses, but from reading the actual source code — the definitive documentation of how a system behaves. The stacked_params_mapping pattern, once confirmed, unlocked the entire GGUF loader patch design and saved the assistant from a potentially wrong approach that would have attempted to split fused tensors at the wrong layer of the architecture.