The Fused Expert Discovery: A Pivotal Moment in the GLM-5 GGUF Integration

Introduction

In the sprawling effort to deploy the GLM-5 model on vLLM using GGUF quantization, one message stands out as a moment of critical discovery. Message [msg 1550] is deceptively brief — a single bash command and its output — but it resolves a fundamental ambiguity about how the model's Mixture-of-Experts (MoE) weights are structured. This knowledge directly determines how the vLLM GGUF loader patch must be written, and it reveals an important design choice in the GLM-5 architecture that differs from what the assistant initially assumed.

The Message

The assistant writes:

Interesting! The HF model's state dict shows model.layers.77.mlp.experts.gate_up_proj (fused) and model.layers.77.mlp.experts.down_proj — not individual experts.0.gate_proj, experts.0.up_proj. This is the fused MoE format. Let me check:

>

``bash ssh root@10.1.230.174 '~/ml-env/bin/python3 -c " import torch from transformers import AutoModelForCausalLM, AutoConfig config = AutoConfig.from_pretrained(\"zai-org/GLM-5\", trust_remote_code=True) with torch.device(\"meta\"): model = AutoModelForCausalLM.from_config(config, trust_remote_code=True) keys = list(model.state_dict().keys()) # Show expert-related keys for k in keys: if \"expert\" in k and \".3.\" in k: print(f\" {k}\") " 2>&1' ``

>

Output: `` model.layers.3.mlp.experts.gate_up_proj model.layers.3.mlp.experts.down_proj model.layers.3.mlp.shared_experts.gate_proj.weight model.layers.3.mlp.shared_experts.up_proj.weight model.layers.3.mlp.shared_experts.down_proj.weight ``

Context: The GGUF Integration Crisis

To understand why this message matters, we must step back to the broader crisis that precipitated it. The entire session had been pursuing deployment of GLM-5 using NVFP4 quantization via SGLang. After weeks of benchmarking, profiling, and optimization — including diagnosing KV cache cast bottlenecks, implementing a gather-then-cast patch, and exploring expert parallelism — the user made a decisive pivot. The NVFP4 path was abandoned in favor of GGUF UD-Q4_K_XL quantization deployed on vLLM.

This pivot immediately hit a wall. The assistant discovered that neither the transformers library (v5.2.0) nor the installed gguf-py (v0.17.1) supported the glm-dsa architecture used by GLM-5. The GGUF file could not be loaded at all. Multiple GitHub issues confirmed that DeepSeek and GLM GGUF models consistently failed with ValueError: GGUF model with architecture deepseek2 is not supported yet.

The user explicitly rejected all alternative paths — reverting to SGLang, switching to llama.cpp, or using FP8 quantization — and directed the assistant to add GGUF support to vLLM directly. This was a non-trivial engineering task requiring deep understanding of three codebases: transformers (for architecture config registration), gguf-py (for tensor name mapping), and vLLM's GGUFModelLoader (for weight loading).

By message [msg 1550], the assistant had already made substantial progress. It had installed vLLM nightly (v0.16.0rc2.dev313), upgraded transformers to v5.3.0.dev0 from git HEAD, and installed gguf-py from llama.cpp source — which finally included the glm-dsa architecture with a complete tensor name map. The assistant had verified that the gguf-py tensor map could correctly translate most HuggingFace tensor names to GGUF tensor names, but had identified two gaps: per-expert weights and e_score_correction_bias returned None from the map, meaning they required manual handling.

The Question That Needed Answering

The central question at this point was: how exactly are the MoE expert weights structured in the HuggingFace model?

This question had profound implications for the GGUF loader patch. The GGUF format stores expert weights as individual tensors per expert — e.g., blk.0.ffn_gate.0.w, blk.0.ffn_up.0.w, blk.0.ffn_down.0.w for expert 0. But the HuggingFace model might store them differently. The assistant had previously observed that the gguf-py tensor name map returned None for paths like model.layers.3.mlp.experts.0.gate_proj, suggesting the HF model doesn't use that naming scheme.

But what naming scheme does it use? There were several possibilities:

  1. Individual per-expert tensors: experts.0.gate_proj.weight, experts.0.up_proj.weight, experts.0.down_proj.weight — the standard pattern used by models like Mixtral and DeepSeek-V2.
  2. Fused gate+up projection: experts.gate_up_proj.weight — a single tensor where the gate and up projections are concatenated along a dimension, reducing the number of weight matrices and simplifying the forward pass.
  3. Some other custom format: GLM-5 inherits from DeepSeek-V2 architecture but has custom modifications (the DSA — Dynamic Sparse Attention — mechanism), so it could use a completely different layout. The assistant's earlier exploration (messages [msg 1542] through [msg 1548]) had revealed that the HF model could be instantiated on the meta device, but the full state dict keys hadn't been examined for the expert layers specifically. The discovery in message [msg 1550] resolves this ambiguity decisively.

The Discovery: Fused Gate-Up Projection

The output shows that for layer 3 (which is an MoE layer — GLM-5 uses a hybrid architecture with both dense and MoE layers), the expert weights are stored as:

Why This Matters: The GGUF Mapping Implications

This discovery has direct, practical consequences for the GGUF loader patch. The vLLM GGUF loader works by:

  1. Creating a dummy HuggingFace model on the meta device
  2. Iterating over the HF model's state_dict() to get all tensor names
  3. Using the gguf-py tensor name map to translate each HF name to a GGUF name
  4. Loading the corresponding GGUF tensor from the file For the expert weights, the GGUF file will have individual per-expert tensors like blk.3.ffn_gate.0.w, blk.3.ffn_up.0.w, blk.3.ffn_down.0.w (for expert 0 in layer 3). But the HF model has a single fused gate_up_proj tensor covering all experts. This means the loader cannot simply map one-to-one — it must:
  5. Split the fused gate_up_proj into separate gate and up components
  6. Slice per-expert weights from the combined tensor
  7. Map each slice to the corresponding GGUF expert tensor This is exactly the pattern used by the existing deepseek_v2 and deepseek_v3 handling in vLLM's GGUF loader. The assistant can now reuse that logic, adapting it for the GLM-5 tensor naming conventions. The shared experts, on the other hand, use separate gate_proj, up_proj, and down_proj tensors — these can be mapped more straightforwardly, similar to dense MLP layers.

Assumptions Made and Corrected

This message reveals an important correction to the assistant's earlier assumptions. In message [msg 1542], the assistant had tested the gguf-py tensor name map against HF names like model.layers.3.mlp.experts.0.gate_proj — which returned None. This assumed that the HF model used individual per-expert tensors with an index (.0., .1., etc.). The discovery in message [msg 1550] shows that this assumption was wrong: the HF model uses a fused format without per-expert indexing.

The assistant's earlier reasoning was reasonable — many MoE models (like Mixtral) do use individual per-expert tensors. But GLM-5, inheriting from DeepSeek-V2, uses the fused format. This is a design choice that reduces the number of weight matrices and improves memory efficiency, but it complicates the GGUF loading process because the fusion must be reversed.

Input Knowledge Required

To fully understand this message, one needs:

  1. GLM-5 architecture knowledge: The model uses a hybrid dense-MoE architecture where some layers have MoE experts and others have dense MLP. Layer 3 happens to be an MoE layer, as shown by the presence of mlp.experts keys.
  2. DeepSeek-V2 lineage: GLM-5's MoE implementation inherits from DeepSeek-V2, which uses the fused gate_up_proj format. Understanding this lineage helps predict the tensor layout.
  3. GGUF tensor format knowledge: GGUF stores per-expert weights individually, requiring the fused HF tensors to be split and sliced during loading.
  4. vLLM GGUF loader architecture: The loader creates a dummy HF model and maps its state dict keys to GGUF tensor names. This is the mechanism that makes the discovery relevant.
  5. PyTorch state dict conventions: Understanding that model.state_dict() returns all learnable parameters with their hierarchical names, and that .weight suffixes indicate nn.Linear or nn.Embedding layers.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The HF expert tensor naming scheme: model.layers.{n}.mlp.experts.gate_up_proj (fused) and model.layers.{n}.mlp.experts.down_proj for MoE layers.
  2. The shared expert naming scheme: model.layers.{n}.mlp.shared_experts.gate_proj.weight, model.layers.{n}.mlp.shared_experts.up_proj.weight, model.layers.{n}.mlp.shared_experts.down_proj.weight — with separate gate/up/down and explicit .weight suffixes.
  3. The fused format confirmation: The MoE experts use a fused gate+up projection, not individual per-expert tensors. This is the key architectural detail that determines how the GGUF loader must handle expert weights.
  4. The absence of per-expert indexing: There is no experts.0, experts.1, etc. in the HF state dict. The expert dimension is implicit within the fused tensor, and the splitting must be done programmatically based on the model configuration (number of experts, top-k routing, etc.).
  5. A validation target for the patch: Once the GGUF loader patch is written, the assistant can verify correct weight loading by comparing the loaded model's state dict structure against this known HF structure.

The Thinking Process

The assistant's reasoning in this message is a model of systematic debugging. Let me trace the thought process:

Step 1: Pattern recognition. The assistant notices that the full state dict keys (from message [msg 1549]) show experts.gate_up_proj rather than experts.0.gate_proj. This is immediately recognized as significant — it's the "fused MoE format."

Step 2: Hypothesis formation. The assistant formulates a hypothesis: the MoE experts use fused gate+up projection, not individual per-expert tensors. This hypothesis is based on the pattern observed in the first 30 state dict keys.

Step 3: Targeted verification. Rather than examining the entire state dict (which has 1629 parameters), the assistant writes a focused query: filter for keys containing "expert" and layer "3" (an MoE layer). This is efficient — it targets exactly the information needed without overwhelming output.

Step 4: Result interpretation. The output confirms the hypothesis. The assistant now knows the exact naming scheme for both MoE experts and shared experts.

Step 5: Integration with prior knowledge. This discovery slots into the broader patch plan. The assistant now knows that the GGUF loader must handle the fused-to-split conversion for expert weights, similar to how deepseek_v2 and deepseek_v3 are handled in vLLM's existing code.

The message is titled "Interesting!" — a rare exclamation that signals genuine surprise. The assistant had been working under the assumption of individual per-expert tensors (as shown by the test names in message [msg 1542]), and this discovery required a mental model update.

Broader Significance

This message, while technically focused on tensor naming, represents a broader pattern in systems integration work: the critical moment when an assumption meets reality. The assistant had been planning the GGUF patch based on an incomplete mental model of the HF model's structure. Message [msg 1550] corrected that model before any code was written, saving what could have been hours of debugging a patch built on wrong assumptions.

The fused expert format also has implications beyond the GGUF loader. It affects how the model handles expert parallelism (EP), how weights are sharded across devices, and how the forward pass is structured. The assistant's earlier exploration of expert parallelism (in segments 8 and 9) might have benefited from this knowledge — the fused format means that EP requires splitting the fused tensor, which has different memory and communication characteristics than EP with individual expert tensors.

Conclusion

Message [msg 1550] is a textbook example of a targeted diagnostic inquiry. In a single bash command, the assistant resolves a critical ambiguity about the GLM-5 model's internal structure, confirming that the MoE experts use a fused gate-up projection format rather than individual per-expert tensors. This knowledge directly shapes the GGUF loader patch, enabling the assistant to reuse existing DeepSeek-V2/V3 patterns for handling fused expert weights. The message also demonstrates the importance of verifying assumptions against reality — the assistant's earlier assumption of per-expert indexing was incorrect, and catching this before writing the patch prevented a costly mistake. In the broader narrative of the GLM-5 deployment effort, this message is the moment when the path forward becomes clear, and the technical challenge shifts from "what needs to be done" to "how to do it correctly."