The Diagnostic Pivot: Tracing Garbage Output to Expert Weight Unmerging in a GLM-5 GGUF Deployment

In the sprawling effort to deploy a 402 GB GLM-5 GGUF model quantized with the UD-Q4_K_XL scheme across eight NVIDIA Blackwell GPUs, message [msg 1913] marks a critical inflection point. After successfully loading the model onto the GPUs and starting the vLLM inference server, the assistant was confronted with a deeply puzzling result: the model generated coherent-looking but semantically meaningless tokens, with flat log-probability distributions that betrayed a fundamentally broken computation. This message captures the moment when the assistant, having systematically eliminated several plausible causes, pivots to a new hypothesis about the root cause — and in doing so, reveals the methodical, forensic approach that characterizes the entire debugging effort.

The Context: A Model That Loads but Cannot Think

To understand the significance of [msg 1913], one must appreciate what preceded it. The assistant had spent dozens of messages wrestling with the GLM-5 deployment — a model based on the DeepSeek-V2 architecture with Multi-head Latent Attention (MLA) and Mixture-of-Experts (MoE) layers, quantized to GGUF's Q4_K format and split across ten files that were later merged into a single 402 GB monolithic GGUF file. The deployment required extensive patching of vLLM's gguf_loader.py and weight_utils.py to support the glm_moe_dsa architecture, including force-dequantizing tensors whose model parameters were created with quant_config=None (such as the Indexer's weights_proj and the MoE routing gate), and adding skip logic for unknown parameter names.

After these patches, the model loaded without errors and the server began serving requests. But the output was incoherent — tokens like BWennes, IRS, and Promo appeared repeatedly across positions, with log-probability values clustered around -5 to -6 for the top candidates, indicating near-uniform distributions. This is the hallmark of a model whose weights are not contributing meaningful signal to the forward pass.

Interpreting the Evidence: What the Logprobs Reveal

Message [msg 1912] had just delivered the most damning evidence yet. The assistant crafted a diagnostic prompt — "1 2 3 4 5 6 7 8 9 10" — designed to test whether the model could predict obvious continuations. A properly functioning language model should assign very high probability (logprob near 0) to the token 2 following 1, 3 following 2, and so on. Instead, the server returned logprobs of -20 to -24 for these tokens, while the generated continuation was garbage: 1 2 3 4 5 6 7 8 9 10iryiryIRSIRSIRS.

Message [msg 1913] opens with the assistant's analysis of this result:

Very telling. The logprobs for known tokens like 2, 3, 4 (that should follow 1) are around -20 to -24, which is extremely low. The model should easily predict these with logprobs near 0. And the generated tokens are random with logprobs around -4. This means the model's first layers are producing garbage hidden states.

This is a crucial diagnostic inference. The assistant recognizes that logprobs of -20 to -24 for obvious continuations are not merely "slightly wrong" — they indicate that the hidden state representations at the earliest layers are corrupted. The model is not making subtle errors; it is fundamentally failing to transform input tokens into meaningful feature vectors. The flat distribution of generated tokens (all around -4 logprob) confirms that the output layer is essentially picking randomly from a nearly uniform distribution over the vocabulary.

The Methodical Elimination of Hypotheses

What makes [msg 1913] so instructive is what it reveals about the assistant's debugging methodology. Before arriving at this message, the assistant had already tested and eliminated several plausible causes:

GGUF dequantization kernel correctness ([msg 1908]): The assistant wrote a test comparing CPU-based dequantization (using gguf-py's quants.dequantize) against GPU-based dequantization (using vLLM's ggml_dequantize CUDA kernel). The maximum difference was 0.00012 — well within float16 precision. The dequantization kernel works correctly on SM120 Blackwell GPUs.
Triton MLA attention backend compatibility (<msg id=1909-1911>): The assistant checked whether the TritonMLABackend supports Blackwell's compute capability. The backend's supports_compute_capability method was found to accept all device capabilities, and the supported dtypes include float16. The KV cache dtype was confirmed as auto, which resolves to the model's float16 dtype.
Weight name mapping correctness: The assistant had verified that the GGUF-to-HF name mapping produces correct tensor names, and that the force-dequantized tensors (like the MoE gate.weight) are yielded with the correct parameter names that match the model's ReplicatedLinear(quant_config=None) expectations. With these possibilities eliminated, the assistant needed a new hypothesis. Message [msg 1913] represents that pivot.

The New Hypothesis: Expert Weight Unmerging

The assistant's next hypothesis is stated directly:

This confirms the model weights are not being loaded correctly for the core computation. Let me check a more fundamental issue — maybe the expert weights (3D merged tensors) are not being unmerged correctly by vLLM's GGUF loader for the MoE layers.

This is a sophisticated diagnostic leap. The GLM-5 model uses Mixture-of-Experts layers where each MLP block contains multiple "expert" feed-forward networks. In the GGUF format, the expert weights for a given layer (gate, up, and down projections) are stored as 3D merged tensors — for example, ffn_gate_exps.weight has shape [num_experts, intermediate_size, hidden_size] rather than separate 2D matrices for each expert. vLLM's GGUF loader must "unmerge" these 3D tensors back into individual per-expert weight matrices that match the model's architecture definition.

The assistant's suspicion is that this unmerging process might be incorrect. If the expert weights are sliced or indexed incorrectly during loading, the MoE layers would receive corrupted parameters, which would propagate through the entire forward pass and produce the observed garbage output.

To investigate, the assistant runs a targeted grep command on the patched gguf_loader.py:

ssh root@10.1.230.174 'grep -n "ffn_gate_exps\|ffn_down_exps\|ffn_up_exps\|experts\|merge_expert" /root/ml-env/lib/python3.12/site-packages/vllm/model_executor/model_loader/gguf_loader.py | head -30'

The output reveals the relevant mapping code:

140:                gguf_to_hf_name_map[f"blk.{idx}.ffn_down_exps.weight"] = (
141:                    f"model.layers.{idx}.mlp.experts.0.down_proj.weight"
143:                gguf_to_hf_name_map[f"blk.{idx}.ffn_gate_exps.weight"] = (
144:                    f"model.layers.{idx}.mlp.experts.0.gate_proj.weight"
146:                gguf_to_hf_name_map[f"blk.{idx}.ffn_up_exps.weight"] = (
147:                    f"model.layers.{idx}.mlp.experts.0.up_proj.weight"

At first glance, the mapping looks correct — each expert weight tensor is mapped to experts.0.<proj>.weight. But the assistant's suspicion is that the loading logic that handles these 3D tensors might not correctly slice them for the model's expected format. The mapping only tells the loader which parameter name to use; the actual unmerging logic (splitting the 3D tensor along the expert dimension and assigning each slice to the corresponding expert index) is a separate code path that could contain bugs.

The Reasoning Process: A Window into Expert Debugging

What makes [msg 1913] particularly valuable as a case study is the transparency of the assistant's reasoning. The message does not simply present a conclusion; it walks through the evidence and the logic:

Observation: Logprobs for expected tokens are -20 to -24, far from the near-0 values a correct model would produce.
Inference: The hidden states at early layers are corrupted — the model cannot transform input into meaningful representations.
Elimination: The dequantization kernel is verified correct. The attention backend supports Blackwell. The weight name mapping is correct.
New hypothesis: The MoE expert weight unmerging might be the culprit, since these are 3D tensors that require special handling.
Investigation: Examine the GGUF loader's expert weight mapping code. This chain of reasoning demonstrates a systematic debugging approach: start with the most direct evidence (logprobs), infer the level at which the failure occurs (early layer hidden states), eliminate already-tested hypotheses, and formulate a new testable hypothesis about the remaining untested component.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message that are worth examining:

Assumption 1: The failure is in weight loading, not in computation. The assistant assumes that because the dequantization kernel works correctly on SM120 and the attention backend is compatible, the remaining suspect is weight loading. This is reasonable but not exhaustive — there could be issues in other computational kernels (e.g., the MoE routing logic, the RMSNorm implementation, or the RoPE computation) that are not related to weight loading.

Assumption 2: Expert weight unmerging is the most likely remaining cause. The assistant focuses specifically on the 3D merged expert tensors. This is a good hypothesis because: (a) the GLM-5 architecture uses MoE, (b) 3D tensor handling is a common source of bugs in GGUF loaders, and (c) the failure mode (garbage output from early layers) is consistent with incorrect expert weights. However, other possibilities exist — such as the kv_b_proj tensor parallelism sharding mismatch that the assistant had been investigating earlier.

Assumption 3: The mapping code shown by grep is the relevant code path. The assistant assumes that examining the mapping in gguf_loader.py will reveal the bug. But the actual unmerging logic might be in a different file (e.g., weight_utils.py or the DeepSeek-V2 model implementation), and the mapping alone may not show the full picture.

Input Knowledge Required

To fully understand [msg 1913], the reader needs knowledge of:

GGUF quantization format: How Q4_K quantized weights are stored and dequantized, and the distinction between the raw packed format and the dequantized float representation.
vLLM's GGUF loader architecture: How gguf_loader.py maps GGUF tensor names to HuggingFace parameter names, and how weight_utils.py handles the actual weight iteration and loading.
Mixture-of-Experts (MoE): The concept of multiple expert feed-forward networks per layer, and how expert weights are stored as 3D merged tensors in GGUF format.
Multi-head Latent Attention (MLA): The attention mechanism used by DeepSeek-V2 and GLM-5, which uses a compressed KV cache representation and split kv_b_proj weights.
Tensor parallelism (TP): How model parameters are sharded across multiple GPUs, and the distinction between ColumnParallelLinear (shards output dimension) and ReplicatedLinear (replicates across ranks).
Log-probability interpretation: Understanding that logprobs near 0 indicate high confidence, while logprobs of -20 indicate extremely low probability (exp(-20) ≈ 2×10⁻⁹).

Output Knowledge Created

This message produces several forms of output knowledge:

A confirmed diagnostic conclusion: The model's early layers produce garbage hidden states, ruling out issues that only affect later layers (e.g., sampling logic, output embedding corruption).
A testable hypothesis: Expert weight unmerging may be incorrect. This hypothesis is immediately testable by examining the unmerging code and potentially writing a diagnostic that checks whether the unmerged expert weights match expected values.
The grep output itself: The mapping code shown in the message output provides concrete evidence that the expert weight mapping exists and appears syntactically correct, narrowing the search to the actual unmerging logic rather than the name mapping.
A documented debugging methodology: The message serves as a record of the systematic elimination process, which is valuable for future debugging sessions and for understanding the state of the investigation.

The Broader Significance

Message [msg 1913] exemplifies the kind of disciplined, evidence-based debugging that characterizes the entire GLM-5 deployment effort. The assistant does not jump to conclusions or make wild guesses; instead, it carefully interprets the available evidence, eliminates already-tested hypotheses, and formulates a new hypothesis that is both testable and grounded in the specific architecture of the model.

The message also reveals the immense complexity of deploying a modern large language model on novel hardware. The interaction between GGUF quantization, tensor parallelism, MoE routing, MLA attention, and a new GPU architecture creates a vast space of potential failure modes. Each hypothesis must be tested independently, and each elimination narrows the search space. By message [msg 1913], the assistant has eliminated dequantization kernels, attention backends, and name mapping — leaving expert weight unmerging as the prime suspect.

In the subsequent messages, we will see whether this hypothesis bears fruit, or whether the assistant must pivot again to yet another potential cause. But regardless of the outcome, [msg 1913] stands as a model of clear diagnostic reasoning in the face of a deeply confusing failure mode.