The Unquantized Names Puzzle: Bridging GGUF Quantization and Model Architecture in vLLM

In the long arc of deploying a 402 GB GLM-5 GGUF model across 8× NVIDIA RTX PRO 6000 Blackwell GPUs, there are moments that look unremarkable at first glance—a simple file read, a brief comment about adding two names to a list. But message [msg 1802] is one of those pivot points where accumulated debugging crystallizes into a precise surgical fix. The assistant reads a patch file to understand how vLLM's gguf_loader.py builds its unquant_names list, with the explicit goal of adding weights_proj and gate to it. This seemingly minor action represents the culmination of a deep debugging chain that had already spanned multiple failed launches, traceback analyses, and source code inspections.

The Debugging Chain That Led Here

To understand message [msg 1802], we must first understand the crisis that preceded it. The assistant had launched the GLM-5 GGUF model on vLLM's nightly build, only to have it crash with a KeyError: 'model.layers.0.self_attn.indexer.weights_proj.qweight_type' ([msg 1791]). This was the second consecutive failure—an earlier attempt had also crashed with the same error ([msg 1789]), and the assistant had initially misdiagnosed the problem as a stale .pyc cache issue ([msg 1783]-[msg 1784]). Only after the second crash did the assistant dig into the full traceback and realize the true nature of the bug.

The traceback pointed to deepseek_v2.py:1540 in the load_weights method. The weight iterator was yielding a tensor named weights_proj.qweight_type, but the model's params_dict had no such parameter. Why? Because the Indexer class in the DeepSeek V2 model creates its weights_proj linear layer with quant_config=None ([msg 1793]). When quant_config is None, the model does not register any qweight_type, qweight, or scales sub-parameters—it expects a plain, unquantized weight tensor. But the GGUF file stored this tensor as Q4_K quantized, so the weight iterator dutifully yielded a qweight_type tensor that the model could not accept.

This is a fundamental mismatch between two representations of the same model: the GGUF quantization format and vLLM's internal model architecture. The GGUF file, created by llama.cpp's quantization tools, had quantized every tensor it could, including the indexer's weights_proj. But vLLM's Indexer class, written for the original DeepSeek V2/V3 architecture, had no concept of quantizing this particular projection. The two sides spoke different languages.

The Two-Pronged Fix Strategy

The assistant devised a two-pronged fix. First, in weight_utils.py, the gguf_quant_weights_iterator function needed to detect tensors whose HuggingFace names matched patterns for parameters created with quant_config=None, and force-dequantize them—just as it already did for the kv_b split tensors using sentinel suffixes __k_b and __v_b ([msg 1796]). The patterns were: *.mlp.gate.weight (the MoE routing gate, also created with quant_config=None at line 252 of deepseek_v2.py) and *.self_attn.indexer.weights_proj.weight (the DSA indexer weights projection at line 634). The assistant had already applied this edit in message [msg 1801].

But force-dequantizing the tensor in the iterator was only half the solution. The gguf_loader.py file also needed to know about these tensors. Specifically, it maintains an unquant_names list that tells the rest of the loading pipeline which parameters should not be expected to have quantized metadata (like qweight_type, scales, etc.). If a tensor is force-dequantized but not added to unquant_names, the loader might still try to look up its quantization metadata and fail.

This is exactly where message [msg 1802] sits: the assistant is reading the gguf_loader.py patch file to understand the exact structure of the unquant_names list, so it can properly add weights_proj and gate to it.

What the Message Actually Shows

The message is a [read] operation on a local patch file at /home/theuser/glm-kimi-sm120-rtx6000bw/gguf_loader.py.patched. The content shown is lines 505-514 of the patched file, which includes the critical section where unquant_names is built:

weight_type_map = self._get_gguf_weight_type(
    model_config, local_model_path, gguf_weights_map
)
# filter out unquantized modules to skip
unquant_names = [
    name.removesuffix(".weight")
    for name, ...

The assistant's comment preceding the read operation reveals the intent: "Now I also need to add weights_proj and gate to the unquant_names in gguf_loader.py so the GGUF quant config knows not to expect quantized parameters for these."

This is a moment of careful, methodical engineering. The assistant is not guessing—it is reading the actual source code to understand the data structure before modifying it. The unquant_names list is built by iterating over the weight_type_map and selecting only those tensors whose weight type is already F32, F16, or BF16 (i.e., already unquantized). The problem is that weights_proj and gate are Q4_K in the GGUF file, so they would NOT appear in this list naturally. The assistant needs to add them manually.

Assumptions and Knowledge Required

To understand this message, one must grasp several layers of the vLLM architecture. First, the concept of GGUF quantization types: GGUF files store tensors in various formats (F32, F16, BF16, Q4_K, Q6_K, etc.), and the loader must handle each type appropriately. Second, the unquant_names mechanism: this list tells the loader which parameters should be treated as unquantized even if the GGUF file stores them in a quantized format. Third, the model architecture itself: the Indexer class in deepseek_v2.py creates weights_proj with quant_config=None, meaning it expects unquantized weights regardless of what the GGUF file contains.

The assistant also assumes that the gate tensor (the MoE routing gate) might have the same problem—it is also created with quant_config=None at line 252 of deepseek_v2.py. This is a reasonable inference: if one quant_config=None parameter causes a crash, another one likely will too. However, the assistant had not yet confirmed that gate was actually Q4_K in the GGUF file—it was being proactive.

The Deeper Significance

This message represents a critical insight about the nature of the bug. The initial fix attempt (modifying only the suffix-based name transformation in weight_utils.py) failed because it addressed the wrong problem. The issue was not about name transformation—it was about the fundamental incompatibility between GGUF's aggressive quantization and vLLM's selective quantization support. The model architecture explicitly marks certain parameters as "do not quantize" by passing quant_config=None, but the GGUF file had quantized them anyway because llama.cpp's quantization tools don't know about vLLM's internal architecture decisions.

This is a recurring theme in the deployment of large language models: the gap between training/inference frameworks and quantization tools. Each framework has its own assumptions about which parameters can be safely quantized, and when these assumptions diverge, subtle bugs emerge. The weights_proj tensor is small—only 7168 × 64 elements, or about 1.8 million parameters in BF16—so quantizing it saves negligible space while causing disproportionate integration headaches.

Output Knowledge Created

This message, combined with the edit in [msg 1801], creates a complete fix for the KeyError crash. The force-dequantization in weight_utils.py ensures the tensor is converted to float32 before it reaches the model, and the unquant_names addition in gguf_loader.py ensures the loader doesn't expect quantized metadata for these parameters. Together, these changes bridge the gap between GGUF's quantization and vLLM's architecture.

The message also establishes a pattern for handling similar mismatches in the future. Any time a model parameter is created with quant_config=None but the GGUF file stores it quantized, the fix involves two changes: force-dequantize in the iterator, and add to unquant_names. This pattern would prove valuable later in the session when other tensor loading issues emerged.

The Thinking Process

The assistant's reasoning in this message is notably systematic. Having already fixed the weight iterator in weight_utils.py, the assistant immediately recognizes the need for a corresponding change in gguf_loader.py. This shows an understanding of the two-phase nature of GGUF weight loading: first, the iterator yields tensors (handled by weight_utils.py), and second, the loader maps those tensors to model parameters (handled by gguf_loader.py and the model's load_weights). A fix in only one phase would leave the system inconsistent.

The assistant also demonstrates careful reading of the existing code. Rather than assuming the structure of unquant_names, it reads the actual patch file to see the exact implementation. This attention to detail is what separates a working fix from a broken one—the unquant_names list uses name.removesuffix(".weight") to strip the suffix, and the assistant needs to understand this pattern to add entries correctly.

In the broader context of the session, this message is a quiet but essential step toward the eventual successful loading of the model. It would take several more rounds of debugging to resolve the incoherent output that followed (caused by a tensor parallelism sharding mismatch in kv_b_proj), but this fix was necessary groundwork. Without it, the model could not even begin loading, let alone serve requests.