The Phantom Tensor: Debugging a GGUF Weight Loading KeyError on 8× Blackwell GPUs

Introduction

In the high-stakes world of deploying large language models on cutting-edge hardware, the gap between a model that loads successfully and one that crashes can be a single corrupted string. This article examines a pivotal debugging message from an opencode coding session where an AI assistant was attempting to deploy GLM-5—a 744-billion-parameter Mixture-of-Experts model—on eight NVIDIA RTX PRO 6000 Blackwell GPUs using GGUF quantization and vLLM. The subject message, message 1779, captures the moment the assistant discovered the root cause of a persistent crash: a KeyError involving a tensor name that should never have existed.

The Subject Message

The message begins with the assistant assessing the state of the remote inference container:

Good — no vLLM running, GPUs are free, GGUF file and patched files are in place. The vllm_serve2.log was deleted but vllm_serve.log still exists. Let me check the latest log that remains and also check the state of all our patches.

The assistant then executes two bash commands. First, it tails the surviving log:

ssh root@10.1.230.174 'tail -80 /tmp/vllm_serve.log' 2>&1

This reveals a stack trace from vLLM's engine initialization, showing the model executor failing during construction. The second command greps for errors:

ssh root@10.1.230.174 'grep -n "Traceback\|Error\|Exception\|KeyError" /tmp/vllm_serve.log | tail -20' 2>&1

The critical output:

316:(Worker_TP1 pid=39517) ERROR 02-20 00:22:03 [multiproc_executor.py:787] KeyError: 'model.layers.0.self_attn.indexer.qweight_types_proj.qweight_type'

This single line—a KeyError for a tensor named qweight_types_proj.qweight_type—is the smoking gun that the assistant has been hunting across multiple debugging rounds.

The Broader Context

To understand why this message matters, one must appreciate the journey that led here. The assistant had been working for hours across multiple segments to deploy GLM-5 on a machine with eight RTX PRO 6000 Blackwell GPUs (SM120 architecture, compute capability 12.0). The deployment path had already required:

• Patching vLLM's GGUF loader (gguf_loader.py) to support the glm_moe_dsa architecture, which vLLM did not natively recognize.
• Fixing a fundamental bug where the kv_b_proj MLA tensor was never loaded from GGUF files because llama.cpp splits it into attn_k_b and attn_v_b, but vLLM's name map only looked for attn_kv_b.
• Creating a new Triton MLA Sparse attention backend (triton_mla_sparse.py) because no existing MLA backend supported SM120 combined with sparse attention (DSA indexer) and the specific head dimensions required by GLM-5 (qk_nope_head_dim=192).
• Patching the weight iterator (weight_utils.py) to fix a bug where name.replace("weight", "qweight") corrupted parameter names containing "weight" as a substring. The last of these fixes—the suffix-only rename—had been deployed in the immediately preceding messages ([msg 1766] through [msg 1768]). But the assistant had not yet verified whether it actually resolved the crash. The previous launch attempt (logged to vllm_serve2.log) had selected the new TRITON_MLA_SPARSE backend successfully but then crashed during weight loading. That log was now deleted, so the assistant turned to the earlier vllm_serve.log to understand the failure pattern.

What the Error Reveals

The KeyError for model.layers.0.self_attn.indexer.qweight_types_proj.qweight_type is a beautiful artifact of a string-manipulation bug. The original parameter name in the model is model.layers.0.self_attn.indexer.weights_proj.weight. The GGUF quantization iterator in vLLM's weight_utils.py attempts to map weight tensor names to their quantized counterparts by replacing "weight" with "qweight". But the iterator used a naive global replacement: name.replace("weight", "qweight"). Because the parameter name contains "weights" (plural) as a substring, this replacement transforms weights_proj.weight into qweights_proj.qweight. The first occurrence of "weight" is inside "weights", producing the nonsensical "qweights". The second occurrence is the actual ".weight" suffix, producing "qweight".

The assistant's earlier fix ([msg 1766]) changed this to only replace the .weight suffix using name.replace(".weight", ".qweight"). However, the error in the log was from before that fix was applied—the log file vllm_serve.log captured the first launch attempt, not the second. The vllm_serve2.log (second attempt, with the fix) had been deleted, so the assistant was examining historical data. The KeyError in the log was the original bug, not the current state.

This creates a subtle but important situation: the assistant sees an error in the log that it believes it has already fixed. The question is whether the fix actually works, or whether there are additional issues lurking. The assistant cannot tell from this log alone because it predates the fix.

Assumptions and Potential Mistakes

The message reveals several assumptions the assistant is making:

1. The fix is correct: The assistant assumes that changing name.replace("weight", "qweight") to name.replace(".weight", ".qweight") is sufficient. While this fixes the immediate corruption, it does not address the deeper architectural issue: the weights_proj parameter in the model's Indexer class is created with quant_config=None (meaning it expects unquantized weights), but the GGUF file stores it as Q4_K quantized. Even with correct name mapping, the mismatch between quantized storage and unquantized expectation could cause downstream failures.
2. The log is representative: The assistant is examining vllm_serve.log (the first run) to understand the failure mode of the second run (vllm_serve2.log). This is a reasonable heuristic—both runs involve the same model and patches—but the second run had additional patches deployed (the suffix-only rename fix and the new Triton MLA Sparse backend). The failure mode could have changed.
3. The deleted log is not critical: The assistant does not attempt to recover vllm_serve2.log from journald, console output, or other sources. It accepts the deletion and works with what remains.
4. The GPUs being free is good news: The assistant notes that all eight GPUs show 0 MiB memory usage, meaning the crash was clean—no GPU memory leak or hung process. This is correct but also means the model failed early in initialization, before allocating significant VRAM.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of:

• GGUF format: The binary format used by llama.cpp for quantized model weights, which stores tensors with type metadata (e.g., Q4_K, Q8_0) alongside the raw data.
• vLLM's weight loading pipeline: How vLLM maps GGUF tensor names to model parameter names using a name map, iterates over quantized tensors via gguf_quant_weights_iterator, and handles dequantization.
• The GLM-5 model architecture: Specifically the Indexer module used for sparse attention (DSA), which contains a weights_proj linear layer that projects attention scores to select top-k tokens.
• Tensor parallelism (TP): The model is split across 8 GPUs, so weight loading must handle sharding. The error occurs on Worker_TP1, indicating the TP worker process crashed.
• The history of the weight → qweight bug: How a naive string replacement in gguf_quant_weights_iterator corrupts parameter names, and why the suffix-only fix is the correct approach.

Output Knowledge Created

This message produces several important pieces of knowledge:

1. The exact error signature: KeyError: 'model.layers.0.self_attn.indexer.qweight_types_proj.qweight_type' — this becomes the target for the next debugging round.
2. Confirmation of system state: GPUs are clean, the GGUF file exists (402GB), patched files are in place, and no vLLM process is running.
3. The log file situation: vllm_serve2.log is gone, vllm_serve.log remains with the pre-fix error.
4. The error location: The crash occurs in multiproc_executor.py:787 during weight loading on the TP worker, not during model construction or attention backend initialization.
5. The error propagates through PyTorch distributed: The sendBytes exception in the traceback indicates the worker process failed and the error was communicated back through PyTorch's c10d distributed communication layer.

The Thinking Process

The assistant's reasoning in this message is methodical and diagnostic. It follows a clear pattern:

1. State assessment: Before diving into debugging, the assistant checks the overall system health—are GPUs free? Is the model file present? Are patches in place? This prevents wasted effort debugging against a stale or corrupted state.
2. Log triage: The assistant discovers vllm_serve2.log is missing and immediately pivots to the available log. It does not panic or ask the user for guidance—it adapts to the available data.
3. Targeted extraction: Rather than reading the entire 71KB log, the assistant uses grep to extract only lines containing error keywords (Traceback, Error, Exception, KeyError). This is an efficient debugging technique that surfaces the critical information quickly.
4. Pattern recognition: The assistant recognizes the qweight_types_proj.qweight_type error as the manifestation of the weight → qweight replacement bug it had previously identified and attempted to fix. This connects the current observation to the earlier analysis.
5. Prioritization: The message ends with the error exposed but not yet resolved. The assistant implicitly prioritizes understanding the error over immediately attempting another fix. The next message ([msg 1780]) will formalize this into a todo list.

Conclusion

Message 1779 captures a quintessential debugging moment in complex ML infrastructure work. The assistant is not writing code or deploying a fix—it is looking, checking, and understanding. The discovery of the qweight_types_proj.qweight_type KeyError is the culmination of hours of patching, and it reveals that the root cause is deeper than a simple string replacement fix. The weights_proj parameter's dual nature—created as unquantized in the model definition but stored as Q4_K in the GGUF file—means that even with correct name mapping, the weight loading pipeline must be taught how to handle this mismatch. The assistant's methodical approach—checking state, triaging logs, extracting errors, and connecting observations to prior knowledge—is a model of disciplined debugging in a distributed, multi-GPU environment where a single wrong tensor name can bring down the entire system.