Tracing the Ghost: How One Bash Command Exposed a Tensor Parallelism Sharding Bug in vLLM's GGUF Loader

The Message

[assistant] I see — the `stacked_params_mapping` loop at line 1399 handles `gate_up_proj` (gate_proj → gate_up_proj with shard_id). For `kv_b_proj`, it should fall through to the `else` block (line 1433+). Let me check what happens in that else block for kv_b_proj:
[bash] ssh root@10.1.230.174 'sed -n "1500,1550p" /root/ml-env/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py' 2>&1
                        param = params_dict[name_mapped]
                        # We should ask the weight loader to return success or
                        # not here since otherwise we may skip experts with
                        # other available replicas.
                        weight_loader = typing.cast(
                            Callable[..., bool], param.weight_loader
                        )
                        success = weight_loader(
                            param,
    ...

The Moment of Insight

This message, indexed as message 1949 in a marathon debugging session spanning over a thousand exchanges, represents a critical turning point. The assistant has just realized something fundamental about how vLLM loads GGUF-quantized weights into its tensor-parallel (TP) sharded model architecture — and the realization is that the weight loading path for the kv_b_proj parameter in the DeepSeek-V2-derived GLM-5 model bypasses the sharding logic entirely.

To understand why this single bash command — reading 50 lines of Python source code from a remote server — matters so much, we must reconstruct the chain of reasoning that led to it. The assistant was deep in the trenches of deploying a 402 GB GGUF-quantized GLM-5 model across 8 NVIDIA Blackwell GPUs, and the model was producing incoherent garbage output despite loading without any errors.

Background: The Garbage Output Mystery

The session had already overcome numerous obstacles. The assistant had patched vLLM's gguf_loader.py to support the novel glm_moe_dsa architecture, fixed a latent DeepSeek V2/V3 GGUF bug in kv_b_proj mapping, implemented a custom Triton MLA sparse attention backend for Blackwell's SM120 architecture, and force-dequantized indexer weights that were causing KeyError exceptions. After all these fixes, the model loaded successfully and the server began serving requests.

But the output was nonsense — flat log-probability distributions producing garbage tokens like "NEGNibTagounderTag" when given a single-token prompt. This was not a subtle quality degradation; it was a complete failure of the model to produce coherent text.

The assistant's systematic investigation had ruled out several possible causes:

• The GGUF dequantization kernel was verified to work correctly on SM120
• The weight name mapping between GGUF tensor names and model parameters was correct
• vLLM's bundled FlashAttention was available and functional on Blackwell The investigation had narrowed to a single suspect: the kv_b_proj weight loading. This parameter is special in the DeepSeek-V2 MLA (Multi-head Latent Attention) architecture — it is reassembled from two separate GGUF tensors (k_b and v_b) into a combined [28672, 512] tensor. The model code declares kv_b_proj as a ColumnParallelLinear, which under tensor parallelism (TP=8) should shard the output dimension from 28672 to 3584 per GPU. The parameter should therefore have shape [3584, 512]. Yet the weight loader's assertion — assert param.size() == loaded_weight.size() — had not fired. This was deeply puzzling.

The Reasoning Chain in Message 1949

Message 1949 is the culmination of a tight reasoning loop that spans the preceding messages (1932–1948). Let me trace that loop:

1. Message 1932–1935: The assistant verified that vLLM's bundled FlashAttention works on SM120, eliminating one potential cause of garbage output.
2. Message 1936: A direct test with a single-token prompt confirmed the garbage output was reproducible.
3. Message 1937: The assistant stepped back to enumerate what's unique about the setup — GGUF quantization, kv_b_proj reassembly, disabled DSA indexer, Triton MLA backend, float16 dtype, enforce-eager mode. The key insight: "The one thing I haven't verified is whether the kv_b_proj weight is correctly loaded after TP sharding."
4. Message 1938–1939: Reading the ColumnParallelLinear.weight_loader revealed the assertion assert param.size() == loaded_weight.size(). Yet no assertion errors appeared in the logs.
5. Message 1940–1941: The assistant checked the model code and confirmed kv_b_proj is indeed a ColumnParallelLinear with output dim num_heads * (qk_nope_head_dim + v_head_dim) = 64 * 448 = 28672, which should be sharded to 3584 with TP=8.
6. Message 1942–1944: The assistant traced the GGUF quantization code path. When kv_b_proj is in unquant_names, the GGUFConfig.get_quant_method returns UnquantizedLinearMethod(), which creates a standard nn.Parameter — not an UninitializedParameter. So the parameter should have the correct TP-sharded shape [3584, 512], and the assertion should fail when loading [28672, 512].
7. Message 1945–1948: No assertion errors were found in the logs. The assistant began to suspect that the load_weights method in deepseek_v2.py might handle kv_b_proj through a special code path that bypasses the assertion. This is where message 1949 enters. The assistant has just realized that the stacked_params_mapping loop (line 1399) handles fused parameters like gate_up_proj, and kv_b_proj falls through to the else block (line 1433+). The assistant is now reading that else block to understand exactly what happens when kv_b_proj weights are loaded.

Input Knowledge Required

To understand this message, one needs:

• DeepSeek-V2 MLA architecture knowledge: Understanding that kv_b_proj is the key-value up-projection in Multi-head Latent Attention, with dimensions determined by kv_lora_rank, num_heads, qk_nope_head_dim, and v_head_dim.
• vLLM's tensor parallelism model: How ColumnParallelLinear shards the output dimension across GPUs, and how parameters are initialized on the meta device with their final sharded shapes.
• GGUF quantization internals: The distinction between GGUFLinearMethod (which creates UninitializedParameter) and UnquantizedLinearMethod (which creates regular nn.Parameter), and how the unquant_names list controls which layers use which method.
• vLLM's weight loading pipeline: The load_weights method in model files like deepseek_v2.py, the stacked_params_mapping mechanism for fused weights, and the weight_loader dispatch that ultimately calls into ColumnParallelLinear.weight_loader.
• The GLM-5 model specifics: That layers 0–2 are dense (non-MoE) and layers 3+ are MoE, and that the model uses a custom DSA indexer that had already been disabled.

Output Knowledge Created

This message produces several forms of knowledge:

1. A confirmed code path: The assistant now knows that kv_b_proj falls through to the else block in load_weights, not through stacked_params_mapping. This means the weight loading for kv_b_proj goes through the generic weight_loader(param, loaded_weight) call, which should trigger the shape assertion.
2. A narrowed hypothesis space: If the assertion doesn't fire despite the shape mismatch, then either (a) the parameter is somehow [28672, 512] despite being a ColumnParallelLinear with TP=8, or (b) the weight_loader for this parameter is not the standard ColumnParallelLinear.weight_loader but something else entirely.
3. A target for the next investigation: The assistant now needs to read the else block code to see if there's any special handling that could explain the missing assertion — perhaps a try/except that swallows the error, or a different weight_loader dispatch.

Assumptions and Potential Mistakes

The assistant is operating under several assumptions:

• That the assertion should fire: The assistant assumes param.size() == [3584, 512] and loaded_weight.size() == [28672, 512], so the assertion should fail. But this assumes the parameter was correctly initialized with TP sharding. If the model initialization on meta device somehow doesn't apply TP sharding to kv_b_proj (perhaps because it's treated as a special case), the parameter could be [28672, 512] and the assertion would pass.
• That kv_b_proj is truly unquantized: The assistant verified that kv_b_proj is in unquant_names, but what if the prefix matching in is_layer_skipped_gguf doesn't match correctly? The unquant name might be stored differently than the actual parameter prefix.
• That the else block is the right place to look: The assistant assumes kv_b_proj doesn't match any stacked_params_mapping entry. This is correct based on the code inspection, but there could be other special-case handling earlier in the loop that the assistant hasn't considered. One subtle mistake in the reasoning: the assistant assumes that because kv_b_proj uses UnquantizedLinearMethod, the parameter is a regular nn.Parameter with the TP-sharded shape. But in vLLM's model initialization on meta device, even regular nn.Parameter objects are created with their shapes on the meta device. The actual TP sharding happens during __init__ of ColumnParallelLinear, which uses tensor_model_parallel_rank and tensor_model_parallel_world_size to compute the output size. If these are correctly set during initialization, the parameter should indeed be [3584, 512].

The Thinking Process Visible in the Message

The message reveals a classic debugging pattern: tracing the control flow. The assistant has identified a contradiction (no assertion error despite apparent shape mismatch) and is now reading the source code to understand why. The thinking is:

1. "I see — the stacked_params_mapping loop handles gate_up_proj" — the assistant has just read the code and understood the structure of the weight loading loop.
2. "For kv_b_proj, it should fall through to the else block" — this is a deduction based on the loop structure. kv_b_proj is not a fused parameter like gate_up_proj, so it won't match any stacked_params_mapping entry.
3. "Let me check what happens in that else block for kv_b_proj" — the assistant is now reading the specific code path that will handle kv_b_proj loading. The bash command reads lines 1500–1550 of deepseek_v2.py. The output shows the else block code, which includes a notable comment: "We should ask the weight loader to return success or not here since otherwise we may skip experts with other available replicas." This comment hints at a design where weight_loader returns a boolean success value — which could mean that a failed assertion might be caught and handled gracefully rather than raising an exception.

Why This Message Matters

In the broader narrative of this debugging session, message 1949 is the moment when the assistant transitions from "something is wrong with the weights" to "I need to understand the exact code path that loads these weights." It's a shift from hypothesis-driven debugging (testing theories about what might be wrong) to code-tracing debugging (following the exact execution path to find the discrepancy).

The message is also notable for what it doesn't contain — there's no conclusion, no fix, no triumphant discovery. It's purely investigative. The assistant is gathering information, reading source code, and building a mental model of the weight loading pipeline. The actual fix will come later, but this message represents the essential groundwork of understanding the system's behavior before making changes.

For anyone reading this session transcript, message 1949 illustrates a crucial debugging skill: when faced with a contradiction between expected behavior (assertion should fail) and observed behavior (no failure), trace the actual code path rather than assuming your mental model is correct. The code is always right — it's our understanding that's wrong.