The Moment of Diagnosis: Tracing GGUF Tensor Parallelism in a 744B-Parameter Model

In the long and arduous journey to deploy the GLM-5 model — a 744-billion-parameter Mixture-of-Experts architecture — on eight NVIDIA Blackwell GPUs using GGUF quantization and vLLM, there came a pivotal moment when the model loaded successfully, served requests without errors, and yet produced nothing but incoherent garbage. Message <msg id=1955> captures the exact instant when the assistant pivoted from high-level symptom observation to deep, systematic code investigation. This message, containing a single task tool call to a subagent, represents the disciplined application of the scientific method to a confounding inference stack failure.

The Message in Its Full Context

The message opens with a simple declarative statement:

Let me start by examining the key vLLM files to understand how GGUF + TP works, and also check how kv_b_proj weight loading actually happens.

This is followed by a task tool invocation that spawns a subagent to read and analyze several critical vLLM source files. The task description requests an investigation into how vLLM handles GGUF weights with tensor parallelism (TP), specifying the container address and the exact file paths to examine. The subagent returns a detailed analysis, beginning with the finding that UnquantizedLinearMethod is defined in linear.py rather than a separate unquantized.py file, and proceeding through the complete code path of GGUFLinearMethod.create_weights().

What makes this message remarkable is what it does not contain: any code changes, any bash commands, any direct interaction with the running server. It is purely an act of reading and understanding. In a session characterized by aggressive patching, kernel debugging, and performance optimization, this message stands out as a moment of deliberate, unhurried investigation.

Why This Message Was Written

The message was motivated by a deeply frustrating situation. After weeks of work — downloading 431 GB of GGUF split files, merging them into a single 402 GB file, writing extensive patches to vLLM's gguf_loader.py, weight_utils.py, and deepseek_v2.py, force-dequantizing tensors, and finally watching the model load across eight GPUs — the output was gibberish. The logprobs told a clear story: known continuation tokens like "2" after the sequence "1 2 3 4 5..." had logprobs around -20 to -24 instead of near zero, and the model was repeatedly emitting tokens like "IRS", "BW", "Promo", and "Version" with near-uniform probability distributions.

The assistant had already ruled out several potential causes. The GGUF dequantization kernel was verified correct on SM120 (maximum difference of 0.00012 between CPU and GPU dequant). The name mapping was verified: 1782 out of 1809 tensors correctly mapped. The kv_b_proj reassembly shape was confirmed correct at [28672, 512]. FlashAttention for prefill worked on SM120. No loading errors or warnings appeared in the logs.

The remaining hypothesis, articulated in the preceding message <msg id=1952>, was a tensor parallelism (TP) sharding mismatch. The kv_b_proj layer was created as a ColumnParallelLinear with TP=8, meaning each GPU should hold a shard of shape [3584, 512]. But the force-dequantized weight was [28672, 512] — the full, unsharded tensor. The critical question was whether vLLM's ColumnParallelLinear.weight_loader properly handled loading a full-size weight into a TP-sharded parameter, or whether each GPU was silently receiving the wrong slice of the weight.

This message was written to answer that question definitively, by reading the actual source code rather than guessing.

How the Investigation Was Structured

The assistant chose to use a task tool — spawning a subagent that runs as an independent multi-round conversation — rather than issuing individual read commands. This is a deliberate architectural choice in the opencode framework. By bundling the investigation into a single task, the assistant could request that multiple files be read in a specific order, with the subagent synthesizing the findings into a coherent analysis.

The task prompt specified reading files in a particular sequence: first gguf.py (the quantization layer that handles GGUF-specific weight creation), then linear.py (where ColumnParallelLinear and its weight_loader are defined), and then the relevant sections of deepseek_v2.py (where the model's load_weights method processes kv_b_proj). This ordering reflects a logical progression: understand how weights are created, then how they are loaded, then how the model-specific code invokes those mechanisms.

The subagent's analysis returned several key findings. It confirmed that UnquantizedLinearMethod is defined in linear.py (not a separate file, as the assistant had initially wondered). It traced the GGUFLinearMethod.create_weights() method, showing how it creates GGUFUninitializedParameter for quantized weights — parameters that materialize to the loaded weight's shape without TP sharding. This was a crucial discovery: for quantized GGUF weights, each rank does load the full-size tensor, and the dequantization kernel handles the sharding implicitly during computation. The TP sharding is not in the parameter storage but in the computation.

Assumptions and Their Validity

The message operates on several key assumptions, some of which proved correct and others incorrect.

The primary assumption was that the garbage output was caused by a TP sharding mismatch in kv_b_proj. This was a reasonable hypothesis given the evidence: the weight was full-size [28672, 512] while the parameter was expected to be [3584, 512] per rank, and no assertion errors appeared in the logs (suggesting the weight was being loaded without proper sharding). However, this assumption turned out to be incorrect. The actual root cause, discovered in subsequent messages, was a combination of two bugs: an output buffer disconnect in the Triton MLA attention backend caused by a custom PyTorch op creating a phantom tensor, and a shard ordering bug in the GGUF dequantization layer for fused projections. The TP sharding was functioning correctly all along.

A secondary assumption was that the investigation could be completed by reading source files alone, without running any experiments or inspecting runtime state. This assumption was valid for the initial phase — understanding the code paths was a necessary prerequisite to formulating more precise hypotheses. However, the investigation ultimately required runtime debugging with a profiler and careful examination of tensor values during model execution.

The assistant also assumed that UnquantizedLinearMethod might exist in a separate file (unquantized.py), reflecting an incomplete mental model of vLLM's codebase organization. The subagent's finding that it was in linear.py corrected this assumption.

Input Knowledge Required

To understand this message, one needs substantial domain knowledge spanning multiple systems. First, an understanding of the GGUF format and its quantization schemes (Q4_K, Q8_0, etc.) is necessary to grasp why force-dequantization was needed and how quantized weights differ from unquantized ones. Second, familiarity with tensor parallelism in model inference — how ColumnParallelLinear splits the output dimension across GPUs — is essential to understand the sharding concern. Third, knowledge of vLLM's internal architecture, particularly its weight loading pipeline (gguf_loader.py → weight_utils.py → model-specific load_weights), is required to follow the investigation's logic. Finally, an understanding of the MLA (Multi-head Latent Attention) architecture used by DeepSeek-V2 and GLM-5, and how kv_b_proj fits into it, provides the necessary context for why this particular weight was suspect.

The conversation context also provides critical background: the previous discovery that llama.cpp's converter splits kv_b_proj into separate attn_k_b and attn_v_b tensors, the sentinel-suffix fix in the name map, and the force-dequant workaround. Without this history, the investigation's focus on kv_b_proj would seem arbitrary.

Output Knowledge Created

This message produced several important pieces of knowledge. First, it confirmed that UnquantizedLinearMethod is defined in linear.py, resolving a minor but nagging uncertainty. Second, it revealed the complete code path for how GGUF weights interact with tensor parallelism in vLLM: quantized weights use GGUFUninitializedParameter which materializes to full size on each rank, with the dequantization kernel handling sharding implicitly during computation. This means that for quantized weights, the full tensor is present on every GPU, and TP sharding is a computational rather than storage concern.

Third, the analysis confirmed that for unquantized (force-dequantized) weights like kv_b_proj, the ColumnParallelLinear.weight_loader does handle TP sharding — it splits the weight along the output dimension according to the rank. This meant the TP sharding hypothesis was unlikely to be the root cause, forcing the investigation to look elsewhere.

Perhaps most importantly, the message established a methodology: when faced with a confounding bug in a complex system, the correct first step is not to make random changes but to trace the code paths systematically, reading the source in logical order, and building a complete mental model before intervening.

The Thinking Process Revealed

The message reveals a disciplined, hypothesis-driven reasoning process. The assistant had formulated a clear hypothesis (TP sharding mismatch for kv_b_proj), identified the specific question that needed answering (does ColumnParallelLinear.weight_loader properly handle full-size weights?), and designed an investigation to answer it (read the relevant source files in order).

The choice to use a task tool rather than individual read commands is itself revealing. It shows the assistant optimizing for coherence over granularity — preferring a synthesized analysis over a sequence of disconnected file excerpts. This is a meta-cognitive decision about how to best manage the investigation's complexity.

The message also reveals intellectual humility. The assistant could have simply assumed the TP sharding was broken and written a patch to slice the weight per rank. Instead, it chose to first verify the assumption by reading the code. This willingness to be proven wrong — to let the code speak for itself rather than imposing a preferred narrative — is a hallmark of effective debugging.

In the end, the investigation did not immediately solve the problem. The actual bugs were elsewhere, in the Triton MLA attention backend and the GGUF dequantization shard ordering. But this message was nonetheless essential: by ruling out the most plausible hypothesis, it cleared the path for the true root cause to be discovered. In the archaeology of a complex debugging session, message <msg id=1955> is the layer where one promising theory was laid to rest, and the search for the real culprit could begin in earnest.