The Split GGUF Investigation: When Merging Is Unavoidable

In a sprawling machine learning deployment session spanning days of infrastructure setup, kernel compilation, and model deployment, message [msg 1644] represents a quiet but pivotal moment of technical investigation. The assistant, having already built a custom llama-gguf-split tool to merge 10 split GGUF files into a single 402GB model file, pauses to ask a critical question: Is merging even necessary? Could vLLM, the inference engine being prepared for deployment, handle split GGUF files natively, eliminating an expensive and time-consuming data preparation step?

The Context: A Complex Deployment Pipeline

By this point in the session, the assistant had already navigated a labyrinth of technical challenges. The team had pivoted from the NVFP4 quantization path to GGUF quantization using unsloth's UD-Q4_K_XL format ([chunk 0.0]). This required downloading a 431GB model split across 10 sharded GGUF files, patching vLLM's gguf_loader.py to support the glm_moe_dsa architecture (a custom architecture for the GLM-5 model), and building llama-gguf-split from the llama.cpp source to merge the shards.

The download itself had been fraught with difficulty. An initial attempt using snapshot_download from huggingface_hub failed on part 4 of 10 with a RuntimeError: Data processing error ([msg 1634]). The assistant had to restart the download using a more reliable Python script and separately re-download the missing shard. At the time of this message, the download was still in progress, with part 4 being fetched while the assistant investigated whether the merge step could be skipped entirely.

The Message: A Moment of Investigation

The subject message captures the assistant in a moment of careful reasoning. It begins by analyzing the download_gguf function from vLLM's weight_utils.py, which it had examined in the preceding message ([msg 1643]):

So download_gguf finds the first file in the sorted list. For split files, it returns the first shard (00001-of-00010). But the GGUF reader would then only read that one shard.

This is the assistant working through the implications of what it just read. The download_gguf function uses allow_patterns to match files like *-{quant_type}-*.gguf (for sharded files) or *-{quant_type}.gguf (for single files). When multiple files match, it sorts them and returns the first one. For a sharded model with files named GLM-5-UD-Q4_K_XL-00001-of-00010.gguf through GLM-5-UD-Q4_K_XL-00010-of-00010.gguf, this function would return only the first shard — a mere 9MB file containing just the metadata header, not the full model weights.

The assistant then has a productive doubt: "Wait — actually, gguf-py's GGUFReader might support reading split files by detecting the naming pattern and reading all shards." This is a reasonable hypothesis. Many modern model serialization formats support sharded reading, where the loader automatically detects the shard naming pattern and transparently reads across all files. If GGUFReader had this capability, the entire merge step could be bypassed, saving significant time and disk space.

To test this hypothesis, the assistant executes a Python command that inspects the GGUFReader.__init__ method:

import gguf
import inspect
sig = inspect.signature(gguf.GGUFReader.__init__)
print('GGUFReader init params:', sig)
src = inspect.getsource(gguf.GGUFReader.__init__)
print(src[:2000])

The result is unambiguous:

GGUFReader init params: (self, path: 'os.PathLike[str] | str', mode: "Literal['r', 'r+', 'c']" = 'r')
    def __init__(self, path: os.PathLike[str] | str, mode: Literal['r', 'r+', 'c'] = 'r'):
        self.data = np.memmap(path, mode = mode)
        offs = 0
        # Check for GGUF magic
        if self._get(offs, np.uint32, override_order = '<')[0] != GGUF_MAGIC:
            raise ValueError('GGUF magic invalid')

The constructor takes a single path parameter and immediately opens it with np.memmap. There is no logic for detecting shard patterns, no multi-file aggregation, no transparent shard concatenation. GGUFReader is designed to read exactly one GGUF file. The hypothesis is disproven.## The Reasoning Process

What makes this message particularly interesting is the structure of the assistant's reasoning. It proceeds through three distinct phases:

  1. Analysis of existing code: The assistant has just read the download_gguf function ([msg 1643]) and understands its behavior — it returns a single file path from a sorted list of matching files. For sharded models, this means only the first shard is returned.
  2. Recognition of a potential problem: If download_gguf returns only the first shard, and the GGUF loader then opens that single file, the model loading would fail catastrophically — it would see only the metadata header (9MB) and miss 402GB of weight data. This would either crash with a tensor-not-found error or silently load a broken model.
  3. Formulation of a hopeful alternative: The assistant wonders if GGUFReader itself might handle sharded files transparently. This is a reasonable architectural assumption — many ML frameworks (like Hugging Face's safetensors library) support sharded loading natively. If GGUFReader had this capability, the merge step would be unnecessary.
  4. Empirical verification: Rather than continuing to speculate, the assistant immediately executes a Python inspection to verify the hypothesis. This is characteristic of the session's approach throughout — every assumption is tested, every hypothesis verified with code. The inspection reveals that GGUFReader.__init__ is remarkably simple: it takes a path and opens it as a memory-mapped numpy array. There is no shard detection, no multi-file logic, no transparent aggregation. The np.memmap call maps the entire file into virtual memory, and the reader then parses the GGUF header and tensor metadata sequentially from that single file.

Assumptions and Their Consequences

The assistant makes several assumptions in this message, most of which are reasonable:

Assumption 1: download_gguf returns the first file from a sorted list. This is correct based on the code examined in [msg 1643]. The function sorts matching files and returns the first one. For sharded models, this means only the first shard is returned.

Assumption 2: The GGUF loader would only read that one shard. This follows logically from Assumption 1, but the assistant wisely questions whether GGUFReader might have its own shard detection logic that operates independently of download_gguf. This turns out to be incorrect — GGUFReader has no such logic.

Assumption 3: The first shard contains only metadata. This is correct for GGUF sharded models — the first shard (00001-of-00010) typically contains the GGUF header, tensor metadata (names, shapes, types), and possibly some initial tensors, but the vast majority of weight data is in subsequent shards. The 9MB size of the first shard confirms this.

The key mistake the assistant avoids is not assuming the merge is unnecessary without verification. Instead, it takes the time to check, even though the download is still in progress and there's pressure to move quickly. This discipline pays off — confirming that merging is required prevents a likely failure when vLLM attempts to load the model later.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. GGUF format structure: GGUF (GGML Universal Format) is a binary format for storing quantized neural network weights. It consists of a header (magic number, version, metadata), followed by tensor information (names, shapes, types, offsets), and finally the raw tensor data. Sharded GGUF files split the tensor data across multiple files, with each file containing a subset of tensors.
  2. vLLM's weight loading pipeline: vLLM uses download_gguf to locate and download the model file, then passes the path to GGUFLoader (or a custom loader like the patched gguf_loader.py) which uses GGUFReader to read the weights. The loader iterates over tensors in the GGUF file and maps them to model parameters.
  3. Sharded model patterns: Large models are often split into shards to work around file size limits and to enable parallel downloading. The naming convention *-XXXXX-of-YYYYY.gguf is standard for GGUF shards, similar to Hugging Face's safetensors sharding convention.
  4. NumPy memory mapping: np.memmap creates a memory-mapped array from a file on disk, allowing efficient access to large arrays without loading the entire file into RAM. GGUFReader uses this as its primary data access mechanism.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Confirmed limitation: GGUFReader does not support sharded file reading. Each shard must be merged into a single file before vLLM can load it. This is a definitive answer that guides the remainder of the session.
  2. Architectural insight: The GGUF reader's simplicity — a single np.memmap call — means it has no awareness of multi-file models. Any shard merging must happen at a higher level (in the download function or as a separate preprocessing step).
  3. Validation of the merge approach: The assistant's earlier decision to build llama-gguf-split and plan for merging was correct. This investigation confirms that there is no shortcut around the merge step.
  4. Documentation of the codebase: By inspecting and quoting the GGUFReader source, the assistant creates a record of the code's behavior that can inform future decisions. The source code shows the constructor's exact signature and initialization logic.

The Broader Significance

This message, while brief and seemingly minor, exemplifies a pattern that recurs throughout the session: the assistant consistently pauses to verify assumptions rather than proceeding on faith. In a deployment pipeline with dozens of interdependent components — custom model architectures, patched inference engines, non-standard quantization formats, multi-file downloads — an incorrect assumption at any point can cascade into hours of debugging. By checking whether GGUFReader supports sharded files, the assistant prevents what would almost certainly be a confusing failure mode: vLLM would start loading, read only the metadata shard, and either crash with tensor-not-found errors or silently load a broken model that produces garbage outputs.

The message also demonstrates the value of reading source code. Rather than relying on documentation or hearsay, the assistant directly inspects the GGUFReader.__init__ method using Python's inspect module. This produces definitive, verifiable evidence about the code's behavior — no ambiguity, no outdated documentation, no guesswork.

In the end, the merge step remains necessary. The assistant will go on to use llama-gguf-split --merge to combine the 10 shards into a single 402GB file, paving the way for the first test of the patched vLLM loader with the GLM-5 GGUF model. But this investigation ensures that when that test happens, the failure — if any — will be due to the architecture patch or the weight reassembly logic, not a fundamental misunderstanding of how GGUF files are loaded.