The Dead End That Became a New Beginning: How a Failed GGUF Deployment Sparked a Deep Dive into vLLM's Architecture

Introduction

In any complex engineering project, the most valuable moments are often the ones where a carefully laid plan collapses under the weight of evidence. Message 1517 in this opencode coding session is precisely such a moment — a crisp, well-reasoned "stop and think" message that lays bare a critical incompatibility between the user's deployment strategy and the reality of the software ecosystem. The assistant had been tasked with deploying the massive GLM-5 model (744B parameters) using a GGUF quantized format (UD-Q4_K_XL) on vLLM, after a long and painful journey through NVFP4 quantization, kernel profiling, and performance optimization. But as this message reveals, the vLLM + GGUF path was not merely difficult — it was structurally impossible without significant engineering intervention.

The Context: A Long Journey Through the ML Inference Landscape

To understand why message 1517 carries such weight, we must appreciate the journey that preceded it. The session had been running for hundreds of messages across twelve segments, covering everything from NVIDIA driver installation on Ubuntu 24.04 to CUDA toolkit configuration, flash-attn compilation troubleshooting, and extensive performance benchmarking of the GLM-5-NVFP4 model on SGLang. The user had eight RTX PRO 6000 Blackwell GPUs (SM120 architecture) with 768 GB of total VRAM, and the team had wrestled with FP4 GEMM kernel efficiency, KV cache cast bottlenecks, and expert parallelism crashes.

The pivotal moment came in segment 11, when the user decided to abandon the NVFP4 path entirely and pivot to unsloth's UD-Q4_K_XL GGUF quantization deployed on vLLM. This was a strategic bet: GGUF quantization promised a much smaller memory footprint (431 GB vs. the ~860 GB needed for FP8), and vLLM offered the continuous batching and paged attention that llama.cpp lacked. The assistant dutifully freed disk space, deleted the old NVFP4 model, and began preparing for the GGUF download. But as the investigation unfolded across messages 1499 through 1516, a troubling picture emerged.

The Investigation: How the Assistant Discovered the Block

The assistant's investigation was methodical and multi-layered, spanning approximately 18 messages of research before presenting the findings in message 1517. Let me trace the reasoning chain.

Step 1: Checking the Environment (messages 1499-1502). The assistant began by verifying disk space (1.3 TB free on /shared), checking installed packages, and confirming that vLLM was not yet installed while gguf (v0.17.1) and transformers (v5.2.0) were present. This established the baseline.

Step 2: Reading the Official Documentation (messages 1503-1504). The assistant fetched the vLLM GLM-5 recipe and the vLLM GGUF documentation. The recipe specified a specific commit for FP8 deployment, but the GGUF docs revealed that vLLM's GGUF support was "highly experimental" and worked by delegating to transformers for metadata parsing.

Step 3: Discovering SM120 Issues (messages 1505-1506). A web search revealed a known bug (vllm#33416) about NVFP4 MoE kernels failing on SM120 Blackwell GPUs. This was a warning sign, but the assistant correctly noted that GGUF uses different quantization kernels and might avoid the issue.

Step 4: Checking Unsloth's Documentation (messages 1507-1509). The assistant visited the HuggingFace page for unsloth/GLM-5-GGUF and the Unsloth guide for running GLM-5 locally. A critical observation emerged: the Unsloth guide only showed llama.cpp and vLLM FP8 deployment — there was no documented vLLM + GGUF path for GLM-5. The architecture was listed as glm-dsa, and the question of whether vLLM supported this architecture became paramount.

Step 5: Searching for Evidence of DeepSeek GGUF Support (messages 1510-1512). Multiple web searches confirmed that deepseek2 architecture GGUF was consistently failing on vLLM with ValueError: GGUF model with architecture deepseek2 is not supported yet. Multiple GitHub issues (vllm#12573, #13085, #13665, #15277, unsloth#2254) documented this failure. Even more damning, MLA (Multi-head Latent Attention) was explicitly not supported with GGUF.

Step 6: Verifying Directly on the System (messages 1513-1516). The assistant didn't stop at web research — it went directly to the installed transformers library on the container and inspected GGUF_CONFIG_MAPPING. The output was definitive: the supported architectures were bloom, deci, falcon, gemma2, gemma3, gpt2, lfm2, llama, mamba, mistral, nemotron, phi3, qwen2, qwen2_moe, qwen3, qwen3_moe, stablelm, starcoder2, t5, tokenizer, umt5. No deepseek2, no glm_moe_dsa, no DeepSeek or GLM variant whatsoever. The assistant cancelled all pending GGUF-related todos.

Message 1517: The Analysis and Decision Point

The subject message itself is structured as a clear, professional "stop and present findings" communication. Here is the message in full:

I need to stop and present you with the findings before proceeding. Here's the situation:

---

## The vLLM + GGUF Path is a Dead End

After thorough investigation, I've confirmed that vLLM cannot load GLM-5 GGUF files. The blocking issue is:

1. transformers (v5.2.0) GGUF support does NOT include the DeepSeek/GLM architecture. The supported GGUF architectures are: bloom, deci, falcon, gemma2, gemma3, gpt2, lfm2, llama, mamba, mistral, nemotron, phi3, qwen2, qwen2_moe, qwen3, qwen3_moe, stablelm, starcoder2, t5. No deepseek2, no glm_moe_dsa.

2. vLLM depends on transformers to parse GGUF metadata into an HF config and map tensor names. Without this mapping, it literally cannot load the weights.

3. Multiple GitHub issues (vllm#12573, #13085, #13665, #15277, unsloth#2254) confirm this — every attempt to run DeepSeek GGUF on vLLM has failed with ValueError: GGUF model with architecture deepseek2 is not supported yet.

4. Even the Unsloth docs for GLM-5 only show llama.cpp for GGUF and vLLM for FP8 — there's no documented vLLM+GGUF path.

## Available Options

Given this, we need to pick a new direction:

| Option | Pros | Cons | |---|---|---| | A. vLLM + FP8 (zai-org/GLM-5-FP8) | Official recipe, properly supported, MTP spec decode | Needs ~860GB VRAM (we have 768GB) — may not fit. Also SM120 has known issues with some vLLM kernels | | B. Stay with sglang + NVFP4 + improved patch | Already working at 13.5 tok/s single-stream; model already deleted though | Need to re-download model (405GB). The KV cache cast bottleneck caps single-stream performance | | C. sglang + FP8 (not NVFP4) | Avoids the NVFP4 KV cache issue since FP8 MLA might work differently | Same ~860GB VRAM concern; sglang FP8 for GLM-5 may have its own issues | | D. llama.cpp (GGUF, as Unsloth recommends) | Proven to work with GLM-5 GGUF, much smaller memory footprint (431GB) | No continuous batching, no paged attention, limited concurrent handling — you explicitly rejected this | | E. Patch transformers/vLLM to add DeepSeek GGUF support | Would enable the original plan | Massive undertaking — need to add architecture mapping, tensor name mapping, MoE weight loading for GGUF |

How would you like to proceed?

The message opens with a direct statement: "I need to stop and present you with the findings before proceeding." It opens with a direct statement: "I need to stop and present you with the findings before proceeding." This is significant — the assistant recognized that proceeding further down the GGUF path would be wasted effort, and chose to escalate to the user rather than continue executing a flawed plan.

The message is organized into three sections:

Section 1: The Dead End Diagnosis. Four bullet points summarize the investigation:

Transformers v5.2.0 does not include DeepSeek/GLM in its GGUF supported architectures
vLLM depends on transformers for GGUF metadata parsing and tensor name mapping
Multiple GitHub issues confirm consistent failures for DeepSeek GGUF on vLLM
Even Unsloth's own docs only show llama.cpp for GGUF and vLLM for FP8 — no combined path Section 2: The Options Table. Five options are presented with pros and cons: - Option A (vLLM + FP8): The official recipe, but needs ~860 GB VRAM (we have 768 GB) and SM120 has known kernel issues - Option B (sglang + NVFP4): Already working at 13.5 tok/s, but the model was deleted and the KV cache cast bottleneck caps performance - Option C (sglang + FP8): Avoids NVFP4 KV cache issues, but same VRAM concern and potential sglang FP8 issues - Option D (llama.cpp): Proven to work with GLM-5 GGUF, smaller memory footprint, but no continuous batching — and the user had explicitly rejected this - Option E (Patch transformers/vLLM): Would enable the original plan, but is a "massive undertaking" Section 3: The Open Question. The message ends with "How would you like to proceed?" — a clean handoff to the user.

The Reasoning and Assumptions

This message reveals several layers of reasoning. First, there's the meta-reasoning about when to stop and ask for help. The assistant had been autonomously executing a plan (install vLLM, download GGUF, merge splits, deploy), but when it encountered evidence that the plan was fundamentally impossible, it chose to escalate rather than continue. This is a sophisticated judgment call — many agents would have pressed ahead, installed vLLM, tried to load the GGUF file, and only then reported the error. The assistant saved hours of wasted download time (431 GB!) by recognizing the blocker early.

Second, there's the reasoning about evidence thresholds. The assistant didn't stop at a single GitHub issue. It gathered evidence from multiple sources: the transformers source code (direct inspection), the HuggingFace docs, the Unsloth guide, the vLLM recipe, multiple GitHub issues, and web search results. Only when all evidence converged did it conclude the path was a dead end.

Third, there are assumptions embedded in the options analysis. Option A assumes FP8 needs ~860 GB, but this is an estimate — the actual memory usage might differ with tensor parallelism and memory optimizations. Option B assumes the KV cache cast bottleneck is fundamental, but the assistant had previously achieved a 29% improvement with a gather-then-cast patch, suggesting further optimization might be possible. Option C assumes sglang FP8 might work differently, but this is speculative. The assistant is transparent about these uncertainties.

Potential Mistakes and Incorrect Assumptions

The most significant potential mistake is in Option A's VRAM calculation. The assistant states that FP8 needs ~860 GB while the system has 768 GB, implying it won't fit. However, this calculation may not account for memory optimizations available in vLLM — memory-efficient attention, tensor parallelism sharding, KV cache quantization, and the fact that GLM-5 has only 40B active parameters despite 744B total. With TP8 (tensor parallelism across 8 GPUs), each GPU would need to hold only 1/8 of the weights, which for a 744B model in FP8 is ~744/8 = 93 GB per GPU for weights alone, plus activations and KV cache. This might actually fit within the 97 GB per GPU. The assistant's "may not fit" qualifier is appropriate, but the framing could have been more nuanced.

Another potential blind spot: the assistant didn't explore whether vLLM's GGUF loader could be made to work without transformers. The message states "vLLM depends on transformers to parse GGUF metadata into an HF config and map tensor names," but in theory, one could bypass transformers entirely and write a custom GGUF loader within vLLM. Option E ("Patch transformers/vLLM") hints at this, but the scope of work is described as a "massive undertaking" without further detail.

The assistant also assumes that the gguf-py package installed on the system (v0.17.1) is sufficient for the investigation. However, the GLM-5 GGUF files use the glm-dsa architecture, which may require a newer version of gguf-py from llama.cpp HEAD. The assistant later discovered this and installed gguf-py from source, but at the time of message 1517, this was an unresolved dependency.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

The ML model serving ecosystem: What vLLM, SGLang, and llama.cpp are, and how they differ in architecture (continuous batching vs. simple inference)
The GGUF format: That it's a single-file format for storing quantized models, originally from llama.cpp, with a metadata header containing architecture type and tensor names
The transformers library's role: That HuggingFace's transformers library serves as the canonical model definition layer, and many tools (including vLLM) depend on it for architecture-specific weight loading
DeepSeek/GLM architecture specifics: That GLM-5 is derived from DeepSeek V3, uses Multi-head Latent Attention (MLA), has 256 routed MoE experts with only 8 active per token, and uses a "glm-dsa" architecture identifier
SM120 Blackwell GPU architecture: That the RTX PRO 6000 Blackwell uses compute capability 12.0, which has known compatibility gaps with some CUDA kernels designed for SM90 (Hopper) or earlier
Memory budgeting for LLM inference: Understanding how model weights, KV cache, and activations consume VRAM, and how tensor parallelism distributes memory across GPUs

Output Knowledge Created

This message creates several important outputs:

A clear decision framework for the user, with five concrete options each evaluated for feasibility
A documented blocker in the transformers GGUF architecture mapping that can be referenced in future work
A risk assessment of the SM120 compatibility issues with vLLM
A cancellation of wasted work — the GGUF download (431 GB) was never started, saving hours of bandwidth and disk I/O
A prioritization of Option E — by presenting it as the last option and the only one that enables the original plan, the message subtly frames it as the most ambitious but potentially most rewarding path

The Thinking Process Visible in the Message

The message's structure reveals the assistant's thinking process. The progression from "here's the problem" to "here's the evidence" to "here are the options" mirrors the investigative flow. The assistant is careful to distinguish between confirmed facts (transformers doesn't support deepseek2 GGUF) and estimates (FP8 may not fit in 768 GB). The tone is professional and transparent — the assistant doesn't hide the complexity or pretend to have all the answers.

One particularly interesting aspect is the cancellation of todos in message 1516, just before this message. The assistant had a todo list with "Install vLLM nightly," "Install gguf-split tool," "Download UD-Q4_K_XL split files," and "Merge 10 split GGUF files" — all marked "cancelled." This shows that the assistant was already acting on its findings before presenting them to the user, updating its internal state to reflect the new reality.

The User's Response and What Followed

The user's response to message 1517 was immediate and decisive: "E. add this gguf support to vllm" ([msg 1518]). This choice is revealing — the user rejected the safer, more conventional paths (FP8, llama.cpp, reverting to sglang) in favor of the most ambitious option. The assistant then launched into a massive engineering effort, spawning three parallel research tasks to understand the transformers GGUF mapping system, the vLLM GGUF loader code, and the GLM-5 GGUF tensor structure. The result was a comprehensive patch for vLLM's gguf_loader.py that enabled GLM-5 GGUF loading — a significant contribution to the open-source ecosystem.

Conclusion

Message 1517 is a masterclass in engineering communication. It demonstrates the value of stopping to think when the evidence contradicts the plan, the importance of multi-source evidence gathering, and the art of presenting complex technical findings in an actionable format. The assistant's willingness to cancel planned work and escalate to the user — rather than blindly executing — saved enormous wasted effort and redirected the project toward a more ambitious and ultimately more rewarding goal. The message also reveals the deep interconnectedness of the ML tooling ecosystem, where a missing entry in a configuration dictionary in transformers can block an entire deployment pipeline, and where understanding those dependencies is essential for effective troubleshooting.

The decision to present Option E last, with the caveat that it's a "massive undertaking," set the stage for the user's ambitious choice. In the end, the dead end wasn't a dead end at all — it was the beginning of a deeper engagement with the code that would ultimately unlock the GLM-5 GGUF deployment on vLLM.