From Research to Optimization: The Methodical Campaign to Deploy GLM-5-NVFP4 at Scale

Introduction

Deploying a 744-billion-parameter Mixture-of-Experts model like GLM-5-NVFP4 on cutting-edge Blackwell GPUs is not a task that yields to guesswork. It demands a systematic, evidence-based approach where every decision is informed by research, every optimization is benchmarked against a clean baseline, and every failure is documented as a lesson for the future. This article synthesizes a remarkable chunk of an opencode coding session that exemplifies exactly this disciplined methodology. What unfolds is a narrative that moves through four distinct phases: comprehensive intelligence gathering via a dedicated research subagent, the application of those findings to real optimization work, the rigorous benchmarking of each intervention, and finally the documentation of all knowledge into a permanent artifact. The story is one of methodical progress — of ideas tested, measured, accepted or rejected based on data rather than speculation, and ultimately woven into a coherent understanding of how to serve one of the most demanding open-weight models in existence.

Phase One: The Intelligence-Gathering Operation

Before any optimization could begin, the assistant needed to understand the deployment landscape for GLM-5-NVFP4. This was not a casual literature review — it was a structured intelligence-gathering operation executed through a dedicated subagent session [2]. The user's research mandate was precise: investigate deployment guides, performance benchmarks, optimal serving configurations, known issues, DeepSeek Sparse Attention (DSA) optimization tips, expert routing patterns, alternative quantization approaches, and comparisons with similar MoE architectures like Kimi K2 and DeepSeek V3.

The subagent's opening move was a masterclass in parallel research methodology [3]. In a single round, it dispatched four simultaneous information-gathering tool calls: a webfetch of the HuggingFace model card for lukealonso/GLM-5-NVFP4, and three Exa web searches targeting deployment guides, benchmarks, and serving framework support. Each query was crafted with domain-specific precision — terms like "NVFP4," "744B MoE," "serving configuration," and "tensor-parallel-size" ensured that results would be technically relevant rather than generic.

The second round deepened the investigation with eight parallel calls targeting specific knowledge gaps [5]. The assistant recognized that the initial broad sweep had left several of the user's questions unanswered. It formulated precise queries for expert routing patterns, DSA optimization, Blackwell-specific deployment experiences, model comparisons, and community discussions on Reddit. Simultaneously, it fetched three authoritative primary sources: the vLLM Recipes documentation for GLM-5, the Lambda AI deployment page, and the official Z.ai blog post. This gap-analysis-driven approach — identify what is missing, formulate a query to fill that gap, execute in parallel — is a textbook example of efficient systematic research.

A pivotal strategic pivot occurred in the third round [1]. The assistant shifted from top-down official sources to bottom-up community knowledge, fetching the Unsloth practical deployment guide, the HuggingFace discussions page, and searching for GGUF/llama.cpp community experiences. This recognition that practical deployment wisdom often lives in community-created guides and grassroots forums rather than polished documentation reflects a sophisticated understanding of how knowledge about LLM deployment is structured across the internet.

The fourth round brought the research to its most targeted point [4]. The assistant fetched a specific HuggingFace discussion thread asking "Possible to run on six RTX Pro 6000 Blackwell with vLLM oder SGLang?" — a question that directly mirrored the user's own hardware context. It also searched for Expert Load Balance (EPLB) techniques, proactively investigating performance optimization before encountering bottlenecks. This transition from "what is known about GLM-5" to "what is known about GLM-5 on our specific hardware" marks the critical juncture where research becomes actionable.

Phase Two: The Research Synthesis

The culmination of the intelligence-gathering phase was a comprehensive research summary spanning nine sections [6]. This document, produced by the subagent as its final deliverable, synthesized findings from the HuggingFace model card, the arXiv technical report, the vLLM Recipes documentation, the Lambda deployment guide, the Unsloth quantization guide, community discussions on Reddit and HuggingFace, and multiple blog posts.

The synthesis revealed several critical findings that would directly shape the subsequent optimization work. First, SGLang was confirmed as the only viable serving engine for NVFP4 quantization on Blackwell GPUs — vLLM lacked NVFP4 support entirely, and the RTX 5090 incompatibility issues in vLLM forums suggested broader Blackwell compute capability problems. Second, the model card's launch command provided a proven baseline configuration with specific NCCL environment variables to prevent P2P hangs. Third, the NVFP4 quantization strategy — using three separate calibration passes (coding, broad, deep/long-context) merged via element-wise max — provided indirect evidence that some experts are indeed "hotter" than others depending on input domain, validating the user's interest in expert routing patterns.

The synthesis also identified critical unknowns: no confirmed NVFP4 community benchmarks existed beyond the model card's stated configuration on 8x RTX Pro 6000, the question of whether 6x GPUs (576GB) would suffice remained unanswered, and no public analysis of GLM-5-specific routing patterns had been published. These gaps defined the frontier where the user's own work would become a novel contribution to the community.

Phase Three: The Optimization Campaign

With the research foundation in place, the assistant pivoted to hands-on optimization work. The first and most impactful intervention was updating SGLang to its latest commit. This single change yielded a dramatic 2x throughput improvement at 256 concurrency compared to earlier baselines — a finding that underscores how rapidly serving framework performance evolves in this space.

The assistant then implemented Opportunistic Expert Activation (OEA), a decode-time routing optimization gated by the SGLANG_OEA_K0 environment variable. The implementation was carefully engineered to handle the unsorted top-k output from the router and to use proper sigmoid scores for weight gathering. However, clean A/B benchmarks revealed a sobering result: near-zero average throughput improvement on random data. Peak throughput improved by approximately 5% at high concurrency, but the average gain was negligible. This confirmed a crucial insight: OEA's benefit depends on non-uniform expert routing patterns. On random data where expert activation is naturally balanced, there is little opportunity for opportunistic skipping. The assistant did not speculate or extrapolate — it measured, documented, and concluded based on evidence.

The Expert Parallelism (EP8) attempt was another instructive episode. The assistant retried EP8 with a memory-safe configuration (--mem-fraction-static 0.75 --max-running-requests 512), and the model loaded successfully, using 59.5GB with approximately 34GB free per GPU. However, during warmup, the server crashed due to CUTLASS tile failures — the 128×256×128 tile configuration exceeded the SM120's 100KB shared memory limit. The server ultimately recovered and ran, but the incident revealed a hardware-specific constraint that would need to be addressed for reliable EP8 deployment.

Single and dual-stream throughput benchmarks provided another data point in the growing performance picture. The assistant measured 10.36 tok/s for single-stream and 19.29 tok/s for dual-stream — demonstrating excellent linear scaling that validated the tensor parallelism configuration and memory management strategy.

Phase Four: Documentation and Theoretical Analysis

The final phase of the chunk saw the assistant engaging in two parallel activities: comprehensive documentation and theoretical performance analysis.

The glm5findings.md document was a 500+ line comprehensive record covering all discoveries, benchmarks, configurations attempted, and lessons learned. This document served as both a reference for future work and a permanent artifact of the optimization campaign. Its creation reflects a commitment to knowledge preservation that is essential in complex infrastructure projects — the insights gained through trial and error are too valuable to be lost when the next engineer picks up the work.

Simultaneously, the assistant began computing the theoretical maximum single-stream performance for this exact model and hardware combination. By gathering model configuration parameters — hidden dimension size, number of layers, expert count, attention head dimensions — the assistant aimed to establish the HBM bandwidth-limited and PCIe allreduce-limited throughput ceilings. This theoretical analysis provides an upper bound against which actual measured performance can be compared, answering the fundamental question: "How close are we to the hardware limit?"

The Methodical Ethos

What unifies these four phases is a consistent methodological approach. Every idea is implemented. Every implementation is benchmarked cleanly against a baseline. Every benchmark result is documented. Every conclusion — whether positive or negative — is accepted based on evidence rather than wishful thinking.

The OEA experiment exemplifies this ethos. The assistant invested significant engineering effort in implementing the optimization, handling edge cases in the top-k routing output, and integrating it with the serving framework. When the benchmarks showed near-zero average improvement, the assistant did not tweak parameters or search for a configuration where OEA would appear more favorable. It accepted the result, documented the finding, and moved on. This intellectual honesty is rare and valuable.

Similarly, the EP8 attempt was not abandoned when the CUTLASS tile failure occurred. The assistant diagnosed the issue (SM120's 100KB shared memory limit), noted that the server ultimately recovered, and preserved the configuration for future investigation. The failure was not a dead end — it was a data point.

Conclusion

This chunk of the opencode session tells the story of a methodical, evidence-based optimization campaign for one of the most demanding AI models available. From the initial intelligence-gathering operation that mapped the deployment landscape, through the research synthesis that identified viable paths, to the hands-on optimization work that tested each idea against reality, and finally to the comprehensive documentation that preserved every finding — the arc is one of disciplined engineering practice.

The 2x throughput improvement from updating SGLang, the sobering OEA benchmarks, the instructive EP8 failure, the excellent linear scaling at low concurrency, and the beginnings of theoretical performance analysis all contribute to a growing understanding of how to serve GLM-5-NVFP4 on Blackwell hardware. Each data point, whether positive or negative, moves the project forward. This is not the work of guesswork or intuition — it is the work of measurement, documentation, and evidence-based decision-making. And it is precisely the kind of work that separates successful ML deployments from failed experiments.