The Final Summary: Deploying GLM-5-NVFP4 on 8x RTX PRO 6000

Introduction

In the course of a complex machine learning deployment session, a single message often crystallizes hours of iterative debugging, benchmarking, and analysis into a coherent status report. Message 311 in this opencode conversation is precisely such a message: a comprehensive summary delivered by the AI assistant after an extended investigation into deploying the GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts (MoE) language model—on eight NVIDIA RTX PRO 6000 Blackwell GPUs using the SGLang inference framework.

This article examines that message in depth: why it was written, what decisions it documents, the assumptions it embeds, the knowledge it presupposes, and the knowledge it creates. The message is not merely a log entry; it is a carefully constructed synthesis of an entire investigative arc, serving as both a handoff document and a reference for future work.

The Context That Produced This Message

To understand why message 311 exists, one must understand the trajectory of the session that preceded it. The assistant had been working through a series of increasingly refined attempts to get GLM-5-NVFP4 running stably and efficiently on an unusual hardware configuration: eight RTX PRO 6000 GPUs (Blackwell architecture, SM120 compute capability) connected via PCIe within a Proxmox virtualized environment. This combination of cutting-edge GPU hardware, a bleeding-edge model quantization (NVFP4), a nightly build of SGLang, and a virtualized infrastructure created a perfect storm of compatibility issues.

The immediate predecessor to message 311 was a sequence of debugging rounds spanning [msg 287] through [msg 310], in which the assistant systematically:

  1. Investigated FlashInfer autotuning ([msg 287]–[msg 290]), discovering that the autotuner only applies to the TRT-LLM MoE path, not the CUTLASS path, and that no tuning configuration exists for SM120 GPUs anyway.
  2. Resolved a critical NaN crash by discovering that only the trtllm NSA (NVIDIA Sparse Attention) backend works on SM120 during decode; all other backends (flashmla_kv, flashmla_sparse) produce NaN outputs.
  3. Benchmarked multiple MoE runner backends (flashinfer_cutlass, flashinfer_cutedsl), finding comparable throughput around 195–225 output tokens per second at saturation.
  4. Tested server tuning parameters including memory fraction and CUDA graphs, finding negligible impact.
  5. Evaluated expert parallelism (EP) as a potential optimization, concluding it offered no benefit due to memory constraints (453 GB of MoE experts cannot be replicated across 96 GB GPUs) and the small hidden size of the model.
  6. Diagnosed virtualization overhead by confirming the Proxmox VM lacks direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory—a significant bottleneck for the small-message all-reduce operations that dominate MoE inference. Message 311 is the capstone of this investigative arc. It is the moment when the assistant stops iterating and synthesizes, providing a definitive answer to the question that had been driving the session: "What is the working configuration, and what performance can we expect?"

The Structure and Content of the Message

The message is organized into six clearly delineated sections, each serving a distinct purpose:

1. The Working Launch Command

The message opens with the complete, production-ready server launch command. This is the most actionable part of the message—anyone who needs to replicate this deployment can copy this command verbatim. The inclusion of environment variable exports (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=PHB, etc.) alongside the SGLang arguments signals that this is a battle-tested configuration, hardened against the specific failure modes encountered during debugging.

2. The Key Fix

The assistant explicitly calls out the NSA backend selection as the critical discovery: --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm. The message states bluntly that "all other NSA backends (flashmla_kv, flashmla_sparse) produce NaN on SM120 during decode." This is the single most important piece of information in the entire message—without it, the deployment would crash on every decode step.

3. Throughput Benchmarks

Two benchmark tables are presented. The first shows a concurrency sweep (1 to 64 concurrent requests) with fixed input/output lengths of 256 tokens each. The second shows a decode-heavy configuration (128 input, 512 output) at two concurrency levels. These numbers tell a clear story: single-stream throughput is approximately 11 tokens per second, while saturated throughput reaches approximately 210–247 output tokens per second.

The inclusion of end-to-end latency (Mean E2E) alongside throughput provides a complete picture of the serving characteristics. Notably, the latency remains relatively stable across concurrency levels (around 16–21 seconds), suggesting the system is well-behaved under load.

4. Profiling Data

The assistant reports GPU utilization at 100%, power draw at only 55% of TDP (330W out of 600W), memory bandwidth utilization at 52–53%, and PCIe links barely utilized at ~1 GB/s. This profiling data is crucial because it directly informs the bottleneck analysis: the GPUs are compute-bound on small-batch MoE expert kernels, not memory-bound or I/O-bound.

5. What Didn't Help

This section is a testament to the systematic methodology the assistant employed. Four categories of attempted optimizations are explicitly ruled out:

6. Bottleneck Analysis

The final section provides the root-cause explanation: the ~11 tok/s single-stream performance is limited by the sheer number of sequential operations per decode step. Each of the 78 layers requires MLA attention, MoE expert dispatch, and all-reduce communication. With NVFP4 quantization and small matrix dimensions (6144 × 2048), the GPU's tensor cores cannot be saturated on single-token decode. Throughput scales linearly with batch size up to approximately 64 concurrent requests, topping out at ~250 output tok/s.

Why This Message Was Written

The assistant's motivation for writing message 311 is multifaceted. At the surface level, it is a status update—the user had been following the investigation in real time, and this message provides a clear "where we stand" summary. But the message serves deeper purposes:

First, it creates a permanent record. The working launch command, the benchmark numbers, and the profiling data are all information that would be time-consuming to reconstruct. By consolidating them into a single message, the assistant ensures that the knowledge gained during this session is not lost.

Second, it demonstrates closure. The message's tone is definitive and final. There is no "we could try X next" or "further investigation needed." The assistant is signaling that the investigation is complete—the configuration is working, the performance is characterized, and the bottlenecks are understood. This is a handoff message, suitable for presenting to stakeholders or archiving in project documentation.

Third, it preempts redundant questions. By explicitly listing what didn't work and why, the assistant saves the user (and any future reader) from asking "What about CUDA graphs?" or "Did you try expert parallelism?" The message is designed to be a complete reference that answers the most likely follow-up questions before they are asked.

Fourth, it establishes credibility. The detailed profiling data and the nuanced bottleneck analysis demonstrate that the assistant has done rigorous work. The message is not just a list of commands and numbers; it includes interpretation and reasoning, showing that the assistant understands why the results are what they are.

Decisions Documented in This Message

While message 311 does not itself make new decisions, it documents and justifies several critical decisions made during the preceding investigation:

  1. NSA backend selection: The decision to use trtllm for both decode and prefill NSA backends is presented as the "key fix." This was a decision forced by compatibility constraints—the other backends simply do not work on SM120 hardware.
  2. MoE runner backend: The flashinfer_cutlass backend is used in the canonical launch command, though the benchmarks show that flashinfer_cutedsl performs equivalently. The choice is somewhat arbitrary but is documented as the preferred configuration.
  3. Memory fraction: The value 0.92 was chosen after testing, balancing KV cache capacity against the risk of out-of-memory errors.
  4. Rejection of expert parallelism: This is an explicit negative decision, documented with clear reasoning about memory constraints and communication volume.
  5. Rejection of CUDA graphs: Another negative decision, supported by profiling data showing that kernel launch overhead is not the bottleneck. These documented decisions are valuable because they capture not just what was chosen, but why alternatives were rejected. In a complex deployment environment, knowing that a particular optimization was tried and found ineffective is often as important as knowing which optimizations work.

Assumptions Embedded in the Message

Message 311 makes several implicit assumptions that are worth examining:

The message assumes the reader understands the domain. Terms like "NSA," "SM120," "MoE," "TP8," "NVFP4," and "all-reduce" are used without explanation. The message is written for an audience familiar with large language model inference, GPU architecture, and distributed computing concepts.

The message assumes the benchmark methodology is sound. The throughput numbers are presented as definitive, but the message does not discuss variance, warmup effects, or statistical significance. The assumption is that a single run of the SGLang benchmark tool at each concurrency level produces reliable results.

The message assumes the bottleneck analysis is complete. The conclusion that the system is "compute-bound on small-batch MoE expert kernels" is supported by profiling data, but it implicitly assumes that no other bottleneck exists that hasn't been measured. For instance, the analysis does not consider potential CPU-side bottlenecks in the SGLang scheduler or Python runtime.

The message assumes the hardware configuration is fixed. There is no discussion of whether different hardware (e.g., NVLink-connected GPUs, a non-virtualized environment, or different CPU-to-GPU topology) would change the results. The analysis is specific to this particular deployment.

The message assumes the model configuration is optimal. The use of NVFP4 quantization, tensor parallelism 8, and the specific SGLang arguments are presented as the working configuration, but the message does not explore whether different model configurations (e.g., different quantization schemes or parallelism strategies) might yield better performance.

These assumptions are not flaws—they are necessary boundaries for a focused investigation. But recognizing them helps the reader understand the scope and limitations of the message's conclusions.

Input Knowledge Required

To fully understand message 311, a reader needs knowledge spanning several domains:

Large language model architecture: Understanding what a 744B MoE model is, how expert routing works, what MLA (Multi-head Latent Attention) is, and why 78 layers with 256 experts and 8 activated per token creates a specific computational profile.

GPU architecture and CUDA: Understanding SM120 (Blackwell) compute capability, tensor cores, memory bandwidth, TDP, and the relationship between compute utilization and power draw. The distinction between compute-bound and memory-bound workloads is central to the bottleneck analysis.

Distributed inference frameworks: Understanding tensor parallelism (TP8), expert parallelism (EP), all-reduce communication, and how SGLang orchestrates multi-GPU inference. The NCCL environment variables (NCCL_IB_DISABLE, NCCL_P2P_LEVEL, etc.) are tuning parameters for the NVIDIA Collective Communications Library.

Quantization formats: Understanding NVFP4 (NVIDIA 4-bit floating point) and how it affects matrix dimensions, memory footprint, and compute efficiency. The modelopt_fp4 quantization path is specific to NVIDIA's model optimization toolkit.

Virtualization and PCIe topology: Understanding how Proxmox/KVM virtualization affects GPU peer-to-peer communication, what "NS status" means for NVLink/NVSwitch support, and why small-message all-reduce performance degrades when transfers must go through host memory.

SGLang server architecture: Understanding the various backend options (--attention-backend, --moe-runner-backend, --nsa-decode-backend, --fp8-gemm-backend), what CUDA graphs are, and how the batching and scheduling system works.

This is a substantial knowledge prerequisite. The message is not self-contained tutorial material; it is a professional summary for an audience that already understands the technology stack.

Output Knowledge Created

Message 311 creates several distinct types of knowledge:

A validated configuration: The working launch command is the primary output—a repeatable, tested recipe for deploying GLM-5-NVFP4 on this specific hardware. Anyone with the same GPUs, the same SGLang build, and the same model can use this command as a starting point.

Performance benchmarks: The throughput tables provide reference numbers for capacity planning. Knowing that the system achieves approximately 210 output tokens per second at saturation (in=256, out=256) or 247 output tokens per second in decode-heavy mode allows operators to estimate serving capacity for their workloads.

A compatibility matrix: The message documents which backends work on SM120 hardware. The finding that flashinfer_trtllm is SM100-only, that flashmla_kv and flashmla_sparse produce NaN on decode, and that trtllm NSA works, is valuable knowledge for anyone deploying on Blackwell GPUs.

A bottleneck diagnosis: The conclusion that the system is compute-bound on small-batch MoE kernels, with supporting evidence from GPU utilization, power draw, memory bandwidth, and PCIe utilization, provides a foundation for future optimization efforts. If someone wants to improve performance, they now know where to focus.

Negative results: The documentation of what didn't work (CUDA graphs, expert parallelism, alternative MoE runners, FlashInfer autotuning) saves future investigators from repeating these experiments. In many ways, negative results are more valuable than positive ones because they narrow the search space.

A methodology template: The structure of the message—working configuration, key fix, benchmarks, profiling, ruled-out alternatives, bottleneck analysis—serves as a template for future deployment reports. The systematic approach demonstrated here is reproducible.

The Thinking Process Behind the Message

While message 311 is presented as a finished summary, the thinking process that produced it is visible in the preceding messages. The assistant's reasoning follows a clear pattern:

Hypothesis generation and testing: Each potential optimization (CUDA graphs, EP, alternative MoE runners, autotuning) was treated as a hypothesis to be tested. The assistant designed experiments, ran them, interpreted the results, and either accepted or rejected the hypothesis.

Iterative refinement: The assistant did not jump to conclusions. It started with a baseline configuration, verified it worked, then systematically varied one parameter at a time (memory fraction, CUDA graphs, MoE backend, NSA backend) while measuring the impact.

Root cause analysis: When the NaN crash occurred, the assistant did not simply try random fixes. It investigated the NSA backend implementations, checked SM120 compatibility, and identified the specific backends that fail.

Bottleneck characterization: Rather than guessing at performance limits, the assistant collected profiling data (GPU utilization, power, memory bandwidth, PCIe bandwidth) and used it to reason about the nature of the bottleneck. The conclusion that the system is compute-bound on small-batch kernels is supported by multiple independent measurements.

Principled elimination: The "What Didn't Help" section is the result of a systematic elimination process. Each alternative was tested or analyzed, found ineffective, and the reasoning for its ineffectiveness was documented.

This thinking process is a model of rigorous engineering investigation. The assistant demonstrates that understanding why something doesn't work is as important as finding something that does.

Potential Mistakes and Limitations

While message 311 is well-reasoned and supported by data, several limitations deserve consideration:

The benchmark methodology may miss variability. Single runs at each concurrency level do not capture run-to-run variance. In practice, throughput can vary due to GPU thermal state, memory fragmentation, or scheduler nondeterminism. The message presents the numbers as definitive without confidence intervals.

The bottleneck analysis focuses on GPU compute but may overlook software overhead. The SGLang server itself—the Python runtime, the scheduler, the HTTP handling—could contribute to the per-token latency. The message implicitly assumes that all time is spent in GPU kernels, but profiling the CPU side could reveal additional bottlenecks.

The virtualization diagnosis is acknowledged but not quantified in its impact on throughput. The message notes that cross-GPU transfers are forced through host memory, but the benchmark results are not compared against a non-virtualized baseline. Without that comparison, it is impossible to say how much of the ~11 tok/s single-stream latency is due to virtualization versus fundamental compute limitations.

The message does not explore alternative quantization schemes. GLM-5-NVFP4 uses a specific NVIDIA-proprietary quantization. It is possible that different quantization formats (e.g., FP8, INT4) would yield different performance characteristics, but this is not investigated.

The message does not discuss power or thermal constraints. The GPUs are at 55% power despite 100% utilization, which could indicate a power capping or thermal throttling issue that limits performance. This is noted but not explored.

These limitations do not invalidate the message's conclusions, but they define the boundaries within which those conclusions apply.

Conclusion

Message 311 is a masterful example of technical communication in the context of an AI-assisted coding session. It synthesizes hours of investigation into a clear, structured, and actionable summary. It documents what works, what doesn't, and why. It provides benchmark data for capacity planning and a bottleneck analysis for future optimization. It preempts follow-up questions by explicitly addressing alternatives that were tried and rejected.

The message succeeds because it is not just a list of commands and numbers—it is a narrative of investigation, with a clear beginning (the NaN crash), middle (systematic testing of alternatives), and end (a working configuration with characterized performance). It demonstrates that rigorous engineering methodology—hypothesis testing, iterative refinement, root cause analysis, and principled elimination—can be applied even in the complex, fast-moving world of large language model deployment.

For anyone attempting to deploy GLM-5-NVFP4 on Blackwell GPUs, or more broadly, for anyone seeking a template for documenting a complex ML infrastructure investigation, message 311 is both a reference and a model. It captures not just the answers, but the process of finding them.