The Final Benchmark: Closing the Loop on MoE Backend Evaluation for GLM-5-NVFP4

In the course of deploying the massive GLM-5-NVFP4 mixture-of-experts model across eight RTX PRO 6000 Blackwell GPUs, a single message at index 303 serves as the quiet capstone to an intensive investigation into MoE kernel backends. The message is deceptively brief — a benchmark result and a single-stream latency test — but it represents the culmination of a multi-threaded debugging and tuning effort that spanned dozens of previous messages. To understand why this particular message was written, one must trace the narrative arc that led to it: a journey through NaN crashes, autotuner investigations, CUDA graph capture attempts, and a deepening understanding of the Blackwell SM120 architecture's idiosyncrasies.

The Context That Produced This Message

The subject message did not emerge in a vacuum. It was the product of a sustained investigation into why the GLM-5-NVFP4 model — a 453-GB mixture-of-experts model with 256 experts and a hidden size of 3584 — was underperforming on the RTX PRO 6000 GPUs. Earlier in the session ([msg 281] through [msg 302]), the assistant had systematically worked through a series of hypotheses about performance bottlenecks.

The immediate precursor to message 303 was a deep dive into FlashInfer's autotuning infrastructure. The assistant had discovered that FlashInfer's autotuner.py looks for device-specific tuning configuration files, and that no such file existed for the NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition GPU ([msg 285]). Configurations existed only for the datacenter Blackwell GPUs (B200 and GB200), which use the SM100 architecture — not the SM120 architecture of the RTX PRO 6000. This meant the flashinfer_trtllm MoE backend, which relies on autotuned tactic choices, was unavailable on these GPUs.

However, the assistant then made a crucial discovery: the flashinfer_cutlass MoE path does not use the FlashInfer autotuner at all ([msg 297]). The CUTLASS-based fused MoE kernels use a fixed kernel implementation with tune_max_num_tokens as the only knob. This meant there was no hidden performance to unlock through autotuning — the SM120 CUTLASS kernels were simply what they were. This realization reframed the investigation: instead of searching for a better kernel configuration, the assistant needed to accept the current throughput as the hardware baseline and explore whether alternative MoE backends could offer any improvement.

What the Message Actually Says

The subject message reports two benchmark results. First, the flashinfer_cutedsl (CuteDSL) MoE backend achieves approximately 206 output tokens per second and 473 total tokens per second under a batch workload of 64 concurrent requests with 256-token input and 256-token output. The assistant explicitly compares this to the flashinfer_cutlass results of 195–225 output tok/s, concluding there is "not a significant difference."

Second, the assistant runs a single-stream latency test — a single request with 128 input tokens and 256 output tokens — which yields 11.10 output tokens per second over a 21.27-second duration. This single-stream figure is dramatically lower than the batched throughput, revealing that the model's per-request latency is poor even though the system can aggregate throughput across many concurrent requests.

The Reasoning Behind the Message

The message reveals two distinct lines of reasoning. The first is comparative: the assistant is systematically evaluating each available MoE backend option to determine whether any provides a meaningful advantage. Having tested flashinfer_cutlass and now flashinfer_cutedsl, the assistant is building a performance matrix. The conclusion that they are comparable is significant — it means the bottleneck is not in the choice of MoE kernel implementation but in something more fundamental, such as the communication overhead between GPUs or the sheer number of kernel launches required by the 78-layer, 256-expert architecture.

The second line of reasoning is diagnostic. The single-stream latency test reveals that a single request takes over 21 seconds to generate 256 tokens. This is a critical data point because it isolates the per-request overhead from the benefits of batching. In the batch benchmark, the system achieves 206 tok/s by processing 64 requests simultaneously, meaning each individual request experiences roughly the same latency as the single-stream test but the aggregate throughput is high. The 11.10 tok/s figure confirms that the model is latency-bound at the single-request level — a characteristic that would make it unsuitable for interactive applications requiring low response times.

Assumptions and Their Implications

The message operates under several implicit assumptions. First, the assistant assumes that the flashinfer_cutedsl backend is a valid alternative that could potentially outperform flashinfer_cutlass. This assumption was reasonable given that CuteDSL is a different kernel generation framework (using NVIDIA's CuteDSL DSL rather than CUTLASS templates), and different code generation strategies can produce meaningfully different performance characteristics even on the same hardware.

Second, the assistant assumes that single-stream latency is a meaningful metric for this deployment. This assumption is valid for understanding the user experience of individual requests, but it must be interpreted carefully. The 11.10 tok/s figure includes model loading overhead, KV cache initialization, and the full prefill-plus-decode cycle for a single request. In a production system with persistent batching, the marginal latency of adding one more request to an existing batch would be lower, but the cold-start latency for the first request in a batch would approach this figure.

Third, the assistant assumes that the benchmark parameters (random input/output lengths of 128/256 for single-stream, 256/256 for batch) are representative of the expected workload. This is a standard benchmarking practice, but it carries the risk that real-world workloads with different token distributions or longer sequences could produce different results.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message 303, one needs several pieces of contextual knowledge. The reader must understand that GLM-5-NVFP4 is a mixture-of-experts transformer with 256 experts, each with a hidden size of 3584 and intermediate size of 512, spread across 78 layers. The model uses NVFP4 quantization (4-bit floating point), which is a novel format that requires specialized kernel support. The RTX PRO 6000 Blackwell GPUs use the SM120 architecture, which differs from the SM100 architecture used in datacenter Blackwell GPUs like the B200 — this distinction matters because kernel implementations are often architecture-specific.

One must also understand the MoE runner backend landscape: flashinfer_cutlass uses CUTLASS-generated fused MoE kernels, flashinfer_cutedsl uses CuteDSL-generated kernels, and flashinfer_trtllm uses TensorRT-LLM-style kernels with autotuned tactic selection (but is SM100-only). The --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags refer to the attention mechanism (NSA — presumably "Nested Sparse Attention" or similar), not the MoE, and were set to trtllm because the FlashInfer NSA backend caused NaN crashes on SM120 GPUs.

Output Knowledge Created by This Message

This message produces several concrete pieces of knowledge. First, it establishes that the flashinfer_cutedsl MoE backend performs comparably to flashinfer_cutlass on SM120 GPUs, with no backend offering a decisive advantage. Second, it provides the final throughput baseline for the deployment: approximately 206 output tok/s and 473 total tok/s under batch workload, with 11.10 tok/s single-stream latency. Third, it implicitly confirms that the performance bottleneck is not in the MoE kernel implementation but in some other aspect of the system — likely the cross-GPU communication overhead identified in subsequent analysis ([chunk 2.0]).

The single-stream latency figure of 11.10 tok/s is particularly valuable because it quantifies the per-request overhead. At 256 output tokens taking 21.27 seconds, the average time per token is approximately 83 milliseconds. For a model with 78 layers and 256 experts, this implies that each forward pass through the model takes on the order of a millisecond or more — consistent with a system where small-kernel launch overhead and PCIe-based inter-GPU communication dominate the runtime.

The Thinking Process Visible in the Message

The message's structure reveals the assistant's thought process. The assistant leads with the batch benchmark result, immediately contextualizing it against the previous flashinfer_cutlass numbers. This comparative framing shows that the assistant is thinking in terms of a decision matrix — evaluating options against each other to find the best configuration. The phrase "not a significant difference" is a conclusion drawn from comparing the 206 tok/s figure against the 195–225 tok/s range observed earlier, which shows that the assistant has internalized the variance in the previous measurements and recognizes that the new result falls within the same noise band.

The decision to then run a single-stream latency test reveals a shift in diagnostic strategy. Having established that the MoE backend choice doesn't materially affect throughput, the assistant pivots to understanding the fundamental latency characteristics of the system. The single-stream test is a classic diagnostic technique: by removing the confounding factor of batching, it isolates the per-request overhead. The resulting 11.10 tok/s figure is dramatically lower than the batched throughput, which confirms that the system is heavily dependent on batching to amortize fixed overheads.

The assistant does not draw explicit conclusions from these numbers in the message itself — the message is purely a data-gathering step. However, the choice of which data to gather reveals the assistant's hypothesis: that the performance limitation is architectural rather than configurational, and that no amount of backend switching will overcome the fundamental constraints of the hardware and virtualization environment. This hypothesis would be confirmed in subsequent analysis when the assistant diagnosed the Proxmox VM's lack of direct GPU peer-to-peer support as the primary bottleneck ([chunk 2.0]).

The Broader Significance

Message 303 represents the moment when the assistant stopped searching for a better kernel configuration and accepted the hardware baseline. The comparable performance of flashinfer_cutlass and flashinfer_cutedsl — two independently developed MoE kernel implementations — strongly suggests that both are operating near the hardware's performance ceiling for this workload. Any further gains would require addressing higher-level bottlenecks: the PCIe topology, the virtualization overhead, or the model architecture itself.

In the context of the full session, this message marks the transition from kernel tuning to system-level diagnosis. The assistant would go on to investigate expert parallelism feasibility, measure cross-GPU bandwidth, and ultimately identify the Proxmox VM's lack of direct GPU peer-to-peer support as the root cause of the latency limitation. But the seed of that investigation was planted here, in the quiet acknowledgment that the MoE backends were all performing similarly and the real bottleneck lay elsewhere.