The Single-Stream Latency Test: Understanding the Performance Ceiling of GLM-5-NVFP4 on Blackwell GPUs

Introduction

In the course of deploying the massive GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts (MoE) language model quantized to NVFP4 precision—across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant reached a critical juncture. After resolving a persistent NaN crash during decode by selecting compatible NSA backends (trtllm), successfully capturing CUDA graphs, and establishing baseline concurrent throughput of approximately 225 output tokens per second with 64 simultaneous requests, the assistant turned to a different but equally important performance metric: single-stream latency. Message 241 captures this precise moment—a focused measurement of how fast the model responds to a single user, revealing the fundamental per-request speed limit of the deployed system.

The Message: What Was Said

The message opens with a succinct summary of an initial measurement:

Single stream: 86 tokens in ~8 seconds = ~10.75 tok/s for single request.

The assistant then immediately executes a more rigorous test, issuing a curl command to the SGLang server with a prompt requesting a 2048-token explanation of neural network learning concepts. The result comes back cleanly:

Completion tokens: 2048, Prompt tokens: 32 real 3m3.784s

This yields approximately 11.1 tokens per second for a long single-stream request—consistent with the initial quick measurement and confirming that the single-request throughput hovers around 10–11 tok/s regardless of output length.

Why This Measurement Matters

The assistant's decision to measure single-stream latency was not arbitrary; it was the logical next step in a systematic performance characterization effort. The preceding messages had established that concurrent throughput peaked at roughly 225 output tok/s with 64 requests in flight. But concurrent throughput alone tells an incomplete story. A server might achieve high aggregate throughput by processing many requests in parallel, yet each individual request might experience poor latency. Conversely, if single-stream throughput is already close to the concurrent throughput divided by the number of concurrent requests, then the system is perfectly parallelized and the bottleneck lies elsewhere.

The ratio between these two numbers is illuminating. With ~225 tok/s at 64-way concurrency and ~11 tok/s for a single stream, the effective parallelization factor is about 20×. This means the server can handle roughly 20 simultaneous requests before saturating—after that, additional requests compete for the same computational resources without increasing aggregate throughput. This ratio is a key performance characteristic for capacity planning: it tells an operator how many concurrent users the system can serve before quality-of-service degrades.

The Measurement Methodology

The assistant employed a straightforward but effective methodology. The first measurement used a short prompt ("Count from 1 to 20, one number per line.") with a 512-token limit, producing 86 tokens in approximately 8 seconds. This gave a quick estimate of ~10.75 tok/s. Recognizing that short responses can be noisy—they include fixed overheads like prompt processing and the initial decode step that don't scale with output length—the assistant then ran a longer test.

The second test used a 2048-token generation with a prompt about neural network learning. By measuring over a much longer generation, the assistant obtained a more stable average: 2048 tokens in 183.784 seconds, or ~11.14 tok/s. The close agreement between the two measurements (10.75 vs 11.14 tok/s) confirms that the single-stream throughput is consistent and reproducible.

Notably, the assistant used the time command to measure wall-clock time, capturing the full end-to-end latency including network overhead, serialization, and deserialization. Since the server was running on the same machine (localhost), network latency was negligible, making this an accurate measure of true server response time.

What the Results Reveal

The single-stream measurement of ~11 tok/s places a hard upper bound on the user-perceived performance. A user sending a single request to this server will experience approximately 90 milliseconds per token of generated output. For a 256-token response, that translates to roughly 23 seconds of wait time. For longer generations like the 2048-token test, the user waits over three minutes.

This latency profile has important implications for the deployment's use case. For interactive applications like chatbots, 90ms per token is noticeably slow—users typically expect sub-50ms per token for fluid conversation. However, for batch processing or offline generation tasks where throughput matters more than per-request latency, the concurrent throughput of 225 tok/s is more relevant.

The fact that single-stream throughput is only about 5% of the concurrent throughput (11 vs 225 tok/s) suggests that the bottleneck is not in the per-step computation itself, but rather in how well the system can parallelize across requests. This is consistent with the assistant's earlier hypothesis that the primary bottleneck is MoE expert computation combined with all-reduce communication over PCIe. When only one request is active, the GPUs are underutilized—most of the 8 GPUs sit idle while waiting for the all-reduce to complete after each transformer layer. With many concurrent requests, the batch size increases, improving GPU utilization and amortizing the communication overhead across more tokens.

The Broader Context

This measurement sits within a larger narrative of deploying a cutting-edge model on non-ideal hardware. The eight RTX PRO 6000 Blackwell GPUs are connected via PCIe rather than NVLink, and they run inside a Proxmox VM (KVM/QEMU) with no direct GPU peer-to-peer support. The assistant had already discovered that cross-GPU transfers must go through host memory, adding significant latency to the all-reduce operations that MoE models require after every layer.

The single-stream latency test provides the final piece of the performance puzzle. Combined with the concurrent throughput benchmarks, it gives a complete picture: the system can deliver ~225 tok/s under load, but individual users experience ~11 tok/s. This asymmetry is characteristic of systems where the bottleneck is communication-bound rather than compute-bound—the GPUs have ample compute capacity, but the PCIe fabric cannot keep up with the all-reduce traffic, forcing the system to rely on batching multiple requests to keep the GPUs fed.

Conclusion

Message 241 represents a moment of clarity in the deployment process. By measuring single-stream latency, the assistant completed the performance characterization of the GLM-5-NVFP4 deployment, revealing the fundamental trade-off between per-request latency and aggregate throughput. The ~11 tok/s single-stream speed, confirmed across two independent measurements, establishes the user-facing performance baseline, while the ~225 tok/s concurrent throughput demonstrates the system's capacity under load. Together, these numbers tell the story of a deployment constrained not by GPU compute but by the PCIe communication fabric—a finding that would guide any future optimization efforts, whether through expert parallelism, kernel fusion, or infrastructure upgrades to reduce virtualization overhead.