Benchmarking After the Autotune: Validating GLM-5-NVFP4 Performance on 8× Blackwell GPUs

Introduction

In the long arc of deploying the GLM-5-NVFP4 mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message 702 represents a moment of cautious validation. The assistant has just restarted the SGLang inference server after a crash, having made two critical changes: enabling the FlashInfer CUTLASS MoE autotune for the SM120 architecture and capping --max-running-requests at 1024 to avoid memory exhaustion. Now, with the server reporting "fired up" and accepting requests, the assistant announces a "comprehensive benchmark sweep" and runs the first two data points: a single-stream latency test and a 64-concurrency throughput test. This message is the opening of an experimental campaign to measure whether the optimizations actually work, and it reveals both the power and the fragility of tuning a cutting-edge inference stack for non-datacenter hardware.

The Full Message

The assistant wrote:

Server is ready. Let me now run a comprehensive benchmark sweep:

>

```bash ssh root@10.1.230.174 "source /root/ml-env/bin/activate && echo '=== 1 request (single-stream) ===' && python3 -m sglang.bench_serving --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --base-url http://localhost:8000 --dataset-name random --num-prompts 1 --random-input-len 256 --random-output-len 500 --request-rate inf 2>&1 | grep -E 'Total token|Output token|Peak output|Mean TPOT|Mean E2E'" === 1 request (single-stream) === #Output tokens: 236 Output token throughput (tok/s): 8.21 Peak output token throughput (tok/s): 11.00 Total token throughput (tok/s): 9.53 Mean E2E Latency (ms): 28729.27 Mean TPOT (ms): 121.58

>

``bash ssh root@10.1.230.174 "source /root/ml-env/bin/activate && echo '=== 64 concurrency ===' && python3 -m sglang.bench_serving --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --base-url http://localhost:8000 --dataset-name random --num-prompts 64 --random-input-len 256 --random-output-len 128 --request-rate inf 2>&1 | grep -E 'Total token|Output token|Peak output|Mean TPOT|Concurrency:'" === 64 concurrency === #Output tokens: 3731 Output token throughput (tok/s): 202.53 Peak output token throughput (tok/s): 409.00 Total token throughput (tok/s): 700.08 Concurrency: 34.48 Mean TPOT (ms): 146.41 ``

Why This Message Was Written: The Context of a Crash

To understand why message 702 exists, one must look at what happened immediately before. The assistant had just achieved a stunning result: after patching model_runner.py to include flashinfer_cutlass in the autotune list and restarting the server without a cap on --max-running-requests, a 256-concurrency benchmark delivered 1,950 tok/s — more than double the previous best of 879 tok/s. This was a breakthrough. But when the assistant immediately tried to push to 512 concurrency, the server crashed with a cryptic error: 'PrefillMetadata' object has no attribute 'page_table_1_flattened'.

The crash forced a restart. The assistant made a judgment call: reduce --max-running-requests from the auto-detected 2048 down to 1024, hoping this would prevent the memory or metadata issue while still allowing high throughput. The server was restarted with the same core optimizations — autotune enabled, --disable-cuda-graph, --attention-backend flashinfer, --moe-runner-backend flashinfer_cutlass — and after several minutes of model loading and autotuning, it came back online.

Message 702 is the first thing the assistant does after that restart. It is a sanity check. Before declaring victory or diving into deeper optimization, the assistant needs to know: Does the server actually work? Is it stable? What is the baseline performance with this new configuration? The "comprehensive benchmark sweep" is announced with the confidence of someone who believes they have fixed the issue, but the choice to start with a single request — the gentlest possible test — betrays a healthy caution.

The Benchmark Methodology: Why Start at 1 and 64?

The sweep begins with two very different workloads. The single-stream test (--num-prompts 1) measures the raw latency of the system: how fast can one request be processed when there is no queue contention? The 64-concurrency test measures throughput under moderate load, directly comparable to earlier benchmarks that had been run at the same concurrency level.

The single-stream test uses a longer output length (500 tokens) compared to the 64-concurrency test (128 tokens). This is intentional: single-stream latency is more meaningful with a longer generation, as it captures the steady-state decode performance rather than the startup overhead. The 64-concurrency test uses a shorter output to match the earlier benchmarks and allow direct comparison.

The assistant filters the output with grep -E to show only the key metrics: total token throughput, output token throughput, peak output, mean TPOT (time per output token), and mean end-to-end latency. This is a pragmatic choice — the full benchmark output is verbose, and the assistant is looking for specific numbers to validate the configuration.

What the Numbers Reveal

The single-stream results are sobering but expected. With a mean TPOT of 121.58 ms, each token takes about 122 milliseconds to generate. This translates to roughly 8.2 output tokens per second. For a 405-billion-parameter model spread across eight GPUs with tensor parallelism, this latency is not surprising — the model is enormous, and the PCIe interconnect between the GPUs adds significant communication overhead. The peak output throughput of 11 tok/s suggests that the first few tokens (which may benefit from KV cache prefilling) are faster than the steady state.

The 64-concurrency results tell a more nuanced story. The total token throughput of 700.08 tok/s is actually lower than the 757 tok/s achieved in an earlier run before the autotune was enabled. This is a critical observation: at low concurrency, the autotune and the --disable-cuda-graph flag have not helped. In fact, they may have hurt. The actual concurrency reached was only 34.48 out of 64 prompts submitted — meaning the server could not keep all 64 requests in flight simultaneously, likely because the KV cache or memory pool was constrained.

The mean TPOT at 64 concurrency (146.41 ms) is only about 20% worse than the single-stream TPOT (121.58 ms), which is actually quite good — it means the system is not collapsing under moderate load. But the throughput is not yet where the assistant knows it can be, because the 256-concurrency run from the previous server instance achieved 1,950 tok/s.

The Unspoken Assumptions

Several assumptions underpin this message. First, the assistant assumes that the crash was caused by excessive concurrency (2048 max-running-requests) rather than a fundamental bug in the NSA decode path. By reducing to 1024, the assistant is betting that the crash was a resource issue, not a code bug. Second, the assistant assumes that the autotune results from the previous server instance are still valid — that the CUTLASS MoE tactics selected during autotuning are deterministic and will produce the same performance. Third, the assistant assumes that --disable-cuda-graph is beneficial overall, based on the K2-Thinking reference run that achieved 5,816 tok/s without CUDA graphs, even though the low-concurrency numbers here suggest otherwise.

There is also an implicit assumption that the benchmark sweep will complete without another crash. The assistant is probing the system's stability as much as its performance. Each successful benchmark run is evidence that the configuration is viable.

What This Message Does Not Show

The most notable absence in message 702 is the rest of the "comprehensive benchmark sweep." Only two data points are presented: 1 request and 64 requests. The sweep was presumably intended to continue to 256, 512, and perhaps 1024 concurrency — the levels where the autotune and high max-running-requests should shine. The message ends without those results, leaving the reader (and the assistant) waiting to see if the higher concurrencies will deliver the promised throughput or trigger another crash.

This incompleteness is itself informative. It shows that the assistant is proceeding step by step, gathering data at each level before escalating. The crash at 512 concurrency in the previous server instance has made the assistant cautious. The sweep is not just a measurement exercise; it is a stability test.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the message. The phrase "comprehensive benchmark sweep" signals a systematic approach. The choice to run single-stream first shows an understanding that latency and throughput are different dimensions that need separate measurement. The use of grep to extract specific metrics shows familiarity with the benchmark tool's output and a focus on actionable numbers rather than raw logs.

The assistant is also thinking about comparability. The 64-concurrency test uses the same parameters (256 input, 128 output) as earlier benchmarks, allowing direct comparison with the 757 tok/s result from before the autotune. This is the mark of someone who is building a mental model of the system's performance landscape, not just collecting numbers.

Input Knowledge Required

To fully understand message 702, one needs to know:

• What tensor parallelism (TP=8) means for model distribution across GPUs
• The significance of FlashInfer CUTLASS MoE autotune for optimizing mixture-of-experts kernel execution
• What --max-running-requests controls in the SGLang server (the maximum number of concurrent requests the scheduler will admit)
• The difference between prefill and decode phases in transformer inference
• What TPOT (time per output token) and end-to-end latency measure
• The hardware context: 8× RTX PRO 6000 Blackwell GPUs with PCIe interconnect, not NVLink

Output Knowledge Created

This message produces two concrete data points:

1. Single-stream latency: 121.58 ms TPOT, 28.7 seconds end-to-end for a 256→500 token request
2. 64-concurrency throughput: 700.08 total tok/s, 202.53 output tok/s, with actual concurrency of 34.48 These numbers serve as the baseline for the new configuration. They will be compared against higher-concurrency results to determine whether the autotune investment paid off. They also reveal that at low concurrency, the configuration is not yet optimal — the throughput is actually slightly worse than before.

Conclusion

Message 702 is a moment of measured optimism in a long debugging session. The assistant has recovered from a crash, applied two significant optimizations (autotune and higher concurrency cap), and is now methodically testing the system's performance. The single-stream and 64-concurrency results are modest, but the assistant knows from experience that the real payoff comes at higher batch sizes. The sweep will continue, and the next data points — 256, 512, 1024 concurrency — will determine whether the autotune was worth the effort. This message captures the tension between the promise of optimization and the reality of system stability, a tension that defines the work of deploying large language models on non-standard hardware.