The Moment of Truth: Benchmarking GLM-5-NVFP4 at Scale

In the high-stakes world of large language model inference optimization, few moments are as tense as the first benchmark run after a major configuration change. Message <msg id=703> captures exactly such a moment — the assistant dispatches two concurrent benchmark commands against a freshly restarted SGLang server running the 405-billion-parameter GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message is deceptively simple: two ssh commands, each launching sglang.bench_serving at different concurrency levels. But behind this sparse surface lies a rich story of iterative optimization, hard-won performance gains, and the frustrating discovery of a latent software bug that would block further progress.

The Road to This Message

To understand why message <msg id=703> was written, we must trace the chain of decisions that led to it. The assistant had been engaged in a multi-session effort to deploy and optimize GLM-5-NVFP4, a Mixture-of-Experts model that pushes the boundaries of what consumer-grade Blackwell hardware can handle. The journey had already been arduous: resolving NaN crashes during decode by selecting the correct NSA attention backends ([msg 674]), diagnosing virtualization-induced PCIe bottlenecks in a Proxmox environment, migrating to an LXC container for bare-metal GPU topology, and finally achieving a baseline throughput of around 880 tokens per second.

The immediate precursor to message <msg id=703> was a breakthrough. In messages <msg id=693> through <msg id=695>, the assistant had enabled FlashInfer CUTLASS MoE autotune for SM120 (the Blackwell compute capability) by patching model_runner.py to include flashinfer_cutlass in the autotune list. Combined with raising --max-running-requests from 64 to an auto-detected 2048, and adding --disable-cuda-graph and --disable-radix-cache, the throughput had jumped from ~880 tok/s to an astonishing 1,950 tok/s at 256 concurrency — more than double the previous peak. The assistant's excitement was palpable: "1,950 tok/s at 256 concurrency!! That's more than double what we had before (879 tok/s)!" ([msg 695]).

But triumph was short-lived. When the assistant attempted to push to 512 concurrency, the server crashed with a cryptic error: 'PrefillMetadata' object has no attribute 'page_table_1_flattened' ([msg 697]). This was a bug in the NSA (Native Sparse Attention) code path, triggered by chunked prefill with large batches. The assistant made a judgment call: restart the server with --max-running-requests 1024 (down from 2048) to avoid pushing too far, while keeping the other optimizations intact. This restart produced message <msg id=703> — the first comprehensive benchmark sweep of the new, more conservatively configured server.

What the Message Actually Says

The message contains two parallel tool calls, both executing ssh commands to the remote server at 10.1.230.174. The first runs a benchmark with 256 prompts (concurrency level 256), the second with 512 prompts. Both use the same workload parameters: random input length of 256 tokens, random output length of 128 tokens, and infinite request rate (meaning requests are fired as fast as possible, saturating the server).

The output is fragmentary. For the 256-concurrency run, we see only #Output tokens: 16548. For the 512-concurrency run, we see #Output tokens: 32671. The grep filter for key metrics — Total token|Output token|Peak output|Mean TPOT|Concurrency — returned empty results beyond these token counts. This is the first clue that something went wrong.

The Hidden Story: What Actually Happened

Following the thread into messages <msg id=704> and <msg id=705>, we discover the full story: both benchmarks crashed the server. The 256-concurrency run failed with the same page_table_1_flattened error that had plagued the earlier 512-concurrency attempt. The 512-concurrency run likely also failed, though the output was cut off by a connection error in the benchmarking client.

This is a critical moment of discovery. The assistant had assumed that reducing --max-running-requests from 2048 to 1024 would avoid the crash. But the bug was not about memory exhaustion or request queue depth — it was a code defect in the NSA attention implementation that manifested whenever chunked prefill processed a sufficiently large batch, regardless of the configured limit. The max-running-requests parameter controls how many requests the scheduler will admit, but the crash occurred during the prefill phase of individual requests within a batch. The bug was structural, not a matter of tuning.

Assumptions and Their Consequences

The message reveals several assumptions, some correct and some not:

Correct assumption: That the autotune and higher concurrency ceiling would improve throughput. This was validated by the earlier 1,950 tok/s result.

Incorrect assumption: That reducing --max-running-requests to 1024 would prevent the crash. The crash was caused by a missing attribute (page_table_1_flattened) in the PrefillMetadata object during NSA attention's chunked prefill logic — a bug independent of the request queue depth.

Implicit assumption: That the server was stable enough to complete a benchmark sweep. The assistant had verified the server was "ready" (message <msg id=701>) and had successfully run single-request and 64-concurrency benchmarks (message <msg id=702>). But stability at low concurrency did not guarantee stability at high concurrency, because the bug was triggered by batch size, not request count per se.

Assumption about the bug's nature: The assistant initially attributed the crash to OOM or memory pressure, writing "The server crashed! Likely OOM" ([msg 695]). Only after examining the logs did the true nature of the bug become clear. This shaped the restart strategy — reducing max-running-requests instead of fixing the underlying code defect.

The Thinking Process Visible in the Message

The message itself is terse — just two commands — but the reasoning behind it is visible in the surrounding context. The assistant is executing a systematic benchmark protocol:

  1. Warm-up / baseline: Single request to measure latency, then 64 concurrency to verify the server handles moderate load.
  2. Scaling up: Jump to 256 concurrency to test the throughput ceiling.
  3. Pushing limits: 512 concurrency to find the breaking point. This is textbook performance engineering: start low, validate, then increase load until you find the bottleneck or failure point. The assistant is looking for the "knee" in the throughput curve — the concurrency level where throughput stops scaling linearly. The parallel execution of the two benchmarks (both dispatched in the same round) is also revealing. The assistant is being efficient, running independent tests simultaneously. But this decision carries risk: if the 256-concurrency benchmark crashes the server, the 512-concurrency benchmark will also fail, wasting time. The assistant likely assumed that 256 concurrency would succeed (based on the earlier successful run at this level) and that 512 would either succeed or fail independently.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. The hardware topology: Eight RTX PRO 6000 Blackwell GPUs with SM120 compute capability, connected via PCIe in a virtualized environment with known P2P limitations.
  2. The model architecture: GLM-5-NVFP4 is a 405B-parameter MoE model using NVFP4 quantization, requiring specialized attention kernels (NSA with trtllm backend) and MoE runners (flashinfer_cutlass).
  3. The optimization history: The autotune patch, the CUDA graph disable, the max-running-requests increase — all decisions made in the preceding messages.
  4. The crash history: The page_table_1_flattened bug that first appeared at 512 concurrency in the previous server instance.
  5. The SGLang server architecture: How tensor parallelism (TP=8) distributes the model across GPUs, how the scheduler manages request queues, and how chunked prefill processes large batches.

Output Knowledge Created

This message produces several important outputs:

  1. Confirmation that the bug is systematic: The crash at 256 concurrency (not just 512) proves the page_table_1_flattened issue is not a rare edge case but a fundamental defect in the NSA attention code path that triggers reliably at moderate batch sizes.
  2. A performance data point: The 256-concurrency run generated 16,548 output tokens before crashing, and the 512-concurrency run generated 32,671. While incomplete, these numbers hint at the throughput the server could achieve if the bug were fixed.
  3. A clear next direction: The assistant now knows that further optimization requires fixing the NSA attention bug, not just tuning server parameters. This leads directly to the investigation in messages <msg id=706> and beyond, where the assistant traces the page_table_1_flattened attribute to its source in the dequant_k_cache.py file.

The Broader Significance

Message <msg id=703> is a classic example of the "breakthrough and setback" pattern that characterizes real-world ML engineering. The assistant had just achieved a stunning 2.2x throughput improvement through careful configuration and code patching. The natural next step was to validate and extend that gain. Instead, the validation itself revealed a hidden defect that blocked further progress.

This pattern is deeply instructive. It shows that:

Conclusion

Message <msg id=703> is a pivotal moment in the optimization journey. It represents the intersection of two trajectories: the upward arc of performance improvement through autotune and configuration tuning, and the downward arc of system stability as hidden bugs are exposed by increased load. The assistant's response to this setback — methodically investigating the NSA attention code to understand and fix the root cause — demonstrates the disciplined approach required for production-grade ML inference optimization. The message may appear to be just two benchmark commands with incomplete output, but it is in fact the critical discovery that shapes the entire subsequent direction of the optimization effort.