The 512 Concurrency Benchmark: A Pivotal Data Point in Scaling GLM-5-NVFP4 on Blackwell

Introduction

In the course of a marathon optimization session for deploying the GLM-5-NVFP4 mixture-of-experts (MoE) model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, one brief message stands out as a critical inflection point. Message <msg id=713> captures the moment when the assistant, having just resolved a persistent crash bug in the Native Sparse Attention (NSA) backend, pushes the inference server to 512 concurrent requests and records a total throughput of 2,800 tokens per second. This single data point—sandwiched between the 256-concurrency validation and the eventual 1,024-concurrency peak of 3,740 tok/s—represents far more than a number. It embodies the culmination of hours of debugging, kernel patching, configuration tuning, and architectural workarounds, all compressed into a single bash command and its output.

The Message in Full

The subject message reads:

[assistant] Good — 1,846 tok/s at 256 concurrency with peak 1,492 tok/s output! And the single chat request generated 500 tokens in 62.6s = ~8 tok/s. The no-radix-cache version seems stable. Let me push to 512 and 1024:

[bash] ssh root@10.1.230.174 "source /root/ml-env/bin/activate && python3 -m sglang.bench_serving --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --base-url http://localhost:8000 --dataset-name random --num-prompts 512 --random-input-len 256 --random-output-len 128 --request-rate inf 2>&1 | tail -25"

Output token throughput (tok/s): 920.98 Peak output token throughput (tok/s): 2348.00 Peak concurrent requests: 512 Total token throughput (tok/s): 2800.25 Concurrency: 354.86 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 24586.87 Median E2E Latency (ms): 24778.40 P90 E2E Latency (ms): 34285.42 P99 E2E Latency (ms): ...

At first glance, this appears to be a routine benchmark run. But to understand why this message matters, one must trace the thread of obstacles that preceded it.

Why This Message Was Written: The Reasoning and Motivation

The assistant's motivation is straightforward but grounded in a complex chain of events. The overarching goal of this segment of the conversation is to maximize inference throughput for the GLM-5-NVFP4 model—a 405-billion-parameter MoE model quantized to FP4—on eight Blackwell GPUs. The assistant had already achieved a major breakthrough by enabling FlashInfer CUTLASS MoE autotune for the SM120 architecture (the compute capability of the RTX PRO 6000 Blackwell GPUs) and raising --max-running-requests from 64 to 1024. These changes had yielded 1,950 tok/s at 256 concurrency ([msg 695]), more than doubling the previous best of 879 tok/s.

However, that victory was immediately followed by a crash. When the assistant attempted a 512-concurrency benchmark, the server died with an AttributeError: 'PrefillMetadata' object has no attribute 'page_table_1_flattened' ([msg 696]). This was a bug in the NSA attention path, triggered by chunked prefill with large batches when prefix sharing (radix cache) was enabled. The assistant spent several messages ([msg 706] through [msg 709]) tracing the bug through the sglang source code, identifying that page_table_1_flattened was only set when prefix sharing was active, but the dequantization code path unconditionally accessed it.

The workaround was decisive: restart the server with --disable-radix-cache ([msg 710]). This eliminated the bug by preventing prefix sharing from ever occurring. The 256-concurrency re-test in <msg id=712> confirmed stability with 1,846 tok/s—slightly lower than the 1,950 tok/s achieved with radix cache enabled, but reliable.

Thus, <msg id=713> is written with two motivations. First, validation: the assistant needs to confirm that the no-radix-cache configuration is not just stable at 256 concurrency but scales predictably. Second, exploration: the assistant is systematically mapping the throughput curve by sweeping concurrency levels (1, 64, 256, 512, 1024) to find the ceiling. The phrase "The no-radix-cache version seems stable. Let me push to 512 and 1024" reveals a methodical, almost scientific mindset—each data point informs the next, and stability at one level is the prerequisite for testing the next.

How Decisions Were Made

The decision to benchmark at 512 concurrency was not arbitrary. It followed a deliberate progression:

Single-stream latency (1 request): 8.21 tok/s output, 121.58 ms TPOT — establishes baseline latency.
Low concurrency (64 requests): 700 tok/s total — shows the system under light load.
Medium concurrency (256 requests): 1,846 tok/s — validates the no-radix-cache fix and shows healthy scaling.
High concurrency (512 requests): the current message — tests whether scaling continues.
Extreme concurrency (1,024 requests): the next step — finds the ceiling. This stair-step approach is classic performance engineering. Each level builds confidence before committing to a more demanding test. The assistant also chose specific benchmark parameters: 256 input tokens and 128 output tokens per request, using the random dataset with infinite request rate (--request-rate inf). These choices create a compute-bound scenario where the model's raw throughput is the limiting factor, not network latency or request scheduling. The server configuration embedded in this test is itself the product of many decisions. The launch command from <msg id=710> includes flags that were carefully selected through earlier debugging: - --disable-radix-cache: the workaround for the NSA bug - --disable-cuda-graph: avoids another crash path related to prefill metadata - --enable-flashinfer-allreduce-fusion: an attempt to improve GPU utilization (though later found to be ineffective on SM120) - --moe-runner-backend flashinfer_cutlass: the autotuned MoE kernel backend - --max-running-requests 1024: raised from the default of 64 to allow large batch processing - --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm: the working NSA backends identified after the NaN crash saga in earlier segments Each flag represents a decision point where the assistant chose one path over alternatives, often after hours of debugging.

Assumptions Embedded in This Message

The message rests on several assumptions, some explicit and some implicit:

The stability assumption. The assistant assumes that because the server survived 256 concurrent requests without crashing, it will survive 512. This is reasonable but not guaranteed—the page_table_1_flattened bug was intermittent, triggered by specific patterns of cached tokens and new requests. Higher concurrency increases the probability of hitting edge cases.

The scaling assumption. The assistant implicitly assumes that throughput will continue to increase with concurrency, at least up to some ceiling. This is the fundamental premise of batch processing in LLM inference: more concurrent requests allow the GPUs to amortize the cost of loading model weights across more tokens. However, the assistant does not yet know where the ceiling lies—that is precisely what this benchmark is designed to discover.

The representativeness assumption. The benchmark uses random input/output lengths (256/128 tokens) with the random dataset. The assistant assumes this is representative enough to guide optimization decisions. Real-world traffic patterns might differ significantly (e.g., longer outputs, variable lengths, prefix sharing), but for comparative benchmarking, a standardized workload is essential.

The measurement validity assumption. The bench_serving tool from sglang is assumed to produce accurate throughput measurements. The assistant trusts that the reported 2,800 tok/s reflects genuine end-to-end throughput, not artifacts of the benchmarking tool.

Mistakes and Incorrect Assumptions

While the message itself is correct in its reported numbers, several underlying assumptions later proved incomplete:

The allreduce fusion assumption. The assistant launched the server with --enable-flashinfer-allreduce-fusion, believing this would improve GPU utilization. In subsequent messages ([msg 716] onward), the assistant discovers that FlashInfer's allreduce fusion is unsupported on SM120—the TRT-LLM communication kernels only target SM90 (Hopper) and SM100 (datacenter Blackwell). The fusion flag was silently ignored. This means the 2,800 tok/s was achieved without allreduce fusion, and the GPU power draw remained at ~250W out of 600W TDP—a significant underutilization that the assistant would later spend considerable effort trying to address.

The linear scaling expectation. The jump from 256 to 512 concurrency yielded only a 1.52× throughput increase (1,846 → 2,800 tok/s), despite doubling the concurrency. This sub-linear scaling hints at a bottleneck—either PCIe communication, MoE kernel efficiency, or memory bandwidth—that the assistant has not yet identified. The message presents the 2,800 tok/s as a victory, which it is, but the diminishing returns foreshadow the ceiling that will appear at 1,024 concurrency (3,740 tok/s, only 1.34× over 512).

The stability guarantee. The server survived this benchmark, but the underlying NSA bug is not fixed—merely worked around. The --disable-radix-cache flag eliminates the triggering condition but also disables a useful feature (prefix caching) that could improve latency for repeated prompts. The assistant implicitly accepts this trade-off without quantifying its cost.

Input Knowledge Required

To fully understand this message, one must possess considerable context:

Knowledge of the model architecture. GLM-5-NVFP4 is a 405B MoE model quantized to FP4 using NVIDIA's ModelOpt framework. It uses Native Sparse Attention (NSA) with separate prefill and decode backends. The --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags reflect the assistant's earlier discovery that only the TRT-LLM NSA backends work on Blackwell GPUs without producing NaN values ([msg 652]).

Knowledge of the hardware topology. The eight RTX PRO 6000 Blackwell GPUs are in a Proxmox VM with virtualized PCIe topology. Earlier segments ([msg 630] through [msg 650]) revealed that each GPU sits on its own PCIe root complex, preventing direct P2P DMA between GPUs. This forces all GPU-to-GPU communication through the host's PCIe root ports, introducing latency that limits allreduce performance. The assistant worked around this by moving to an LXC container on the Proxmox host, which provided bare-metal GPU topology.

Knowledge of the sglang server stack. The assistant is intimately familiar with sglang's configuration flags, having spent hours debugging crashes, tuning parameters, and patching source code. The flags in the launch command represent accumulated knowledge from dozens of failed experiments.

Knowledge of the bug history. The page_table_1_flattened crash is only understandable in the context of the NSA attention implementation, the FlashInfer backend, and the interaction between prefix caching and FP8 KV cache dequantization. The assistant traced this bug through multiple source files before arriving at the workaround.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

A validated throughput data point. 2,800 tok/s at 512 concurrency is now a known quantity. This serves as a baseline for future optimization attempts and a reference point for capacity planning.

Confirmation of the workaround's effectiveness. The --disable-radix-cache workaround is validated at 512 concurrency, giving the assistant confidence to push further. Without this confirmation, the assistant might have hesitated to attempt 1,024 concurrency.

Evidence of sub-linear scaling. The 1.52× increase from 256 to 512 concurrency (compared to the ideal 2×) signals that the system is approaching a bottleneck. This informs the assistant's subsequent investigation into GPU power utilization and allreduce fusion.

A benchmark methodology validated. The stair-step approach of 1 → 64 → 256 → 512 → 1024 concurrency is proven effective at revealing scaling behavior without crashing the server prematurely.

The Thinking Process Visible in the Message

Although the message is brief, the assistant's reasoning is laid bare in its structure. The opening line—"Good — 1,846 tok/s at 256 concurrency with peak 1,492 tok/s output! And the single chat request generated 500 tokens in 62.6s = ~8 tok/s."—serves as a status check. The assistant is summarizing the state of play before taking the next action. This is characteristic of a methodical debugger: always re-establish context before proceeding.

The phrase "The no-radix-cache version seems stable" is a deliberate hedge. The assistant is not declaring the server bug-free; it is saying that the specific failure mode (the NSA crash) has not manifested in the 256-concurrency test. The word "seems" acknowledges the possibility that the bug could reappear at higher concurrency.

"Let me push to 512 and 1024" reveals the assistant's plan. The assistant is not content with a single data point; it wants the full curve. The mention of both 512 and 1024 in the same sentence indicates that the assistant is thinking ahead, already planning the next experiment while executing the current one. This forward-looking mindset is essential in performance optimization, where each test informs the next.

The benchmark command itself is a model of precision. Every flag is explicit; nothing is left to defaults. The assistant has learned from earlier crashes that default values can be dangerous. The tail -25 at the end shows the assistant knows exactly which lines of output contain the metrics of interest—a sign of familiarity with the tool.

Conclusion

Message <msg id=713> is a deceptively simple artifact. On its surface, it is a benchmark result: 2,800 tok/s at 512 concurrency. But beneath that number lies a dense web of context: hours of debugging NSA crashes, patching FlashInfer kernels for SM120 support, working around PCIe topology limitations in a virtualized environment, and methodically sweeping concurrency levels to map the throughput curve. The message captures a moment of validation—the workaround holds, the scaling continues, and the next step (1,024 concurrency) beckons. It is a testament to the iterative, hypothesis-driven nature of performance engineering, where each data point is both a destination and a launching point for the next inquiry.