Reading the Signal Through the Noise: Benchmarking GLM-5-NVFP4 After the NaN Crisis

In the aftermath of a grueling debugging session that had consumed hours of effort — chasing NaN crashes during decode, swapping attention backends, and finally discovering that --nsa-decode-backend trtllm was the magic incantation that made the GLM-5-NVFP4 model produce coherent output on SM120 Blackwell GPUs — the assistant in message 224 faces a quieter but equally important challenge: measuring whether the fix actually works under load. This message is the hinge point between "the model doesn't crash" and "the model performs acceptably," and it reveals a great deal about how the assistant thinks about measurement, tooling limitations, and what constitutes actionable information.

The Context: From Crash to Coherence

To understand message 224, one must appreciate what came immediately before it. The assistant had spent multiple rounds fighting a persistent NaN crash that occurred whenever the model attempted to generate tokens beyond the initial prefill. The root cause was architectural: the GLM-5-NVFP4 model uses a Native Sparse Attention (NSA) mechanism, and on the SM120 GPUs (NVIDIA RTX PRO 6000 Blackwell), several of the available NSA backends — flashmla_kv, flashmla_sparse — produced floating-point garbage. Only the trtllm backend worked, as confirmed by successful inference tests in messages 215–218 where the model correctly answered "What is 2+2?" with "4" and wrote a proper is_prime Python function.

With the crash resolved, the assistant's todo list (visible in message 218) shifted from debugging to performance tuning: "Tune sglang params for parallel query throughput" and "Run load tests with sglang.bench_serving." Messages 219–223 represent the first stumbling steps into benchmarking territory — checking GPU memory (80GB used out of 97GB per GPU), discovering that the bench_serving module exists, and then hitting a series of tooling failures.

The First Benchmark Attempts: A Study in Tooling Friction

Message 224 does not exist in isolation. It is the fourth attempt to run a meaningful benchmark, and the assistant's opening sentence — "Good, the sglang native backend works" — carries the weight of those preceding failures.

The first attempt (message 221) used --backend sglang-oai-chat and crashed immediately with a 401 Unauthorized error because the tool tried to fetch a tokenizer config from Hugging Face using the served model name glm-5 rather than the full repository path lukealonso/GLM-5-NVFP4. The second attempt (message 222) corrected the tokenizer parameter but crashed during metrics calculation: the reasoning model returns content: null with reasoning_content instead, and the benchmark script's tokenizer call choked on the None value. The third attempt (message 223) switched to --backend sglang (the native SGLang protocol rather than OpenAI-compatible chat) and finally completed — but produced metrics that made no physical sense.

This is where message 224 begins, with the assistant staring at a benchmark output that reports negative TPOT (time per output token) and 0ms ITL (inter-token latency). The assistant immediately diagnoses the problem: the native /generate endpoint returns results as a single non-streaming response without per-token timing information. The benchmark tool, designed for streaming endpoints that emit timing data with each token, has no way to compute per-token metrics from a bulk response and falls back to nonsensical values.

The Reasoning in Message 224: What Matters and What Doesn't

The critical thinking in this message is the assistant's decision about what to do with the broken metrics. Rather than getting stuck on the fact that TTFT (time to first token) and TPOT are garbage, the assistant identifies the one reliable number in the output: output token throughput: 50.55 tok/s. This is a coarse but trustworthy aggregate metric — total output tokens generated divided by total wall-clock time — and it does not depend on per-token instrumentation.

The assistant then makes two judgments about this number:

  1. "This is low" — 50 tokens per second across 8 GPUs for a 744-billion-parameter model is indeed underwhelming. For context, a single H100 might deliver several hundred tokens per second on a model of this scale. The assistant knows the hardware is capable of more.
  2. "but expected with --disable-cuda-graph" — This is the key explanatory hypothesis. CUDA graphs are a GPU optimization that pre-compiles sequences of kernel launches into a single graph, reducing launch overhead and improving utilization. They were disabled in the current server configuration because earlier attempts with different NSA backends had crashed when CUDA graphs were enabled. The assistant implicitly assumes that CUDA graphs are the primary bottleneck and that enabling them later (or tuning other parameters) will substantially improve throughput. This second judgment is an assumption — and a reasonable one — but it is not yet tested. The assistant is essentially saying "we know why it's slow, and the fix is on our roadmap." This prevents premature optimization: rather than tweaking random parameters based on a single data point, the assistant has a theory about the bottleneck and plans to address it systematically.

The Decision to Scale Up

The most concrete action in message 224 is the decision to run a larger benchmark: 32 prompts (up from 16) with a request rate of 8 (up from 2), and longer output sequences (256 tokens instead of 128). The reasoning is twofold:

First, statistical reliability. A benchmark with 16 prompts at rate 2 may not saturate the server or reveal stability issues under concurrent load. Doubling both dimensions increases the chance of exposing problems — memory pressure, request queueing, or throughput collapse under contention.

Second, representative workload. The assistant notes that the current test uses --random-output-len 256, which better matches real usage where the model generates substantial reasoning chains before producing an answer. The earlier test with 128-token outputs might have been too short to measure steady-state decode performance.

The assistant also checks the server log (implied by "check what the server log shows") to correlate client-side metrics with server-side observations. This dual-perspective approach — measuring both from the client benchmark tool and from the server's internal logging — is a hallmark of thorough performance analysis. The server log reports decode batch throughput in tokens per second per GPU, which can reveal whether the bottleneck is on the client side (network latency, request scheduling) or the server side (actual GPU computation).

Assumptions Embedded in the Message

Several assumptions are worth examining:

  1. The native backend metrics are fundamentally unreliable for per-token analysis. This is correct — the non-streaming endpoint simply cannot provide token-level timing. But the assistant does not consider whether the streaming endpoint (sglang-native with streaming enabled) might work. The choice to abandon per-token metrics and focus on aggregate throughput is pragmatic but leaves a gap: without TPOT and TTFT, the assistant cannot distinguish between prefill-bound and decode-bound performance.
  2. CUDA graphs are the primary performance lever. This is an informed guess based on prior experience with SGLang and GPU inference, but it may not hold. The actual bottleneck could be PCIe bandwidth between GPUs (since these are Blackwell GPUs in a virtualized environment without NVLink), memory bandwidth within each GPU, or the NSA attention implementation itself. The assistant will discover later (in subsequent messages) that virtualization-induced PCIe latency is indeed a major constraint.
  3. The benchmark tool's aggregate throughput numbers are trustworthy. The assistant trusts that bench_serving correctly counts output tokens and measures wall-clock time even when per-token metrics fail. This is a reasonable assumption for a mature tool in the SGLang ecosystem, but it is worth noting that the tool was not designed for the native backend's non-streaming response format, and other edge cases may lurk.
  4. Throughput scales with concurrency. The assistant expects that increasing the request rate from 2 to 8 will increase throughput, not degrade it. This assumes the server has headroom — that it is not already saturated at 16 concurrent requests. The results from message 225 (144 output tok/s at rate 8) confirm this assumption was correct.

Input Knowledge Required

To fully understand message 224, a reader needs:

Output Knowledge Created

Message 224 produces several pieces of actionable knowledge:

  1. A confirmed working benchmark methodology: the sglang native backend with --disable-stream produces valid aggregate throughput numbers, even if per-token metrics are broken.
  2. A baseline throughput figure: ~50 tok/s output throughput for 16 prompts at rate 2 with CUDA graphs disabled.
  3. A hypothesis about the bottleneck: CUDA graph absence is the likely cause of low throughput.
  4. A scaling plan: increase concurrency to 32 prompts at rate 8 to stress-test the server and get more representative numbers.
  5. A diagnostic procedure: cross-reference client-side benchmark results with server-side log output to identify where time is actually spent.

The Broader Significance

Message 124 is, on its face, a mundane transitional message — the assistant runs one benchmark, interprets the results, and queues up a bigger one. But it exemplifies a pattern that recurs throughout the session: the assistant encounters tooling that was designed for conventional transformer models and discovers that reasoning models (which return content: null and reasoning_content instead of a unified response) break assumptions throughout the stack. The bench_serving crash in message 222, the broken per-token metrics in message 223, and the workaround in message 224 are all symptoms of the same underlying gap: the inference ecosystem is still catching up with the architectural peculiarities of models like GLM-5 that separate reasoning traces from final answers.

The assistant's response to this gap is instructive. Rather than trying to fix the benchmark tool (which would be a major engineering detour), it adapts its measurement strategy to work within the tool's limitations. It identifies the one metric that survives the broken pipeline — aggregate throughput — and uses that as its north star. This is a pragmatic tradeoff: coarse but reliable data is more useful than fine-grained but broken data. The assistant implicitly acknowledges that the goal is not perfect instrumentation but actionable insight — enough information to decide whether the model is fast enough and, if not, where to focus tuning efforts.

This message also marks the transition from "does it work?" to "how well does it work?" — a shift that is surprisingly hard in ML engineering. Getting a model to produce correct output is often celebrated as the finish line, but in production deployments, it is merely the starting point. The assistant's methodical approach to benchmarking — starting small, validating the toolchain, identifying measurement gaps, and scaling up — reflects an understanding that performance engineering is a discipline unto itself, separate from debugging correctness.

Conclusion

Message 224 is a quiet but pivotal moment in the conversation. It captures the moment when the assistant stops fighting crashes and starts measuring performance — a transition that requires a different kind of thinking. The assistant must interpret ambiguous data, distinguish signal from noise, make reasonable assumptions about bottlenecks, and design experiments that will produce clearer answers. The decision to trust aggregate throughput over broken per-token metrics, the hypothesis about CUDA graphs, and the scaling plan for the next benchmark all demonstrate a methodical approach to performance analysis. And the broader pattern — of reasoning models breaking standard tooling, and the assistant adapting around those breaks — foreshadows the deeper investigation into virtualization overhead and PCIe latency that will dominate the subsequent conversation.