The Benchmark After the Workaround: Validating Stability at 1,846 tok/s

Introduction

In the long-running effort to deploy the GLM-5-NVFP4 mixture-of-experts model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 712] represents a moment of cautious validation. After discovering and working around a crash-causing bug in the NSA (Native Sparse Attention) attention path, the assistant restarts the SGLang server with a critical configuration change—--disable-radix-cache—and immediately benchmarks to confirm that the workaround hasn't destroyed performance. The message contains two parallel tool calls: a 256-concurrency throughput benchmark and a single-request latency test. The results are encouraging (1,846 tok/s total throughput) but also reveal a persistent latency problem (62.6 seconds for 500 output tokens) that hints at deeper inefficiencies yet to be resolved.

The Context: A Bug in the NSA Attention Path

To understand [msg 712], we must look at the events immediately preceding it. In [msg 705] and [msg 706], the assistant discovered that the server was crashing during high-concurrency benchmarks with the error AttributeError: 'PrefillMetadata' object has no attribute 'page_table_1_flattened'. This crash occurred in the NSA attention code path, specifically in the FP8 KV cache dequantization logic within forward_mha.py. The assistant traced the issue to a conditional code path: page_table_1_flattened is only populated when prefix sharing is active (i.e., when the radix cache detects shared prefixes across requests), but the dequantization code at line 443 of forward_mha.py unconditionally accesses this attribute. When a batch of requests arrived that triggered FP8 dequantization (because the KV cache uses float8_e4m3fn dtype) without any prefix sharing, the attribute was None, and the server crashed.

This is a classic software bug—a missing guard clause for an edge case that only manifests under specific runtime conditions. The assistant correctly identified the root cause by examining the NSA backend code in nsa_backend.py (see [msg 708]), where the comment # page_table_1_flattened is only used when prefix sharing is enabled confirmed the conditional nature of the attribute. Rather than attempting a deep fix in the NSA attention code—which would require understanding the full interaction between the radix cache, prefix sharing, and FP8 dequantization—the assistant chose a pragmatic workaround: disable the radix cache entirely with --disable-radix-cache. This eliminates the prefix-sharing path and thus avoids the bug, at the cost of losing the performance benefits of cached prefix computation.

Message 712: The Validation Run

Message [msg 712] opens with the assistant's observation: "Good, faster startup since autotune cache is warm." This is a notable detail—the FlashInfer CUTLASS MoE autotune, which had been enabled in [msg 700] by patching model_runner.py to include flashinfer_cutlass in the autotune list, had completed its profiling run in the previous server instance. The autotune results are cached on disk (in /root/.cache/flashinfer/), so the new server instance skips the expensive profiling phase and starts much faster. This is a small but meaningful win, demonstrating that the autotune infrastructure works correctly and persists across restarts.

The assistant then launches two parallel benchmarks. The first is a 256-concurrency throughput test using sglang.bench_serving with 256 random prompts (256 input tokens, 128 output tokens each). The results are:

• Total token throughput: 1,846.23 tok/s
• Output token throughput: 615.45 tok/s
• Peak output token throughput: 1,492.00 tok/s
• Peak concurrent requests: 256
• Mean E2E latency: 17,489.56 ms (~17.5 seconds) This represents a slight regression from the 1,950 tok/s achieved in [msg 695] before the crash, which is expected: disabling the radix cache means the server can no longer reuse KV cache entries across requests with shared prefixes, so each request must compute its full KV cache from scratch. The ~5% drop is a reasonable price to pay for stability. The second benchmark is a single-request test: asking the model to "Count from 1 to 100" with max_tokens=500. The result is sobering: 500 completion tokens in 62.6 seconds, which works out to approximately 8 tok/s. This is dramatically slower than the batch throughput numbers and reveals a fundamental issue: the server is highly optimized for batch throughput but performs poorly on individual requests. The 62.6-second latency for a single request suggests that either the model is compute-bound even for a single sequence, or there's overhead in the serving stack (scheduling, attention computation, MoE routing) that disproportionately affects single-stream performance.

Decisions and Assumptions

The message reveals several implicit decisions and assumptions:

Decision to accept the radix cache trade-off: The assistant implicitly decides that the ~5% throughput regression from disabling the radix cache is acceptable given that it eliminates the crash. This is a pragmatic engineering judgment—stability trumps peak performance.

Assumption that the autotune cache is the sole reason for faster startup: The assistant attributes the faster server startup entirely to the warm autotune cache. While this is likely correct, there could be other factors—the OS may have cached model weight files, or the GPU driver may have retained some state from the previous run. The assistant doesn't investigate further, accepting the surface-level explanation.

Assumption that 256 concurrency is the right stress level: The assistant chooses 256 prompts for the benchmark, which is the same level that succeeded in [msg 695]. This is a conservative choice—the assistant could have pushed to 512 or 1024 concurrency (which were attempted earlier and caused crashes), but the priority is validating stability first. The "full benchmark sweep" mentioned in the message text is deferred.

Assumption that the single-request test is meaningful: The assistant includes a single completion request alongside the batch benchmark. This is a good practice—batch throughput numbers can hide latency pathologies—but the assistant doesn't analyze the 62.6-second result in this message. The stark contrast between 1,846 tok/s batch throughput and ~8 tok/s single-stream performance is left as an observation for future investigation.

Input Knowledge Required

To fully understand this message, the reader needs:

1. Knowledge of the NSA bug: The page_table_1_flattened crash and its relationship to prefix sharing and radix cache. Without this context, the --disable-radix-cache flag seems arbitrary.
2. Understanding of FlashInfer autotune: The concept of CUDA kernel autotuning, where multiple kernel implementations are benchmarked at runtime to select the fastest for the specific GPU architecture. The assistant's comment about "autotune cache is warm" assumes the reader knows that autotune results are persisted.
3. Familiarity with SGLang's serving architecture: The distinction between --disable-cuda-graph and --disable-radix-cache, the role of the radix cache in prefix sharing, and the interaction between KV cache management and attention backends.
4. Benchmark methodology: Understanding that bench_serving with --request-rate inf sends all requests as fast as possible, measuring peak throughput under saturation, while the single-request test measures raw latency.

Output Knowledge Created

This message produces several important data points:

1. Stability confirmation: The server with --disable-radix-cache survives a 256-concurrency benchmark without crashing. This validates the workaround.
2. Throughput baseline with radix cache disabled: 1,846 tok/s total, 615 tok/s output at 256 concurrency. This becomes the new baseline for comparison with future optimizations.
3. Single-stream latency data: 62.6 seconds for 500 tokens, or ~8 tok/s. This is a critical finding that will likely drive future optimization work—the server is heavily optimized for batch throughput but has poor tail latency.
4. Autotune persistence confirmed: The faster startup time confirms that FlashInfer autotune results are correctly cached and reused across server restarts, which is important for operational reliability.

The Thinking Process

The assistant's reasoning in this message is visible in the sequence of actions:

1. Observation: "Good, faster startup since autotune cache is warm." This shows the assistant is actively monitoring server behavior and forming hypotheses about what's working.
2. Prioritization: The assistant runs the benchmark before the single-request test, even though both are dispatched in parallel. The benchmark is the priority—it validates the stability fix. The single-request test is supplementary.
3. Cautious benchmarking: The assistant uses 256 concurrency (not 512 or 1024), despite having --max-running-requests 1024 configured. This shows restraint—the assistant wants to confirm stability at a known-safe level before pushing harder.
4. Missing analysis: Notably, the assistant does not comment on the 62.6-second single-request latency in this message. In subsequent messages, this would likely be investigated, but here it's simply recorded. This suggests the assistant is focused on the throughput validation and hasn't yet processed the latency implications.

Broader Significance

Message [msg 712] sits at a transition point in the optimization journey. The assistant has solved the immediate stability crisis (the NSA crash) and confirmed that the workaround preserves most of the throughput gains from the FlashInfer autotune and increased max-running-requests. But the single-request latency of 62.6 seconds for 500 tokens is a flashing warning light. It suggests that the server, while capable of impressive batch throughput, is fundamentally inefficient for interactive use cases. The 8 tok/s single-stream rate is far below what would be acceptable for a production serving system, and it points to deeper issues—perhaps in the MoE kernel scheduling, the attention backend overhead, or the NCCL communication patterns across the 8 GPUs.

The message also illustrates a recurring pattern in this optimization effort: the assistant repeatedly hits crashes and bugs in the SGLang/FlashInfer stack (NaN decode crashes in segment 1, the NSA page_table_1_flattened bug here, the allreduce fusion failure in chunk 0), and each time applies a pragmatic workaround rather than a deep fix. This is a rational strategy given the complexity of the codebase—the assistant is not the maintainer of SGLang or FlashInfer and cannot invest the weeks needed to fully understand and fix every bug. Instead, the approach is to find the combination of flags and configuration options that works, document it, and move on. The cost is that some performance is left on the table, and the system becomes increasingly reliant on a fragile set of non-default flags.

Conclusion

Message [msg 712] is a brief but pivotal moment in the GLM-5-NVFP4 deployment story. It validates a stability workaround, confirms that the FlashInfer autotune infrastructure is working correctly, and produces a new throughput baseline of 1,846 tok/s at 256 concurrency. But it also surfaces a troubling latency pathology—62.6 seconds for a single 500-token completion—that will need to be addressed before this system can be considered production-ready. The message captures the assistant at a moment of cautious optimism, having solved one problem and discovered another, pressing forward with the relentless benchmarking cadence that defines this optimization effort.