The Moment of Truth: Benchmarking a Patched Inference Stack

Introduction

In the high-stakes world of large language model inference optimization, every change to the serving stack is a gamble. When an engineer patches a core component of the runtime — enabling an autotuner that was explicitly commented out as problematic — the first benchmark run carries enormous weight. Message [msg 693] captures exactly this moment: the first test of a newly patched SGLang server after the assistant enabled FlashInfer CUTLASS MoE autotune for SM120 (Blackwell consumer GPU architecture), raised the maximum running requests limit, and disabled CUDA graphs. The result, 689 total tokens per second at 64 concurrent requests, was underwhelming — slightly worse than the 757 tok/s achieved before the changes. This message is a masterclass in the scientific method applied to systems optimization: form a hypothesis, make a change, measure the result, and iterate.

Context: The Optimization Journey

To understand why this message was written, we must understand the path that led to it. The assistant had been engaged in an intensive multi-session effort to deploy the GLM-5-NVFP4 model — a 405-billion-parameter Mixture-of-Experts (MoE) model quantized to NVFP4 — across eight NVIDIA RTX PRO 6000 Blackwell GPUs. Earlier segments had resolved NaN crashes during decode by selecting compatible NSA backends ([msg 667]), established baseline throughput, and diagnosed virtualization-induced PCIe P2P latency as a key bottleneck. The most recent segment (Segment 6) had identified that throughput was plateauing at around 880 tok/s, and the assistant had been systematically investigating why.

A critical breakthrough came when the assistant examined the logs from a related model deployment — the Kimi K2-Thinking model — which had achieved an astonishing 5,816 tok/s peak throughput on similar hardware ([msg 670]). Comparing configurations revealed stark differences: the K2 run used --disable-cuda-graph, had max_running_requests=2048 (effectively uncapped), and crucially, ran with moe_runner_backend='auto' which enabled FlashInfer CUTLASS MoE autotune. The GLM-5 deployment, by contrast, was capped at --max-running-requests 64, had CUDA graphs enabled, and was running with the flashinfer_trtllm MoE backend.

The Patch: Enabling FlashInfer CUTLASS Autotune

The most technically significant change was patching the SGLang model runner to enable flashinfer_cutlass in the autotune list. The code at line 1837 of model_runner.py contained an explicit TODO comment: "flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed." ([msg 671]). The line was commented out. The assistant uncommented it with a targeted sed command ([msg 677]):

sed -i '1838s|# "flashinfer_cutlass",|"flashinfer_cutlass",|'

This was a deliberate, calculated risk. The upstream developers had identified that enabling this backend caused compilation errors, but the assistant reasoned that the K2-Thinking run's success implied the autotune was valuable enough to attempt. The assumption was that the compilation errors might be intermittent, architecture-specific, or already resolved in the installed version of FlashInfer. This assumption would prove partially correct — the autotune did run successfully across all 8 TP ranks ([msg 690]), but its effect on throughput was not immediately positive.

The Restart: A New Configuration

Alongside the patch, the assistant made two other significant changes when restarting the server ([msg 682]):

  1. Removed --max-running-requests 64: This parameter had been artificially capping the number of concurrent requests the server would accept. Without it, the server auto-detected a limit of 2,048 — a 32× increase. The assistant's reasoning, drawn from the K2-Thinking comparison, was that higher concurrency was essential for saturating the GPUs and achieving peak throughput.
  2. Added --disable-cuda-graph: CUDA graphs capture a sequence of GPU operations for efficient replay, but they can also limit flexibility in batch size and sequence length. The K2 run had used this flag, and the assistant hypothesized that the overhead of graph recompilation at varying batch sizes was hurting throughput. The server came up successfully with FlashInfer autotune completing across all 8 tensor-parallel ranks, KV cache allocated at 495,488 tokens, and max_running_requests auto-detected at 2,048 ([msg 692]). The stage was set for the first benchmark.

The Benchmark: Message [msg 693] in Detail

The subject message executes a benchmark using SGLang's built-in bench_serving script with the following parameters:

The Thinking Process: What the Assistant Learned

The assistant's reasoning is visible in the immediate follow-up message ([msg 694]): "689 tok/s at 64 concurrency — slightly worse than before (757 tok/s). This is likely because --disable-cuda-graph hurts at low batch sizes. Let me test at higher concurrency where the real benefit should show."

This reveals a sophisticated mental model of the system's behavior. The assistant understood that:

  1. CUDA graphs are a double-edged sword: They improve performance for stable, predictable workloads but can cause overhead when batch sizes vary. At low concurrency, the graph overhead is minimal, so disabling them only removes their benefit without avoiding their cost.
  2. The autotune's value scales with concurrency: The FlashInfer CUTLASS autotune selects optimal GEMM tactics for the MoE kernels. These tactics matter most when many expert computations are running simultaneously — i.e., at high concurrency. At 64 concurrent requests, the GPU is not fully saturated, so the autotune has limited impact.
  3. The real test is at higher concurrency: The assistant immediately pivoted to testing at 256 concurrency ([msg 694]), where the combined effect of higher max_running_requests and the autotune would be visible. This test achieved 1,950 tok/s — more than double the previous baseline.

Assumptions and Their Validity

The assistant operated under several assumptions, some of which proved incorrect:

Assumption 1: Enabling flashinfer_cutlass autotune would improve throughput. This was based on the K2-Thinking run's success. However, the K2 run used a different model (Kimi K2) with potentially different MoE routing patterns and expert sizes. The autotune's effectiveness depends on the specific GEMM shapes in the model. The assumption was partially validated — the autotune ran successfully, but its benefit at low concurrency was negligible.

Assumption 2: Disabling CUDA graphs would improve throughput. This was incorrect at low concurrency. The assistant correctly identified this and adjusted the hypothesis, noting that CUDA graphs hurt at low batch sizes. Later in the chunk, the assistant would restart with CUDA graphs re-enabled ([msg 698]), demonstrating adaptive decision-making.

Assumption 3: The server could handle 2,048 concurrent requests. This was tested at 256 concurrency (success), but when pushed to 512 concurrency ([msg 695]), the server crashed with a PrefillMetadata attribute error ([msg 696]). The crash was not an OOM but a code bug in the NSA attention path when handling large batches without CUDA graphs. This revealed that the software stack had latent bugs that only manifested at scale.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the previous baseline: The assistant had established that the GLM-5 model achieved ~757 tok/s at 64 concurrency with the old configuration. This is the reference point for comparison.
  2. Understanding of the changes made: The patch to model_runner.py (enabling flashinfer_cutlass), the removal of --max-running-requests, and the addition of --disable-cuda-graph are all prerequisite context.
  3. Familiarity with SGLang's architecture: The distinction between MoE backends (flashinfer_trtllm vs flashinfer_cutlass), the role of CUDA graphs, and the autotune mechanism are all SGLang-specific concepts.
  4. Knowledge of the hardware: The 8× RTX PRO 6000 Blackwell GPUs, their SM120 architecture, and the PCIe P2P limitations from the Proxmox VM setup all inform why certain optimizations matter.

Output Knowledge Created

This message produced several forms of knowledge:

  1. A new data point: 689 tok/s at 64 concurrency with the new configuration, establishing that --disable-cuda-graph hurts at low batch sizes.
  2. Validation of the autotune's stability: The server ran successfully with the patched autotune, confirming that the compilation errors mentioned in the TODO comment did not manifest in this environment.
  3. A refined hypothesis: The assistant now knows that the benefits of the new configuration will appear at higher concurrency, motivating the immediate follow-up test at 256 concurrency.
  4. A benchmark methodology: The choice of 64 prompts (matching the old cap) as a controlled comparison demonstrates good experimental design — isolate one variable at a time.

Conclusion

Message [msg 693] is a pivotal moment in the optimization journey. It represents the first empirical test of a bold set of changes — patching a core component that upstream developers had flagged as problematic, removing an artificial concurrency cap, and disabling a performance optimization that had been assumed beneficial. The result was underwhelming, but the assistant's response was not disappointment — it was a refined hypothesis and an immediate next step. This is the essence of systems optimization: each measurement, even a negative one, narrows the search space and guides the next iteration. The real payoff came moments later at 256 concurrency, where throughput more than doubled to 1,950 tok/s, validating the assistant's intuition that the changes would unlock performance at scale.