Mining Prior Art: How the Assistant Used Historical Benchmarks to Diagnose MoE Kernel Bottlenecks

Introduction

In the high-stakes world of large-scale ML inference deployment, every tok/s counts. When the assistant in this opencode session found itself staring at a disappointing ~880 tok/s throughput for the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs — with GPU power draw idling at a mere 83W out of a 600W TDP — it needed answers. Message 668 captures a pivotal investigative moment: the assistant turning to historical artifacts from a previous deployment of a similar model (Kimi K2-Thinking) on the exact same hardware, mining those logs for clues about how to unlock the GPUs' true potential.

This message is not about making changes or deploying code. It is a research probe — a deliberate, hypothesis-driven examination of prior work to inform the next set of performance optimizations. In the narrative of the session, it sits at the inflection point between "we have a working deployment with poor utilization" and "let's apply the lessons learned from previous successes."

The Performance Crisis That Drove the Investigation

To understand why message 668 exists, we must understand the crisis that preceded it. Throughout segment 6, the assistant had been iteratively improving GLM-5-NVFP4 inference throughput. The server was running with --moe-runner-backend flashinfer_cutlass, but the GPUs were barely breaking a sweat. At 256 concurrent requests, total token throughput plateaued at ~880 tok/s. At 512 concurrency, it inched up to only ~913 tok/s. The GPUs were sipping power at 82-85W each — a far cry from their 600W thermal design power.

The user had flagged this directly in message 657: "Note, still quite low power and lowish pcie bw, probably want to tune kernels." This observation crystallized the problem: the MoE (Mixture-of-Experts) kernels, which are the computational heart of models like GLM-5 with 256 routed experts, were not being properly optimized for the SM120 architecture of the RTX PRO 6000 Blackwell GPUs.

The assistant had already begun investigating the MoE code path. It had inspected the model configuration (E=256 experts, N=2048 intermediate size, top_k=8), explored the sglang source tree for MoE kernel config files, and even spawned a subagent task to trace the full kernel call chain for the flashinfer_cutlass backend with NVFP4 quantization. But something was missing: a concrete reference point for what good performance looked like on this hardware.

Why This Message Was Written: The Search for a Baseline

Message 668 is motivated by a specific, practical need: the assistant needed to understand what performance the previous Kimi K2-Thinking deployment achieved on the same eight RTX PRO 6000 GPUs, and critically, how it achieved it. The local research repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ contained logs, configuration files, and tuning artifacts from that prior work. The assistant had already explored the directory structure in messages 664-667, finding MoE tuning configs, benchmark logs, and server startup logs.

Now, in message 668, the assistant drills into the most relevant artifact: sglang-k2-thinking-final-bench.log. This log represents the culmination of the previous deployment effort — the "final benchmark" after all tuning had been applied. By examining it, the assistant hoped to answer several questions:

  1. What peak throughput did the K2-Thinking model achieve? This would set a target for the GLM-5 deployment.
  2. What server configuration was used? The server args would reveal flags like --max-running-requests, attention backend choices, and MoE runner backend selections.
  3. Were there FlashInfer autotune messages? This was the most critical question. The assistant suspected that FlashInfer's CUTLASS-based MoE kernels required explicit autotuning for the SM120 architecture, and that the previous successful deployment had solved this problem. The specific grep patterns chosen reveal the assistant's mental model of what matters for MoE performance: autotun|AutoTun for autotuning events, tactic for kernel selection decisions, fallback for when optimal kernels aren't available, flashinfer.*moe and cutlass.*moe for the specific backend, fused_moe for the Triton-based MoE path, warmup for kernel compilation events, and throughput|tok/s for performance numbers.

What the Commands Actually Revealed

The message contains two tool calls executed in parallel: a grep command and a read command. The grep searched for the patterns listed above across the first 40 matching lines of the log. The result was underwhelming: only line 6 matched, showing the server_args with the model path nvidia/Kimi-K2-Thinking-NVFP4 and a truncated set of configuration parameters. The absence of autotune, fallback, or throughput messages in the grep output was itself a significant finding — it suggested that either the log didn't contain explicit autotune messages, or the patterns didn't capture them.

The read command loaded lines 100-103 of the same log file, revealing server startup messages: a warning that "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0" — a known compatibility issue with Blackwell — followed by "Init torch distributed begin" messages from TP1 and TP6 (tensor parallelism ranks). These lines are from early in the server startup sequence, not from benchmark execution, confirming that the assistant was still digging through the log to find the actual performance data.

Assumptions Embedded in the Investigation

This message rests on several assumptions, some explicit and some implicit:

The relevance assumption: The assistant assumes that the Kimi K2-Thinking deployment is a valid analog for the GLM-5 deployment. Both are NVFP4-quantized MoE models running on the same eight RTX PRO 6000 GPUs with tensor parallelism. However, they have different architectures: K2-Thinking has hidden_size=7168 and intermediate_size=18432, while GLM-5 has hidden_size=6144 and intermediate_size=12288 (with moe_intermediate_size=2048). These differences could affect optimal kernel configurations.

The grep pattern assumption: The assistant assumes that the patterns it selected will capture the relevant signals. This is a reasonable engineering heuristic, but it could miss important information encoded in different log formats or message levels.

The log completeness assumption: The assistant assumes that the final benchmark log contains the tuning and performance information it needs. If the tuning was done in a separate session or the log was truncated, this assumption would fail.

The causality assumption: The assistant implicitly assumes that FlashInfer autotuning was a key enabler of the K2-Thinking performance. This hypothesis drives the entire investigation but may not be correct — the performance gains could have come from other optimizations like attention backend selection, CUDA graph compilation, or batch size tuning.

Input Knowledge Required to Understand This Message

A reader needs substantial context to grasp what is happening in message 668:

  1. The hardware platform: Eight NVIDIA RTX PRO 6000 Blackwell GPUs with SM120 architecture, each with 96GB VRAM and 600W TDP, connected via PCIe in a Proxmox virtualized environment where P2P DMA is limited.
  2. The model: GLM-5-NVFP4, a 256-expert Mixture-of-Experts model quantized to 4-bit floating point (NVFP4), with 78 layers and top_k=8 experts per token.
  3. The inference stack: SGLang server with flashinfer_cutlass MoE backend, tensor parallelism 8, running in an LXC container on Proxmox.
  4. The prior work: A repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ containing research artifacts from deploying Kimi K2 and K2-Thinking models on the same hardware, including tuned MoE configurations, server logs, and benchmark results.
  5. The performance problem: GPU utilization and power draw are far below theoretical maximum, suggesting kernel-level bottlenecks in the MoE computation path.
  6. The technical challenge: SM120 (consumer/workstation Blackwell) has different characteristics than SM100 (datacenter Blackwell), including smaller shared memory and missing kernel support for features like allreduce fusion that are optimized for datacenter GPUs.

Output Knowledge Created by This Message

While the message did not produce dramatic results — no breakthrough throughput numbers or discovered tuning parameters — it created valuable negative knowledge:

  1. The K2-Thinking log does not contain obvious FlashInfer autotune messages in the first 40 grep matches. This suggests that either autotuning was not explicitly logged, or it was not the primary mechanism for achieving performance.
  2. The server configuration for K2-Thinking is partially visible: the model path, tokenizer, and basic server args. This provides a starting point for comparing configurations.
  3. The DeepGemm scale format warning appears in both the K2-Thinking and GLM-5 logs, confirming it's a known and presumably tolerated issue for NVFP4 models on Blackwell.
  4. The log structure reveals that the benchmark log begins with server startup messages, not benchmark results — the assistant would need to read further into the file to find actual throughput numbers. This negative knowledge is valuable because it forces the assistant to broaden its search. If FlashInfer autotune messages aren't the key, perhaps the performance gains came from other mechanisms: different attention backends, CUDA graph optimization, batch size tuning, or the specific MoE configuration files found in the moe-configs/ directory.

The Thinking Process Visible in the Message

The assistant's reasoning is laid bare in the structure of this message. It begins with a clear statement of intent: "Let me see the final benchmark results from the K2-Thinking run to understand what performance they achieved and how, and also check the K2 log for any FlashInfer autotune messages." This is classic hypothesis-driven debugging: formulate a hypothesis (FlashInfer autotuning was the key), then seek evidence to confirm or refute it.

The choice of grep patterns reveals a sophisticated understanding of what signals matter in an inference server log. The patterns cover:

Mistakes and Incorrect Assumptions

The most significant issue with this message is that the grep command, as executed, may not have captured the information the assistant was looking for. The output shows only line 6 (the server args), which is a very long single line. The head -40 limit means we see only the first 40 lines of grep output, but the grep matched at least 40 lines (since we see line 6 and the output is truncated). However, the displayed output only shows one match line, suggesting that either:

  1. The grep output was extremely long (each match line could be hundreds of characters due to the verbose server args), and the head -40 cut off after showing only the first match.
  2. Or the grep matched fewer than 40 lines but the output was still truncated by the display. This is a limitation of the tool output display in the conversation — the full grep results may not be visible to the reader, which means the assistant itself might not have gotten the full picture from this single command. Additionally, the assistant may have been too narrow in its search. By focusing on FlashInfer autotune messages, it might miss other important signals in the log, such as memory allocation patterns, NCCL initialization messages, or kernel compilation timing information that could reveal bottlenecks.

Conclusion: The Value of Negative Results

Message 668 is a reminder that not every step in a debugging journey produces a breakthrough. The assistant's investigation of the K2-Thinking log did not immediately reveal the magic tuning parameters that would unlock 4x throughput. But it did provide important directional information: the log doesn't contain obvious autotune messages, the DeepGemm warning is a known issue, and the server configuration is similar to the current GLM-5 setup.

This negative result forces a pivot. If the K2-Thinking performance wasn't achieved through FlashInfer autotuning, then the assistant must look elsewhere — perhaps at the MoE configuration files in moe-configs/, or at different server flags like --max-running-requests and --disable-cuda-graph. And indeed, in the messages that follow (within the same chunk), the assistant does pivot: it patches model_runner.py to enable FlashInfer CUTLASS autotune for SM120, raises --max-running-requests from 64 to 1024, and achieves a dramatic jump from ~880 tok/s to ~3,740 tok/s. The investigation in message 668 was the necessary precursor — the reconnaissance that confirmed the problem space and ruled out one hypothesis, clearing the path for the actual solution.