The Pivot to Prior Art: How a Directory Listing Unlocked MoE Kernel Tuning for GLM-5 on Blackwell

Introduction

In the course of a complex machine learning deployment session, some messages appear deceptively simple. Message 664 in this conversation is one such moment: an assistant reading the directory listing of a local repository. On its surface, it is a single read tool call returning eleven directory entries. But in context, this message represents a critical turning point — a shift from abstract code investigation to leveraging concrete prior empirical knowledge. It is the moment when the assistant, having spent several rounds deep in the source code of SGLang's MoE kernel dispatch, heeds the user's guidance to consult a repository of hard-won lessons from a similar deployment on identical hardware.

Context: The Performance Plateau

To understand why this message matters, we must trace the events leading up to it. The session had been focused on deploying the GLM-5-NVFP4 model — a massive Mixture-of-Experts (MoE) language model with 256 routed experts — across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had achieved a working server and was running benchmarks, but performance was plateauing. At 256 concurrent requests, total token throughput was ~880 tok/s. At 512 concurrency, it barely budged to ~913 tok/s. GPU power draw sat at a mere ~83W per GPU — a fraction of the 600W thermal design power (TDP) of each RTX PRO 6000 Blackwell card.

The user had noted this directly in message 657: "Note, still quite low power and lowish pcie bw, probably want to tune kernels." This observation crystallized the problem: the GPUs were severely underutilized. The bottleneck was not raw compute capacity but the efficiency of the MoE kernel implementations — the specialized CUDA and Triton kernels that route tokens through the model's 256 experts and compute their weighted outputs.

In response, the assistant launched a deep investigation into the MoE kernel code path (message 662), spawning a subagent task to trace the exact call chain for the flashinfer_cutlass backend with NVFP4 quantization. The task returned a comprehensive analysis of how ModelOptNvFp4FusedMoEMethod.apply() dispatches to CUTLASS-based fused MoE kernels. But this analysis, while thorough, was architectural — it described how the code worked, not what parameters would make it perform well on SM120 hardware.

The User's Intervention

Then came message 663: "Read info in ./ from a similar run which did tuning on a similar model." This was a pivotal intervention. The user was pointing the assistant to a local repository — /home/theuser/glm-kimi-sm120-rtx6000bw — that contained research artifacts from a prior deployment of Kimi K2.5 and K2-Thinking on the exact same hardware: 8x NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. The Kimi K2 model family, like GLM-5, uses a MoE architecture with a large number of experts. The user knew that this prior work had already tackled the MoE kernel tuning problem and achieved significant performance gains.

This is a common pattern in complex engineering work: the most valuable knowledge is often not in documentation or source code but in the logs, configs, and notes from previous attempts. The user was effectively saying, "Stop trying to reason from first principles about the kernel code path — go look at what actually worked last time."

The Subject Message: Reading the Repository

Message 664 is the assistant's response to this directive. It reads the directory listing of the prior research repository:

[assistant] 

Let me look at what's in the local repo from the prior Kimi K2 run, especially the MoE tuning artifacts.
[read] /home/theuser/glm-kimi-sm120-rtx6000bw
<path>/home/theuser/glm-kimi-sm120-rtx6000bw</path>
<type>directory</type>
<entries>
.git/
configs/
FINDINGS.md
logs/
moe-configs/
patches/
README.md
report-kimi-k2-sm120.html
report-kimi-k2-sm120.pdf
sm120-attention-fix-research.md
source/

(11 entries)
</entries>

The assistant's own comment — "especially the MoE tuning artifacts" — reveals its intent. It immediately recognizes the value in the moe-configs/ directory, the logs/ directory (containing files like moe-tuning.log), and the FINDINGS.md document. The repository is well-organized, with clear separation of concerns: configuration files for different tensor parallelism configurations (tp4_config.json, tp8_config.json), source patches (source/), and benchmark reports (report-kimi-k2-sm120.html).

What This Message Reveals About the Assistant's Reasoning

Several aspects of the assistant's thinking are visible in this brief message:

First, the assistant correctly interprets the user's intent. The user said "Read info in ./ from a similar run" — a somewhat ambiguous instruction. The assistant immediately understands that "info" means the repository structure and "similar run" refers to the Kimi K2 deployment. It frames its action as looking for "MoE tuning artifacts," showing it has already connected the user's hint to the performance bottleneck it was investigating.

Second, the assistant prioritizes the most relevant subdirectories. It doesn't just dump the listing and move on — it explicitly calls out "especially the MoE tuning artifacts," signaling to the reader (and to its own subsequent actions) that moe-configs/, logs/, and FINDINGS.md are the high-value targets. This prioritization is crucial because the repository contains 11 entries, and not all are equally relevant to the immediate problem of MoE kernel performance.

Third, the assistant is operating under a key assumption: that the Kimi K2 tuning approach is transferable to GLM-5. This is a reasonable assumption — both models use MoE architectures, both run on SM120 hardware, and both use NVFP4 quantization. But it is still an assumption. The models have different hidden sizes (7168 for Kimi K2.5 vs 6144 for GLM-5), different intermediate sizes (18432 vs 12288), and different expert counts. The tuning parameters that worked for Kimi K2 may need adjustment for GLM-5's different dimensions.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. The session's history: That the assistant had been struggling with MoE kernel performance, that benchmarks showed GPU underutilization, and that the user had previously suggested kernel tuning.
  2. The hardware context: That the target hardware is 8x NVIDIA RTX PRO 6000 Blackwell GPUs with SM120 architecture, and that these are consumer/workstation cards with different characteristics than datacenter Blackwell (SM100) GPUs.
  3. The model architecture: That GLM-5-NVFP4 is a MoE model with 256 experts, top-8 routing, and NVFP4 (4-bit floating point) quantization using the modelopt format.
  4. The prior work: That a similar deployment was done for Kimi K2 models on the same hardware, and that this work likely involved MoE kernel tuning to overcome the same SM120-specific challenges.
  5. The repository structure conventions: That configs/ likely contains hardware and model configs, logs/ contains benchmark and server logs, moe-configs/ contains tuned kernel configuration files, patches/ contains source code modifications, and FINDINGS.md is a running document of discoveries.

Output Knowledge Created

This message produces several forms of knowledge:

Immediate knowledge: The directory structure of the prior research repository. This is a map of where to find relevant artifacts — the tuned MoE configs, the benchmark logs showing what performance was achieved, the patches that were applied, and the summary document explaining what was learned.

Contextual knowledge: The assistant's intent to focus on MoE tuning artifacts. This frames how subsequent reads will be interpreted and prioritized.

Meta-knowledge: The existence of a prior successful tuning effort. Before this message, the assistant was operating in a vacuum — it knew the code paths but not what performance was achievable. Now it knows that someone else has already solved a similar problem on identical hardware, which sets an expectation that significant gains are possible.

The Follow-Through

The assistant does not stop at this directory listing. In the very next message (message 665), it immediately drills into the subdirectories: reading the contents of moe-configs/ (finding five config files including tp8_config.json), source/ (finding patches for attention and MXFP4), configs/ (finding hardware info and model configs), and logs/ (finding 12 log files including moe-tuning.log). It then reads the actual content of key files: the README.md for context, the moe-tuning.log for the tuning process, the tp8_config.json for the actual tuned parameters, and the Kimi K2.5 model config for comparison.

This rapid consumption of prior work is only possible because message 664 established the map. The directory listing is the table of contents; the subsequent reads are the chapters.

Why This Message Matters

In the broader narrative of the session, message 664 is the hinge point. Before it, the assistant was deep in code analysis — tracing call chains, reading Python source, understanding dispatch mechanisms. After it, the assistant shifts to empirical tuning — applying the lessons from Kimi K2 to GLM-5, running the MoE autotuner, and ultimately achieving a dramatic throughput improvement from ~880 tok/s to ~3,740 tok/s.

The message also illustrates a fundamental truth about AI-assisted engineering: the most valuable resource is often not the AI's ability to reason from scratch but its ability to rapidly ingest and apply prior work. The assistant could have spent hours reverse-engineering the MoE kernel parameters from first principles. Instead, the user's nudge and this single directory listing pointed it to a repository of pre-existing solutions, saving orders of magnitude in time.

Conclusion

Message 664 is a testament to the power of context and prior knowledge in complex engineering workflows. A simple directory listing — eleven entries in a local repository — became the catalyst that transformed the session from abstract code investigation into targeted, empirically-driven optimization. The assistant's ability to recognize the relevance of this prior work, prioritize the most valuable artifacts, and pivot its approach accordingly is precisely the kind of contextual reasoning that makes AI assistants powerful partners in technical work. The message itself may be brief, but its impact on the trajectory of the session was profound.