A Pivot to Prior Art: The User's Terse Directive to Leverage Existing MoE Tuning Artifacts

Subject message (msg 663): "Read info in ./ from a similar run which did tuning on a similar model"

In the middle of an intense optimization session for the GLM-5-NVFP4 model running on 8x NVIDIA RTX PRO 6000 Blackwell GPUs, the user interjects with a message that is deceptively simple. Just eight words, no formatting, no punctuation — yet this single instruction fundamentally redirects the trajectory of the entire session. To understand why this message was written, we must examine the context immediately preceding it and the assumptions that led to its necessity.

The Context: An Assistant Going Down a Rabbit Hole

In the messages leading up to this point, the assistant had been engaged in a systematic but increasingly deep investigation of the MoE (Mixture-of-Experts) kernel code path. The session had already achieved modest throughput gains — from roughly 800 tok/s to about 912 tok/s at 512 concurrency — but the user had noted in [msg 657] that GPU power draw remained low (~83W per GPU, against a 600W TDP) and PCIe bandwidth utilization was poor, suggesting the GPUs were severely underutilized. The user's diagnosis was clear: kernel tuning was needed.

The assistant's response to this diagnosis was methodical but perhaps overly academic. It began by extracting GLM-5's MoE parameters (E=256 experts, N=2048 intermediate size, top_k=8), then started investigating the triton-based MoE config files in the sglang source tree ([msg 659]). It checked the available config directories (triton_3_1_0 through triton_3_5_1), looked for server log messages about default configs, and then — most significantly — launched a subagent task to investigate the entire MoE kernel code path for the flashinfer_cutlass backend with NVFP4 quantization ([msg 662]).

This subagent task was the tipping point. The assistant was essentially commissioning a deep-dive research project into the internals of sglang's MoE execution, tracing the call chain from ModelOptNvFp4FusedMoEMethod.apply() through to the actual CUDA kernel invocations. While this investigation would eventually yield useful knowledge, it was a time-consuming detour — and crucially, it was duplicating work that had already been done.

The User's Intervention: Why This Message Was Written

The user's message at [msg 663] is a masterclass in concise redirection. The user recognized that the assistant was about to spend significant effort reverse-engineering MoE kernel behavior from first principles, when the local repository at ./ (which is /home/theuser/glm-kimi-sm120-rtx6000bw/) already contained the fruits of a previous, similar effort: a tuning run for the Kimi K2 model on the exact same RTX PRO 6000 Blackwell hardware.

The phrase "a similar run which did tuning on a similar model" is carefully chosen. It communicates several things at once:

There is prior art. The user is asserting that this problem has been solved before, at least partially.
The prior art is local. The artifacts are in ./, meaning the assistant can read them directly without any network access or external research.
The similarity is high enough to be useful. Both Kimi K2 and GLM-5 are large MoE models deployed on SM120 Blackwell GPUs using NVFP4 quantization. The tuning methodology transfers.
The assistant should stop investigating and start reading. This is an implicit critique of the current trajectory — the assistant is generating new knowledge when existing knowledge is readily available.

Assumptions and Their Implications

The assistant made several assumptions that this message implicitly corrects. First, it assumed that the GLM-5 deployment was a novel problem requiring original investigation. In reality, the user had already conducted extensive MoE tuning for a very similar model on identical hardware, and the findings were sitting in a local directory. Second, the assistant assumed that understanding the full code path was a prerequisite for effective tuning — a reasonable engineering instinct, but one that prioritizes depth over speed. The user's message suggests a different priority: leverage existing results first, optimize later.

The user, for their part, assumed that the assistant would naturally check the local repo before embarking on deep investigation. This assumption proved incorrect — the assistant had referenced FINDINGS.md earlier (<msg id=655, 656>) but hadn't thought to look at the moe-configs/ directory or the tuning logs. The message at [msg 663] is thus both a correction and a nudge: "you have everything you need, look here."

Input Knowledge Required

To understand this message, the reader needs to know several things. First, that ./ refers to the working directory of the coding session, which is /home/theuser/glm-kimi-sm120-rtx6000bw/ — a repository created during a previous deployment of the Kimi K2 model on the same Blackwell GPU hardware. Second, that "a similar run" refers to the Kimi K2 MoE tuning effort documented extensively in that repo, including pre-computed MoE kernel configs for SM120, tuning logs, and benchmark results. Third, that the assistant had just completed a subagent investigation into the MoE kernel code path ([msg 662]) and was about to act on those findings — making this the perfect moment for redirection.

Output Knowledge Created

The immediate output of this message is the assistant's response at [msg 664], where it reads the directory structure of the local repo and discovers:

moe-configs/: Contains five pre-computed MoE kernel configuration files, including E=161,N=192,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition,dtype=fp8_w8a8,per_channel_quant=True.json and TP4/TP8 configs.
logs/: Twelve log files from various tuning runs, including moe-tuning.log, moe-tuning-tp4.log, and multiple sglang server logs with different configurations.
configs/: Hardware configuration files, GPU topology data, and the Kimi K2 model config.
source/: Patched source files including attention fixes and SM120 detection code. This discovery transforms the assistant's approach. Instead of starting from zero, it can now examine how MoE tuning was done for Kimi K2, what configs worked, and what the performance outcomes were — then adapt those findings to GLM-5. The output knowledge is essentially a map of prior solutions that can be directly applied to the current problem.

The Thinking Process Visible in This Exchange

The user's thinking process, though not directly visible, can be inferred. They were watching the assistant's progress, saw the subagent task being launched to investigate MoE code paths, and recognized the inefficiency. The assistant was about to spend potentially dozens of tool calls understanding the internals of flashinfer_cutlass MoE execution when the practical question was simpler: "What MoE configs work well on SM120 for a large MoE model?" The Kimi K2 repo already answered that question.

The terseness of the message is itself informative. The user didn't explain why the repo was relevant, didn't specify which files to look at, and didn't elaborate on how the tuning transferred. They trusted that the assistant would infer all of this from context — and indeed, the assistant immediately understood the implication, reading the directory structure and recognizing the value of the prior work.

This message represents a critical moment of alignment. The assistant was operating in "research mode" — investigate deeply, understand fully, then act. The user was operating in "engineering mode" — find the fastest path to better throughput, even if it means standing on the shoulders of previous work. The message at [msg 663] bridges these two modes, and the session's subsequent trajectory — achieving 3,740 tok/s through FlashInfer CUTLASS MoE autotune — validates the pivot.