Mining Prior Art: How the Assistant Leveraged Kimi K2 Research to Inform GLM-5 MoE Tuning

In the high-stakes world of large language model deployment on novel hardware, every optimization cycle builds on the last. Message 667 captures a quiet but pivotal moment in a longer performance-tuning session: the assistant, at the explicit direction of the user, pauses its active benchmarking work to read prior research artifacts from a similar deployment. This message is not flashy — it contains no kernel patches, no benchmark results, no server restarts. It is a reading operation, pure information gathering. Yet it represents a critical methodological choice: rather than reinventing the wheel, the assistant is mining a local repository of hard-won knowledge from a previous run with Kimi K2, a model architecturally similar to the GLM-5-NVFP4 currently being tuned. This article examines the reasoning, context, assumptions, and knowledge flows embedded in this single message.

The Message: Three File Reads, One Purpose

The subject message, message 667, consists of three read tool calls executed in parallel. Here is the message exactly as the assistant wrote it:

[assistant] [read] /home/theuser/glm-kimi-sm120-rtx6000bw/configs/kimi-k25-config.json
<path>/home/theuser/glm-kimi-sm120-rtx6000bw/configs/kimi-k25-config.json</path>
<type>file</type>
<content>50:     "hidden_size": 7168,
51:     "id2label": {
52:       "0": "LABEL_0",
53:       "1": "LABEL_1"
54:     },
55:     "initializer_range": 0.02,
56:     "intermediate_size": 18432,
57:     "is_decoder": false,
58:     "is_encoder_decoder": false,
59:     "kv_lora_rank": 512,
60:     "label2id": {
61:       "LABEL_0": 0,
62:       "LABEL_1": 1
63:     },
64:     "length_penalty": 1.0,
65...
[read] /home/theuser/glm-kimi-sm120-rtx6000bw/logs/sglang-k2-thinking-final-bench.log
<path>/home/theuser/glm-kimi-sm120-rtx6000bw/logs/sglang-k2-thinking-final-bench.log</path>
<type>file</type>
<content>1: [2026-01-29 01:28:40] INFO model_config.py:859: Using CLI-specified quantization (modelopt_fp4) which is compatible with HF config quant_method (modelopt).
2: [2026-01-29 01:28:40] WARNING model_config.py:885: DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
3: [2026-01-29 01:28:40] INFO server_args.py:1750:...
[read] /home/theuser/glm-kimi-sm120-rtx6000bw/logs/sglang-k2-nvfp4-sm120fix.log
<path>/home/theuser/glm-kimi-sm120-rtx6000bw/logs/sglang-k2-nvfp4-sm120fix.log</path>
<type>file</type>
<content>1: [2026-01-29 01:08:19] INFO model_config.py:859: Using CLI-specified quantization (modelopt_fp4) which is compatible with HF config quant_method (modelopt).
2: [2026-01-29 01:08:19] WARNING model_config.py:885: DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
3: [2026-01-29 01:08:19] INFO server_args.py:1750: Atten...

The message contains no reasoning text, no commentary, no analysis — just three raw file reads. Yet its placement in the conversation, immediately following the user's directive to consult prior work, makes it a deliberate act of knowledge acquisition. The assistant is not acting; it is listening to the archive.

The Context: A Performance Plateau on Blackwell GPUs

To understand message 667, we must first understand where the conversation stands. The assistant is deep into a multi-session effort to deploy GLM-5-NVFP4 — a Mixture-of-Experts (MoE) model with 256 routed experts — across 8 NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The deployment has been running on an LXC container that bypasses the Proxmox hypervisor's IOMMU virtualization, giving the GPUs direct peer-to-peer PCIe access. Despite this architectural victory, performance has plateaued at roughly 880–913 total tokens per second across concurrency levels from 256 to 512. The user has noted that GPU power draw remains low (around 83W idle, far below the 600W TDP) and PCIe bandwidth is underutilized. The bottleneck, both parties agree, lies in the MoE kernel implementation.

The assistant has just finished investigating the MoE kernel code path via a subagent task (msg 662), discovering that the flashinfer_cutlass backend uses CUTLASS kernels compiled at runtime via autotuning. The server is running with --moe-runner-backend flashinfer_cutlass, which bypasses the older Triton-based MoE path entirely. This means the pre-tuned Triton MoE config files sitting in the sglang source tree are irrelevant — the performance tuning must happen at the CUTLASS autotune level, not through static config files.

It is at this juncture that the user intervenes with a simple but powerful instruction (msg 663): "Read info in ./ from a similar run which did tuning on a similar model." The assistant immediately pivots from active experimentation to archival research.

The Message: Three File Reads, One Purpose

Message 667 is the assistant's response to that instruction. It contains three file read operations, each targeting a different artifact from the prior Kimi K2 research repository located at /home/theuser/glm-kimi-sm120-rtx6000bw/:

1. The Kimi K2.5 model configuration (configs/kimi-k25-config.json): The assistant reads lines 50–65 of this JSON file, revealing key architectural parameters: hidden_size: 7168, intermediate_size: 18432, kv_lora_rank: 512. These numbers provide a direct comparison point with GLM-5's own architecture, which the assistant had extracted earlier (msg 658): hidden_size: 6144, moe_intermediate_size: 2048, n_routed_experts: 256. The contrast is instructive: Kimi K2.5 has a larger hidden dimension (7168 vs 6144) and a much larger intermediate size (18432 vs 2048), but fewer total experts (the Kimi K2 config shown earlier had E=161 vs GLM-5's E=256). Understanding these differences is essential for determining whether the MoE tuning parameters from the Kimi K2 run can transfer directly to GLM-5.

2. The final benchmark log (logs/sglang-k2-thinking-final-bench.log): This log captures the server startup for what appears to be the most successful Kimi K2 configuration. The first three lines reveal critical information: the server used modelopt_fp4 quantization (specified via CLI), and it triggered a warning about DeepGemm scale format incompatibility — "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." This warning is identical to one the assistant has likely seen in the current GLM-5 run, confirming that both models share the same quantization pipeline and the same unresolved issue with DeepGemm's scale format expectations.

3. An earlier SM120 fix log (logs/sglang-k2-nvfp4-sm120fix.log): This log, dated January 29, 2026, shows the same modelopt_fp4 quantization path and the same DeepGemm scale_fmt warning. Its presence in the repository suggests it represents an intermediate step in the Kimi K2 optimization journey — perhaps the first successful run after applying SM120-specific attention fixes. The assistant is likely reading it to understand the progression of fixes and to identify which patches were ultimately retained in the final configuration.

The Reasoning: Why Read, Not Act?

The assistant's decision to read these specific files — rather than, say, immediately applying the Kimi K2 tuning configs to GLM-5 — reveals a methodical, comparative approach. The assistant is not blindly copying parameters; it is building a mental model of the similarities and differences between the two deployments. This is visible in the sequence of actions across messages 664–667:

First, the assistant surveys the entire repository structure (msg 664), noting the presence of moe-configs/, logs/, patches/, and source/ directories. Then it drills into the MoE configs (msg 665), finding pre-tuned configs for TP4 and TP8 layouts, plus a config specifically for E=161,N=192 on the RTX PRO 6000 Blackwell. It reads the tuning log (msg 666) to see the exact command-line arguments used for the Kimi K2 MoE tuning run: tp_size=8, ep_size=1, dtype=&#39;fp8_w8a8&#39;, per_channel_quant=True. Finally, in message 667, it reads the model config and server logs to complete the picture.

The assistant is essentially performing a differential analysis: "What was the model architecture? What quantization path did it use? What warnings appeared? What was the final configuration?" Each answer helps the assistant decide which parts of the Kimi K2 playbook apply to GLM-5 and which must be adapted.

Assumptions and Their Validity

This message rests on several key assumptions, some explicit and some implicit:

Assumption 1: Kimi K2 and GLM-5 are sufficiently similar for tuning transfer. Both are large MoE models deployed on the same 8x RTX PRO 6000 Blackwell hardware, using the same quantization format (NVFP4/modelopt_fp4) and the same inference engine (sglang with flashinfer_cutlass backend). This is a reasonable assumption, but the architectural differences (different hidden_size, different expert counts, different intermediate_size) mean that optimal kernel block sizes and tile configurations may differ. The assistant implicitly acknowledges this by reading the model config for comparison rather than blindly applying the old tuning.

Assumption 2: The Kimi K2 tuning was successful and worth emulating. The repository contains multiple log files, some labeled "fix," "pr," "optimized," and "final-bench." The assistant reads both an intermediate fix log and the final benchmark log, suggesting it is trying to understand which configuration was actually the winning one. The very existence of a "final-bench" log implies a successful outcome, but the assistant has not yet verified what throughput numbers that final configuration achieved.

Assumption 3: The logs contain actionable information. The assistant reads only the first three lines of each log file. This is a risky assumption — the critical information (actual benchmark results, error messages, configuration flags) may appear later in the file. The truncated reads (indicated by "..." in the output) suggest the assistant is either sampling the beginning of the files for a quick overview or is limited by the tool's output truncation. In either case, the assistant may be missing crucial details about what made the Kimi K2 deployment succeed.

Assumption 4: The local repository is the authoritative source. The assistant does not cross-reference the Kimi K2 findings with external documentation or the sglang source code. It trusts that the prior work, captured in these logs and configs, represents the ground truth for what works on SM120 hardware. Given that the same hardware and software stack are in use, this is a defensible assumption, but it does risk propagating any mistakes or suboptimal choices from the earlier run.

Input Knowledge: What the Reader Needs to Understand

To fully grasp message 667, one needs to understand several layers of context:

The hardware landscape: The RTX PRO 6000 Blackwell is a workstation GPU based on the Blackwell architecture (compute capability SM120). Unlike datacenter Blackwell GPUs (B200/B100, SM100), the SM120 variant has different shared memory sizes, different warp scheduling, and different support for advanced features like allreduce fusion. This architectural divergence is the root cause of many of the performance challenges.

The MoE inference pipeline: In sglang, MoE layers can be executed through multiple backends: Triton kernels, CUTLASS kernels (via flashinfer), or DeepGemm. Each backend has different autotuning mechanisms and different hardware requirements. The flashinfer_cutlass backend compiles CUTLASS kernels at runtime, selecting tile sizes and block configurations based on the model dimensions and GPU architecture.

The quantization format: NVFP4 (modelopt_fp4) is a 4-bit floating-point quantization format developed by NVIDIA's Model Optimizer. It stores weights in FP4 format with per-group scaling factors. The DeepGemm library, which provides optimized GEMM operations for Blackwell, expects a specific scale format (ue8m0) that the NVFP4 checkpoints do not use, triggering the warning visible in the logs.

The prior work: The Kimi K2 project was a previous effort to deploy a different MoE model (Kimi K2.5/K2-Thinking) on the same 8x Blackwell GPU setup. That project produced the repository being mined, including tuned MoE kernel configurations, server logs, and patches for SM120-specific issues like Triton shared memory exhaustion.

Output Knowledge: What the Assistant Gains

From these three file reads, the assistant extracts several pieces of actionable knowledge:

  1. Architecture comparison data: The assistant can now compare GLM-5's hidden_size=6144, intermediate_size=2048, E=256 against Kimi K2.5's hidden_size=7168, intermediate_size=18432, E=161. This comparison informs whether the MoE tuning parameters (BLOCK_SIZE_M, BLOCK_SIZE_N, BLOCK_SIZE_K, etc.) from the Kimi K2 configs can be applied directly or need adjustment.
  2. Quantization path confirmation: Both models use the same modelopt_fp4 quantization path, and both trigger the same DeepGemm scale_fmt warning. This confirms that the quantization handling is identical and that any DeepGemm-related issues will affect both models equally.
  3. Server startup pattern: The log files show the server starting with modelopt_fp4 quantization specified via CLI, confirming that the --quantization modelopt_fp4 flag is the correct approach for NVFP4 models on Blackwell.
  4. Tuning methodology: The moe-tuning.log (read in msg 666) shows the exact command used for Kimi K2 tuning, including tp_size=8, ep_size=1, dtype=&#39;fp8_w8a8&#39;, and per_channel_quant=True. This provides a template for running the MoE tuning script on GLM-5.

A Subtle Mistake: The Truncated Log Problem

One notable limitation of this message is the truncated log reads. Both sglang-k2-thinking-final-bench.log and sglang-k2-nvfp4-sm120fix.log are cut off after three lines (indicated by "..."). The assistant is seeing only the server startup messages — the quantization confirmation and the DeepGemm warning — but not the actual benchmark results, error messages, or configuration details that would appear later in the files.

This truncation could lead to a significant blind spot. The assistant might assume that the Kimi K2 final configuration was purely a modelopt_fp4 setup, when in fact the critical tuning might have involved switching to a different MoE backend, adjusting CUDA graph settings, or modifying the attention implementation. Without reading the full logs, the assistant cannot verify which configuration actually produced the best results.

The assistant does not appear to notice or correct for this truncation in message 667. It simply accepts the first three lines and moves on. In subsequent messages (not shown in this segment), the assistant may need to re-read these files with more targeted queries to extract the full picture.

The Broader Significance: Knowledge Transfer in ML Engineering

Message 667, for all its apparent simplicity, illustrates a fundamental pattern in machine learning engineering: the critical importance of knowledge transfer between similar projects. The assistant is not starting from scratch; it is standing on the shoulders of the Kimi K2 work. This is how real-world ML infrastructure development happens — each deployment generates artifacts (configs, logs, patches, benchmarks) that inform the next deployment.

The user's instruction to "read info in ./ from a similar run" is itself a recognition of this pattern. Rather than asking the assistant to reason from first principles about MoE kernel tuning on SM120, the user directs it to consult empirical evidence from a closely related effort. This is a form of few-shot learning applied to engineering: show the system what worked before, and let it adapt.

The assistant's response demonstrates that it understands this meta-lesson. It does not just read one file; it reads three, covering model architecture, final configuration, and intermediate debugging steps. It is building a comprehensive picture of the prior work before making any decisions about how to proceed. This methodological discipline — research before action — is one of the hallmarks of effective engineering problem-solving, and message 667 captures it in microcosm.