The Systematic Archaeology of Inference Optimization: Mining a Blackwell GPU Experiment Repository

Introduction

In the high-stakes world of large language model (LLM) inference on cutting-edge hardware, the difference between a struggling deployment and a finely tuned engine often lies not in the model architecture itself, but in the invisible constellation of environment variables, launch flags, communication protocol settings, and kernel backends that surround the serving stack. When an AI assistant is tasked with systematically mining a repository of prior experiments for every optimization discovered across months of work on an 8× NVIDIA RTX PRO 6000 Blackwell (sm_120) system connected via PCIe without NVLink, the resulting exploration reveals the intricate archaeology of applied systems optimization.

This article examines a complete subagent session—messages 0 through 15—that represents a masterclass in systematic knowledge discovery. The subagent, invoked with the mandate "Find prior inference optimizations (agent: explore)," navigated a sprawling repository containing experiment files for multiple model families (Kimi-K2.5, Kimi-K2.6, GLM-5, EAGLE/K2), service configurations, benchmark results, improvement proposals, and planning documents. Over the course of 16 messages, the assistant evolved from broad reconnaissance to targeted deep dives, from raw data collection to sophisticated synthesis, ultimately producing a comprehensive findings report that documented not just what worked, but—critically—what failed and why.

The Mission: A Detailed Search Request

The session began with a meticulously structured user request at [msg 0]. The user asked the assistant to search the repository at /home/theuser/glm-kimi-sm120-rtx6000bw—especially the kdtree-engine/ subdirectory with its plans/, scripts/, sglang_ext/, and bench_results_ct200/ folders—for "ALL inference performance optimizations, environment variables, and launch flags that were applied or discovered in the previous Kimi-K2.6 / DDTree experiments on the 8x RTX PRO 6000 (sm_120, PCIe, no NVLink) box."

The user specified six detailed categories of interest: NCCL tuning (protocols like LL/LL128/Simple, algorithms, P2P levels, NVLS), all-reduce optimizations (custom all-reduce, MSCCL, SGLANG flags), memory allocator and CUDA flags (expandable segments, TF32 precision, CUDA graphs), attention backend choices (Triton vs FlashInfer vs FlashMLA), MoE backend choices (Marlin, expert-parallel, DeepEP, EPLB), and any systemd unit files, launch scripts, or environment files that set variables for the SGLang server. The user demanded exact file paths, line numbers, and quoted lines—not summaries or interpretations [6].

This request encoded deep domain expertise. The six categories mapped directly onto the known bottlenecks of multi-GPU LLM inference on PCIe-connected hardware, and the specific variable names revealed an understanding that on this hardware—8 GPUs connected via PCIe Gen5 without NVLink—inter-GPU communication is the primary performance constraint. The request also reflected a disciplined engineering methodology: rather than randomly trying flags on the live deployment, the user wanted to mine prior experimental knowledge first.

Phase One: Broad Reconnaissance and the Signal-to-Noise Problem

The assistant's first response ([msg 1]) was a coordinated reconnaissance package: a root directory listing, a subdirectory mapping of kdtree-engine/, and file-type globbing for markdown, text, and shell scripts. This revealed the key organizational units of the project—bench_results_ct200/, build/, docs/, .git/, plans/, scripts/, and src/common/—along with benchmark visualization screenshots suggesting an iterative optimization process spanning at least 11 days [1].

In [msg 2], the assistant ran broad grep searches for NCCL_[A-Z_]+ and all.?reduce patterns across the entire repository. These returned 1,200 and 4,970 matches respectively—but the vast majority came from the vendored sglang-dsv4 upstream source tree, not from experiment-specific files. The assistant was flooded with noise from upstream test files like test_update_weights_from_distributed.py and test_server_args.py, while the actual K2.6 service files remained hidden [12].

This was a critical calibration moment. The assistant recognized that its initial approach was producing noise rather than signal and pivoted decisively in [msg 3]. The reasoning block captured the insight: "I need to filter out the upstream SGLang source code and focus on the experiment-specific directories like kdtree-engine with its plans, scripts, and benchmark results, along with the root-level documentation files that contain the actual Kimi-K2.6 and DDTree experiment configurations." This meta-cognitive awareness—the ability to recognize when a search strategy is failing and adjust course—is one of the most valuable properties of an effective exploration agent [5].

Phase Two: Reading the Ground Truth

With the refined strategy, the assistant began reading the actual deployment artifacts. In [msg 4], it examined the systemd service files for Qwen3.6-based services (ct200-sglang-ddtree-tree211.service, ct200-sglang-ddtree-shadow211.service, ct200-sglang-dflash-smoke211.service, etc.) along with the reconfig_ddtree.sh and run_opt_sweep.sh scripts. These files revealed the operational workflow: the DDTree speculative decoding parameters (budget, topk, window) were tuned through a process of editing the service file, restarting systemd, and running automated sweeps. However, these service files were surprisingly sparse in NCCL tuning—they set CUDA_VISIBLE_DEVICES=1 (single GPU), the PATH, and LD_LIBRARY_PATH, but no NCCL environment variables [8].

The breakthrough came in [msg 5], when the assistant discovered the ct200_sglang_working/services/ directory and found the actual K2.6 production service files. A targeted grep for tp-size 8, Kimi-K2\.6, and sglang-k26 returned the critical hit: the ExecStart line of sglang-k26-ddtree.service, revealing flags like --tp-size 8, --mem-fraction-static 0.85, --context-length 32768, --page-size 1, --num-continuous-decode-steps 8, --attention-backend triton, and --speculative-algorithm DDTREE. Even more valuable was the discovery of the DDTree kernel integration architecture: kdtree_mla_backend.py monkeypatched TritonAttnBackend.forward_extend to inject custom sm_120 kernels, launch_with_kdtree.py provided the entrypoint, and sitecustomize.py used Python's auto-import mechanism to propagate the override to all TP worker processes [4].

In [msg 6], the assistant read the four K2.6 service files in full: sglang-k26-ddtree.service ("K2.6 TP8 + DDTree + cuda graphs"), sglang-k26-tp8-tuned.service ("K2.6 TP8 tuned (no EP, cuda graphs)"), sglang-k26-ep8-best.service ("K2.6 EP8 best"), and sglang-k26-dflash.service ("K2.6 EP8 + DFlash speculative"). Each represented a distinct optimization approach, and the NCCL environment variables within them—NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=5, NCCL_MIN_NCHANNELS=8, NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216, NCCL_NTHREADS=512, OMP_NUM_THREADS=8—formed the canonical NCCL configuration for PCIe-only Blackwell inference [15].

Phase Three: The Critical Distinction

The most important analytical insight of the entire exploration emerged in [msg 10]. The assistant noticed a critical pattern: "the K2.6 sglang services and GLM-5 scripts use NCCL_P2P_LEVEL=5, while the vLLM K2.5 services use NCCL_P2P_LEVEL=SYS." This divergence in peer-to-peer communication strategy was not random—it correlated with the model family and serving framework. The SYS setting (system memory) is the safest and most conservative option for PCIe topologies, while the numeric value 5 enables direct peer-to-peer transfers between GPUs connected via PCIe switches, exploiting the PIX topology where GPUs are paired as 0↔1, 2↔3, 4↔5, 6↔7 [9].

This observation drove the assistant to read four key improvement documents: FINDINGS.md (GLM-4.7 deployment findings), glb5improvement-02-mscclpp.md (MSCCLPP one-shot allreduce), glb5improvement-05-allreduce-fusion.md (FlashInfer allreduce fusion), and glb5improvement-04-expert-parallelism.md (expert parallelism). These documents revealed a tiered prioritization framework: MSCCLPP was "Tier 1 (flag flip)" with expected 10-30% improvement, allreduce fusion was "Tier 2 (code change required)" with 5-15%, and expert parallelism was "Tier 2 (small code change)" with 30-100% improvement—the single biggest optimization opportunity [9].

Phase Four: The Allreduce Bottleneck

The synthesis reached its climax in [msg 14], where the assistant assembled the evidence for the allreduce bottleneck. From eagle-fast-verify.md, it extracted the latency budget: "122 allreduces × 0.15-0.20 ms each ≈ 18-25 ms. The compute for 3 INT4-quantized tokens is only ~5-8 ms. The verify pass is ~70% idle, waiting on NCCL." From k25b6000bench1.md, it found the comparative bottleneck analysis showing that after switching to Marlin W4A16 kernels (which eliminated the dtype cast bottleneck at 69%), allreduce became the #1 bottleneck at 51.5% [2].

This was the moment of synthesis. The assistant had connected the NCCL configuration (from service files) with the benchmark data (from findings reports) with the improvement proposals (from planning documents) to produce a coherent diagnosis: on PCIe-only Blackwell, the decode regime is communication-bound, with the verify pass spending 70% of its time waiting on NCCL allreduce operations. The path forward required either reducing the number of allreduces (through fusion or expert parallelism) or accelerating the allreduce itself (through MSCCLPP or custom kernels) [2].

Phase Five: The Final Sweep and Comprehensive Report

Messages 11 through 13 represented the final data-gathering phase. The assistant read remaining files to fill knowledge gaps: ddtree_openai_server.py for TF32 settings, eagle-fast-verify.md for SM120 patches, marlin_moe.txt for Marlin MoE benchmarks, verify_attn_flash.txt for custom verify kernel results [3]. It performed targeted ripgrep searches for SGLANG_[A-Z_]+, KDTREE_[A-Z_]+, NCCL_P2P_LEVEL, and other patterns across the repository, excluding vendored source trees [14]. It identified bench_runner.py as the generator of service units with NCCL environment variables, and eagle-fast-verify.md line 360 as containing an actual PCIe custom-all-reduce experiment result [11].

The culmination was [msg 15], a comprehensive 5,027-word findings report organized into six sections mirroring the user's original request categories. The report documented the canonical NCCL configuration block, the graveyard of failed all-reduce optimizations (custom all-reduce: 2.3× slower; FlashInfer fusion: 7.9× slower; Tree algorithm: crashes; torch symmetric memory: KeyError; MSCCL++: +1.9% marginal), the CUDA graph optimization delivering 3.8× speedup at C=1, the Triton attention backend mandate imposed by DDTree's custom mask requirement, the Marlin W4A16 MoE path achieving ~1.21 TB/s (67% of GDDR7 peak), and the sm_120 shared memory wall affecting all Triton-based MoE backends [16].

The Knowledge That Emerged

Across the 16 messages, several profound insights emerged that no single file contained:

The constraint cascade. The choice of DDTree speculative decoding cascaded through the entire system: DDTree requires custom masks → requires Triton attention backend → requires page size 1 → scatters KV cache → motivates native verify kernel. This causal chain was invisible in any single configuration file but became clear when the full system was documented [16].

The misdiagnosed bottleneck. The experimenters had been pursuing all-reduce optimizations based on the assumption that communication was the bottleneck. The report revealed that at low concurrency, the GPUs were running at 33-42% power—underutilized, not communication-starved. The real bottleneck was the inability to amortize fixed costs across enough tokens [16].

The Pareto frontier. The benchmark deltas table ranked optimizations by impact: CUDA graphs (3.8×), DDTree speculative decoding (150-170 tok/s at C=1), native verify kernel (2.0-6.1×), NVLS on B300 (+5-6%), MSCCL++ (+1.9% marginal). The biggest wins came from architectural changes, not communication micro-optimizations [16].

The negative result documentation. The most valuable knowledge was often what didn't work. Custom all-reduce on PCIe produced a 2.3× slowdown. FlashInfer allreduce fusion crashed on SM120. The Tree algorithm broke CUDA graph capture. These dead ends saved future experimenters from repeating the same mistakes [16].

Methodology: How Systematic Exploration Works

The assistant's methodology across these 16 messages reveals a pattern applicable to any complex knowledge-discovery task:

  1. Broad reconnaissance (messages 1-2): Map the terrain, understand the directory structure, identify file types.
  2. Strategic filtering (message 3): Recognize noise sources, refine search scope, focus on experiment-specific artifacts.
  3. Ground-truth reading (messages 4-6): Read deployment artifacts (service files, scripts) that encode actual production configurations.
  4. Pattern recognition (messages 7-10): Identify divergences across configurations (NCCL_P2P_LEVEL), cross-reference findings across models.
  5. Hypothesis formation and validation (messages 11-13): Form hypotheses about bottlenecks, seek evidence in benchmark data and improvement proposals.
  6. Synthesis and reporting (messages 14-15): Assemble the pieces into a coherent diagnosis, document both positive and negative results. This methodology mirrors how a skilled human engineer would approach the same task—but the assistant executed it with relentless thoroughness, reading dozens of files, running targeted searches, and cross-referencing findings across model families without the cognitive fatigue that would limit a human researcher.

Conclusion

The 16-message subagent session examined in this article represents a complete arc of systematic knowledge discovery: from broad reconnaissance to targeted deep dives, from raw data collection to sophisticated synthesis, from scattered configuration files to a unified diagnosis of the system's bottleneck hierarchy. The assistant's ability to navigate a large, heterogeneous repository, identify patterns across disparate files, and produce a structured findings report demonstrates the power of AI-assisted engineering research.

For anyone deploying large language models on PCIe-only Blackwell systems, the accumulated knowledge in this session is invaluable. It documents the canonical NCCL configuration, the dead ends to avoid, the Pareto-optimal levers to pull, and the fundamental constraints that cannot be overcome by tuning alone. But perhaps more importantly, it demonstrates a methodology—systematic exploration, cross-referencing, synthesis, and reframing—that is applicable far beyond this specific hardware configuration. In a field where hardware evolves rapidly and the optimization space is vast, the ability to mine one's own history for insights is perhaps the most underrated skill in the ML engineer's toolkit.