The Digital Archaeologist: How an AI Agent Systematically Uncovered a Hidden Benchmark Tool

Introduction

In the course of a sprawling machine learning infrastructure session—one that had already spanned NVIDIA driver installation, CUDA toolkit configuration, flash-attention compilation battles, and the deployment of the GLM-5-NVFP4 model across eight Blackwell GPUs—the conversation arrived at a critical juncture. The user needed to test the performance of the deployed inference server. Somewhere in the working directory /home/theuser/glm-kimi-sm120-rtx6000bw, a load testing tool was rumored to exist. The assistant's task was to find it.

What follows is a remarkable case study in systematic digital investigation. Over the course of thirteen messages, a subagent—a spawned child session operating independently from its parent—methodically explored a research repository, sifted through log files, parsed documentation, and ultimately identified the load testing tool through forensic evidence rather than direct discovery. The tool was never stored in the repository at all. It was SGLang's built-in bench_serving module, and its presence had to be inferred from the artifacts it left behind.

This article examines that investigation in detail, tracing the subagent's reasoning, decisions, and assumptions through each phase of the search.

The Mission: Exploring an Unknown Repository

The subagent's mission began with a clear directive from the parent session: "Explore the directory /home/theuser/glm-kimi-sm120-rtx6000bw thoroughly. The user mentioned there's a load testing tool somewhere in this directory. Look through all subdirectories (source/, configs/, moe-configs/, logs/, patches/) and find any scripts, configs, or tools related to load testing, benchmarking, or performance testing of LLM inference servers (like sglang, vllm). Also check the README.md and FINDINGS.md for any references to load testing tools or benchmarks." [2]

This message reveals several important contextual details. First, the directory is a research workspace—likely a clone or fork of a repository associated with deploying GLM (General Language Model) variants on high-end NVIDIA hardware. The subdirectories (source/, configs/, moe-configs/, logs/, patches/) suggest an organized research project with custom source code, configuration files for Mixture-of-Experts models, logs from previous runs, and patches applied to upstream frameworks. Second, the parent agent is operating on an unverified user claim—the human user mentioned a load testing tool "somewhere," but this has not been confirmed. Third, the parent agent has already done some reconnaissance, knowing the exact subdirectory names to specify.

The subagent's mission is not merely to find a file, but to reconstruct the experimental history of this repository and identify the tools used to generate its benchmark results.

Phase 1: The Broad Survey

The subagent's first move was to get the lay of the land [1]. It issued two parallel tool calls: a directory read and a recursive glob search. The directory listing revealed eleven entries: .git/, configs/, FINDINGS.md, logs/, moe-configs/, patches/, README.md, report-kimi-k2-sm120.html, report-kimi-k2-sm120.pdf, sm120-attention-fix-research.md, and source/. The glob search began streaming results but was truncated, showing only .git internal files before cutting off.

This initial survey established the repository's structure and hinted at its purpose. The presence of FINDINGS.md and report-kimi-k2-sm120.html indicated that this was not just a code repository but a research artifact collection with documented results. The sm120-attention-fix-research.md file pointed to a specific hardware challenge—SM120 (Blackwell) GPUs had required attention kernel fixes. The logs/ directory, with twelve log files, promised a detailed record of previous experiments.

The subagent's decision to start broad rather than diving into specific files was strategically sound. In unfamiliar territory, the first priority is always orientation. You cannot know which files are relevant until you know what files exist.

Phase 2: Reading the Documentation and Searching Broadly

With the directory structure established, the subagent moved to its second round [3]. It read README.md and FINDINGS.md while simultaneously running a comprehensive grep for load testing terminology. The grep pattern was itself a work of careful design:

load[_ -]?test|bench|benchmark|perf|throughput|latency|stress[_ -]?test|requests per|tokens per|tok/s|rps|qps

This pattern covered explicit testing terms (load_test, stress_test), benchmarking terms (bench, benchmark), performance metrics (throughput, latency), and domain-specific abbreviations (tok/s, rps, qps). The inclusion of rps (requests per second) and qps (queries per second) was particularly telling—these are the kind of abbreviations that appear in load testing output but might be missed by a generic search.

The grep returned 1,241 matches, an overwhelming number that included many false positives from .git template files. But the documentation reads were immediately valuable. README.md revealed that the repository contained "research artifacts, patches, and benchmark results from deploying Kimi K2.5 and K2-Thinking on 8x NVIDIA RTX PRO 6000 Blackwell GPUs." It even included a benchmark table showing 5,816 tok/s for Kimi K2-Thinking with NVFP4 quantization. FINDINGS.md was even more detailed—an 871-line document with comprehensive benchmark tables, configuration notes, and debugging narratives.

The subagent now had a critical piece of information: the repository contained benchmark results, but the tool that generated those results was not immediately visible.

Phase 3: Targeted Searches and Log Analysis

The third round marked a shift from broad exploration to targeted investigation [4]. The subagent ran two more grep patterns: one for specific load testing tool names (bench_serving, sglang.bench, locust, wrk, hey, ab, k6, gatling, vegeta, etc.) and another for benchmark invocation patterns (python3 -m sglang.bench, bench_serving, sglang_bench, openai.*bench, benchmark_serving). The first grep found only 9 matches—all false positives from .git hooks. The second found nothing at all.

This was a pivotal moment. The subagent had systematically ruled out the most common load testing tools. No Locust configuration, no wrk script, no hey binary, no ab benchmark plan. The repository did not contain any of the standard tools one would expect for load testing an LLM inference server.

But the logs/ directory beckoned. The subagent read several log files, including sglang-k2-thinking-final-bench.log (5,631 lines), sglang-k2-nvfp4-sm120fix.log, sglang-k25-int4-sm120fix.log, and sglang-optimized.log. These logs revealed a consistent pattern: SGLang server startup messages, model loading, CUDA graph capture, and then bursts of POST /generate requests from clients. The logs showed the server-side perspective of benchmark runs, but they did not contain the benchmark client's identity.

Phase 4: The Forensic Breakthrough

The subagent's investigation entered its most creative phase when it realized that the benchmark client's identity could be inferred from its behavior rather than its name [5]. It searched for the specific output format of SGLang's built-in benchmark tool:

Serving Benchmark Result|bench_serving|sglang\.bench_serving|benchmark_serving|openai.*benchmark|python3.*bench

No files found. The benchmark results were not stored in the repository's text files in that format.

But then the subagent noticed something in the log files. At the end of benchmark runs, the server logs showed GET /get_server_info requests—a deprecated endpoint that SGLang's benchmark client calls to collect server metrics after completing a test run [12]. This was the signature. The /get_server_info calls, combined with the bursts of POST /generate requests (rather than OpenAI-compatible /v1/chat/completions), pointed unmistakably to SGLang's own bench_serving module.

The subagent confirmed this by cross-referencing the benchmark parameters documented in FINDINGS.md with sglang.bench_serving's known options. The concurrency levels tested (1, 8, 32, 64, 128, 256, 384, 500, 512, 1000, 1024, 2048), the input/output token length sweeps, and the reported metrics (TTFT, TPOT, request throughput, output token throughput, total token throughput) all matched the built-in tool's output format exactly.

Phase 5: The Comprehensive Summary

The subagent's final message [13] delivered a thorough summary of its findings. The key conclusion: no standalone load testing tool existed in the repository. The benchmarking had been performed using SGLang's built-in sglang.bench_serving module, which is part of the SGLang framework installation and not vendored into the repository. What the repository did contain was:

The Thinking Process: What Made This Investigation Effective

The subagent's investigation succeeded because of several deliberate strategic choices:

1. Hierarchical exploration. The subagent started broad (directory listing), then narrowed (documentation), then targeted (specific tool names), then forensic (behavioral signatures). Each phase informed the next, preventing wasted effort on irrelevant files.

2. Parallel information gathering. Multiple tool calls in each round maximized efficiency. The subagent read documentation and ran grep searches simultaneously, never waiting for one result before dispatching the next.

3. Domain-specific search design. The grep patterns were crafted with deep knowledge of LLM serving infrastructure. Terms like tok/s, rps, qps, TTFT, and TPOT are domain-specific jargon that a general-purpose search would miss but that anyone familiar with LLM benchmarking would know to include.

4. Forensic reasoning. When direct discovery failed, the subagent shifted to indirect inference. It recognized that the benchmark tool's identity could be deduced from its effects on the server logs—the /get_server_info calls, the POST /generate endpoint usage, and the concurrency patterns.

5. Cross-referencing. The subagent validated its hypothesis by cross-referencing multiple sources: the benchmark output format in FINDINGS.md, the server-side log patterns, and the documented parameters of sglang.bench_serving. This triangulation produced a confident conclusion.

Assumptions and Limitations

The investigation was not without assumptions. The subagent assumed that the load testing tool, if present, would be discoverable through file enumeration and text search. It assumed that the documentation in README.md and FINDINGS.md was accurate and up-to-date. It assumed that the grep patterns would cover the relevant terminology. And it assumed that the user's claim about a load testing tool "somewhere" in the directory was accurate—which turned out to be true in a sense (the tool was used from within this directory context), but misleading in another (the tool was not actually stored there).

The most significant limitation was the subagent's inability to run the benchmark tool itself. The parent session had not yet provided the SGLang server configuration or ensured the server was running. The subagent's task was purely investigative—find the tool, not use it.

Conclusion

This subagent session is a masterclass in systematic digital investigation. Faced with an unknown repository, an unverified user claim, and a complex technical domain, the subagent methodically worked through phases of exploration, documentation analysis, targeted search, and forensic inference. It ultimately identified the load testing tool not by finding it in the filesystem, but by reconstructing the experimental history of the repository from the artifacts left behind.

The investigation transformed raw directory listings and log files into a coherent narrative: the repository documented a successful deployment of large MoE models on Blackwell GPUs, the critical SM120 attention fix that made it possible, and the SGLang built-in benchmark tool used to measure performance. The subagent's comprehensive summary provided the parent session with everything it needed to understand the repository's benchmarking capabilities—and, crucially, to know that the load testing tool was already installed as part of SGLang, ready to be used.

In the broader context of the coding session, this investigation represents a pivot from infrastructure setup to performance evaluation. The environment was stable, the model was deployed, and now the tools for measuring its performance had been identified. The next phase—actual load testing—could begin.