Orchestrating Parallel Research: How an AI Agent Systematically Investigates GPU Kernel Optimization

In the long and winding campaign to optimize inference throughput for the GLM-5-NVFP4 model on NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a pivotal moment that exemplifies the methodical, evidence-driven approach that defines the entire effort. Message 1066 in this conversation is not a flashy breakthrough or a dramatic benchmark result—it is something arguably more important: the disciplined orchestration of parallel research before committing to implementation. This message shows the assistant reading five existing improvement documents while simultaneously launching three independent research agents to survey the latest developments in sglang, GLM-5 deployment strategies, and SM120 FP4 kernel optimization. The result is a rich synthesis of prior knowledge and fresh intelligence that sets the stage for the next wave of optimization work.

The Context: An Optimization Campaign in Full Swing

To understand the significance of this message, one must appreciate where it sits in the broader narrative. By message 1066, the assistant and user have already been through an extraordinary journey spanning multiple segments and dozens of messages. They have set up an entire ML environment from scratch on Ubuntu 24.04, resolved complex build issues for flash-attn, deployed the GLM-5-NVFP4 model using SGLang, and conducted an extensive optimization campaign that has already yielded impressive results—throughput improved from roughly 880 tokens per second to over 3,740 tokens per second through techniques like FlashInfer CUTLASS MoE autotune for SM120 and careful server parameter tuning.

But the optimization campaign has also encountered stubborn bottlenecks. The core problem is now well-understood: the FP4 MoE expert GEMMs on SM120 are memory-bandwidth-bound rather than compute-bound, and the small per-expert matrix multiplications fail to fully utilize the GPU's theoretical capacity. The assistant has documented eleven improvement ideas in separate markdown files (glb5improvement-01 through glb5improvement-11), ranging from simple flag changes to significant engineering efforts. Some have been tested; others remain unexplored.

The user's instruction in message 1064 is deceptively simple: "Continue implementing improvements, read related docs and maybe start few explore agents to comb through latest on the model on the internet." This is a high-level directive that gives the assistant significant autonomy in deciding which improvements to pursue and how to gather the necessary information.

What the Message Actually Contains

Message 1066 is the assistant's response to that instruction. It consists of three distinct phases of activity, all dispatched in a single round:

Phase 1: Reading existing documentation. The assistant reads five improvement documents: glb5improvement-04 (Expert Parallelism), glb5improvement-05 (Allreduce Fusion), glb5improvement-06 (L2 Cache Pinning), glb5improvement-07 (Persistent Grouped GEMM), and glb5improvement-11 (Column-Major GEMM Scheduling + XShare Expert Deduplication). These are the improvement ideas that are either ready to test or require significant engineering. The assistant is refreshing its understanding of what has been proposed, what the expected impact is, and what the implementation path looks like.

Phase 2: Launching parallel research agents. The assistant spawns three task tool calls simultaneously, each running as a subagent that independently browses the internet and reads documentation:

1. Research latest sglang SM120 updates — Investigating recent commits, issues, and PRs in the sglang repository related to Blackwell GPU support, MoE inference optimization, FP4 GEMM improvements, and expert parallelism.
2. Research GLM-5 deployment strategies — Searching for deployment guides, community discussions, and optimization tips for the specific GLM-5-NVFP4 model.
3. Research SM120 FP4 kernel optimization — Surveying the latest developments in NVIDIA's CUTLASS library and other kernel frameworks for FP4 computation on SM120 (Blackwell RTX) GPUs. Phase 3: Receiving and synthesizing research results. The task results are returned inline within the same message, providing comprehensive summaries of what each research agent discovered. These are not trivial outputs—each runs to multiple paragraphs covering architecture, known issues, optimization strategies, and specific recommendations.

The Reasoning Behind the Parallel Research Strategy

The decision to launch three research agents in parallel rather than sequentially is a deliberate architectural choice that reveals the assistant's understanding of the optimization landscape. Each research direction is independent: sglang updates, GLM-5 deployment strategies, and SM120 kernel optimization address different layers of the problem stack. The sglang research covers the serving framework layer, the GLM-5 research covers model-specific deployment knowledge, and the SM120 research covers the hardware/kernel layer. There is no dependency between them, so they can safely run concurrently.

This parallelism is made possible by the task tool, which spawns a subagent that runs its own multi-round conversation to completion before returning. The parent session is blocked during subagent execution, but since all three subagents run simultaneously, the total wall-clock time is roughly the maximum of the three individual runtimes rather than their sum. This is a practical efficiency gain, but it also reflects a deeper design principle: the assistant treats research as a first-class operation that deserves dedicated computational resources, not as an afterthought squeezed between implementation steps.

The choice of which five improvement documents to read is also revealing. The assistant skips the lower-priority or already-tested ideas (improvements 01-03, 08-10) and focuses on the documents that represent the next frontier: Expert Parallelism (the "SINGLE BIGGEST optimization opportunity" per the document's own assessment), Allreduce Fusion (a Tier 2 code change with modest expected impact), L2 Cache Pinning (significant engineering with high potential), Persistent Grouped GEMM (the most ambitious with 2-5x projected improvement), and Column-Major GEMM Scheduling (a clever scheduling optimization). This selection shows the assistant is thinking in terms of a pipeline: start with the low-effort, high-impact items (EP8), then move to the more involved engineering efforts if needed.

Assumptions Embedded in the Approach

The assistant makes several assumptions in this message, some explicit and some implicit.

The most fundamental assumption is that the latest developments in open-source repositories and community discussions are worth investigating before proceeding with implementation. This is not a trivial assumption—it implies that the optimization landscape is evolving quickly enough that yesterday's best approach might be superseded by today's commit. Given that the assistant is working with a very recent model (GLM-5 was released in February 2026) on brand-new hardware (RTX PRO 6000 Blackwell GPUs), this assumption is well-justified. The sglang framework, CUTLASS library, and FlashInfer are all under active development, and new optimizations for SM120 are appearing regularly.

Another assumption is that the improvement documents represent a complete and accurate catalog of optimization opportunities. The assistant treats these documents as authoritative sources, reading them to refresh its understanding rather than questioning their analysis. This is reasonable given that the documents were written by the same assistant in earlier messages, based on extensive benchmarking and analysis. However, it does mean that any blind spots in the original analysis will propagate forward.

The assistant also assumes that the three research directions are independent and can be pursued in parallel without interference. This is true at the information-gathering stage, but the implementation decisions that follow from each research stream may interact in complex ways. For example, a new sglang commit might change the API for expert parallelism, or a CUTLASS kernel optimization might render the allreduce fusion approach obsolete. The assistant will need to reconcile these findings in subsequent messages.

Perhaps the most subtle assumption is that research should precede implementation. The user's instruction was to "continue implementing improvements," but the assistant chooses to first gather more information. This reflects a deliberate risk-management strategy: better to spend time researching now than to implement an approach that has already been superseded or that misses a better alternative. The assistant is implicitly prioritizing correctness and optimality over speed of execution.

Mistakes and Incorrect Assumptions

While the message is well-structured and the reasoning is sound, there are potential pitfalls worth examining.

One concern is the completeness of the research coverage. The three research agents cover sglang, GLM-5 deployment, and SM120 FP4 kernels, but there are other relevant domains that are not explored. For instance, the FlashInfer library (which handles the MoE routing and attention kernels) is not directly researched, even though it was identified in earlier messages as a critical component. The allreduce fusion approach mentioned in improvement 05 depends on FlashInfer internals. Similarly, the NCCL (NVIDIA Collective Communications Library) tuning space is not explored, even though allreduce performance is a key bottleneck for expert parallelism.

Another potential issue is the depth versus breadth trade-off in the research agents. Each agent is given a broad prompt covering multiple sub-topics. The sglang agent, for example, is asked to investigate "SM120 (Blackwell RTX) GPU support, MoE inference optimization, FP4 GEMM improvements, and expert parallelism" all in one query. A more focused approach might yield deeper insights on any single topic, but the assistant has chosen breadth to get a quick survey of the entire landscape. This is a reasonable choice given the time constraints, but it means that some important details might be missed.

The assistant also assumes that the research agents will find and correctly interpret the most relevant information. The agents are autonomous and can browse the web, but they may not have deep domain expertise in GPU kernel optimization or the specific quirks of the GLM-5 architecture. Their summaries are generally accurate, but they may miss subtle interactions or draw incorrect conclusions from incomplete data.

Input Knowledge Required to Understand This Message

To fully appreciate what is happening in message 1066, a reader needs significant background knowledge spanning multiple domains.

First, one must understand the GLM-5 model architecture: a 744-billion parameter Mixture-of-Experts model with 256 experts per MoE layer, 8 activated per token, 80 layers, and FP4 block-scaled quantization. The model is enormous and requires careful parallelization across GPUs.

Second, one needs familiarity with NVIDIA's GPU architecture naming: SM120 refers to the Blackwell RTX architecture (RTX PRO 6000, RTX 5090), distinct from SM100 (Blackwell datacenter like B200) and SM90 (Hopper/Ada). The distinction matters because SM120 has different instruction sets (mma.sync rather than tcgen05) and shared memory limits (100KB per SM).

Third, one must understand the inference serving stack: SGLang as the serving framework, FlashInfer for attention and MoE kernels, CUTLASS for GEMM operations, and NCCL for inter-GPU communication. Each layer has its own optimization surface and compatibility constraints.

Fourth, one needs to know the history of this specific optimization campaign: the earlier benchmarks, the identified bottlenecks, the eleven improvement documents, and which approaches have already been tried and discarded.

Finally, one must understand the tooling model: the task tool spawns subagents that run independently, the read tool reads file contents, and all tool calls in a single round are dispatched in parallel with results returned in the next round.

Output Knowledge Created by This Message

Message 1066 produces several valuable outputs that feed into subsequent work.

The most concrete output is the three research summaries returned by the task agents. The sglang research reveals that the project has a Blackwell tracker issue, provides a dedicated Docker image, and has seen recent commits related to SM120 support and MoE optimization. The GLM-5 deployment research uncovers community discussions about memory optimization, batch sizing strategies, and known issues with the NVFP4 format. The SM120 FP4 kernel research provides a detailed survey of CUTLASS kernel paths, shared memory constraints, and optimization strategies for the specific hardware.

The message also produces an implicit prioritization of the improvement documents. By choosing to read documents 04, 05, 06, 07, and 11, the assistant signals that these are the next candidates for implementation. Documents 04 (Expert Parallelism) and 05 (Allreduce Fusion) are Tier 2 items that can be tested with relatively small code changes, while 06, 07, and 11 are Tier 3 items requiring significant engineering. The assistant is likely planning to start with the low-effort items and escalate to the engineering-heavy approaches only if needed.

Perhaps most importantly, the message creates alignment between the assistant and the user about the next steps. The user asked for continued implementation and research; the assistant has demonstrated that it is systematically gathering intelligence before committing to a specific implementation path. This builds trust and ensures that the user understands the rationale behind subsequent decisions.

The Thinking Process Visible in the Message Structure

While the assistant's internal reasoning is not explicitly shown (there is no separate "thinking" block in this message), the structure of the message itself reveals the thinking process.

The decision to read documents and launch research in the same round shows that the assistant views these as parallel, non-conflicting activities. It does not need to finish reading before starting research, because the research is independent of the document contents. This is a sophisticated understanding of task dependencies.

The selection of research topics reveals a layered analysis of the optimization problem. The sglang research targets the serving framework layer—are there new flags, configuration options, or code paths that could improve throughput? The GLM-5 research targets the model layer—are there deployment strategies specific to this architecture? The SM120 kernel research targets the hardware layer—what are the latest kernel implementations for this GPU? By covering all three layers, the assistant ensures that no optimization opportunity is missed due to a narrow focus.

The choice to read five documents but not all eleven shows prioritization thinking. The assistant is focusing on the documents that represent the frontier of untested ideas, skipping those that have already been tested or that are lower priority. This is a working-memory management strategy: the assistant cannot hold all eleven documents in context simultaneously, so it selects the most relevant subset.

The inclusion of full task results within the message (rather than summarizing them) shows that the assistant values transparency and completeness. The user can see exactly what the research agents discovered, without filtering or interpretation. This is particularly important for a technical audience that may want to verify claims or follow up on specific findings.

Conclusion

Message 1066 is a masterclass in systematic research orchestration. It demonstrates how an AI assistant can effectively parallelize information gathering across multiple independent dimensions, synthesize existing documentation with fresh intelligence, and set the stage for informed decision-making. The message is not about taking action—it is about ensuring that subsequent actions are based on the best available information.

In the context of the broader optimization campaign, this message represents a deliberate pause before a new wave of implementation. The assistant has been making rapid progress, achieving dramatic throughput improvements through server tuning and kernel optimization. But now it faces a set of harder problems that require deeper investigation. Rather than charging ahead with potentially suboptimal approaches, it invests time in research.

This is a pattern that distinguishes mature optimization work from naive hacking: the willingness to step back, gather intelligence, and let the research guide the implementation. The three research summaries returned in this message will directly inform the next several messages, as the assistant decides which improvement ideas to pursue, in what order, and with what specific techniques. The result is not just faster code, but a deeper understanding of why the code is fast—and that understanding is ultimately more valuable than any single benchmark number.