The Green Light: How a Three-Word Directive Transformed an AI's Plan Into Action

"Write down a plan/spec for each phase, then execute"

This seven-word message from the user, appearing at index 12214 in a sprawling technical conversation about GPU kernel optimization for speculative decoding, is deceptively brief. On its surface, it reads as a simple instruction. But within the context of the conversation—after thousands of words of investigation, analysis, and deliberation by the AI assistant—it functions as a decisive pivot point. It is the moment when planning ends and building begins.

The Context That Made This Message Necessary

To understand why this message was written, one must understand the conversation that preceded it. The assistant had been deep in a multi-session investigation of a performance bottleneck on the CT200 server—a machine running the Kimi K2.6 model with DFlash speculative decoding on RTX PRO 6000 Blackwell GPUs (sm_120 architecture). The bottleneck was severe: decode was achieving only ~14 GB/s effective memory bandwidth against a 1.8 TB/s peak, with GPU tensor core utilization at a mere 3%. The root cause had been traced to the DDTree verify attention kernel, which was locked to a Triton-based MLA implementation that couldn't exploit the hardware's capabilities.

In message [msg 12212], the assistant presented a comprehensive investigation report. It had discovered, through binary-level analysis with cuobjdump and file system searches, that every optimized MLA kernel (FlashMLA, cutlass-MLA, flashinfer-MLA) ships compiled exclusively for sm_90a/sm_100a/sm_103a architectures—the Hopper and high-end Blackwell datacenter GPUs. None support sm_120, the architecture of the RTX PRO 6000 Blackwell consumer GPU. Moreover, the libraries contain no embedded PTX code that could serve as a JIT fallback. The service only runs at all because Triton JIT-compiles its own kernels at runtime, but those Triton kernels are the source of the bandwidth starvation.

The assistant proposed a multi-phase plan and, crucially, asked the user a series of strategic questions: where should the optimized kernel land (custom SGLang backend vs. standalone library), how should the dense prefix MLA kernel be sourced (reuse existing code or write from scratch), how far should K/V defragmentation go, and which machine should be used for development.

The user answered decisively: custom SGLang backend, write from scratch (R2), Tier 0 + Tier 1 defrag, develop directly on CT200.

In message [msg 12213], the assistant synthesized these answers into a refined, execution-ready plan. It laid out five phases (Phase 0 through Phase 4), detailed guiding engineering choices (a single streaming kernel eliminating the need for a separate merge step, subclassing TritonAttnBackend to minimize risk, restructuring to KV-tile-outer/heads-inner to kill the 64× latent re-read), validation gates, risks, and deliverables. Then it ended with a question: "This is ready to execute. Want me to proceed... or adjust scope/sequencing first?"

The Message Itself: A Study in Compression

The user's response—"Write down a plan/spec for each phase, then execute"—is remarkable for what it communicates in so few words. It does three things simultaneously:

1. It affirms the plan. The user does not say "change the scope" or "I have concerns about X." The implicit message is that the assistant's refined plan is accepted.
2. It demands formalization. The assistant had presented the plan as a conversational artifact—embedded in a message with reasoning, hedging, and a trailing question. The user wants it elevated to a proper specification document: written down, structured, precise.
3. It commands execution. The word "then execute" is unambiguous. The assistant is told to stop asking for permission and start building. The planning phase is over. This is a classic engineering management move: when a subordinate presents a plan and asks "should I proceed?", the correct answer from a decisive leader is not "yes" but rather "document it formally, then proceed." The act of writing the spec forces clarity, exposes gaps, and creates an artifact that can be referenced during execution.

Assumptions Embedded in the Directive

The user's message makes several assumptions that are worth examining:

That the assistant can autonomously write a specification. The assistant is an AI, not a human engineer. The user assumes it has sufficient understanding of the codebase, the CUDA programming model, SGLang's attention backend architecture, and the DDTree algorithm to produce a coherent spec document. Given the assistant's demonstrated depth of analysis in the preceding messages—including binary-level investigation of compiled CUDA libraries—this assumption is well-founded.

That the phases are well-defined enough to specify. The assistant's plan had phases like "Phase 1 — sm_120 flash MLA verify kernel" with a brief description. The user assumes these are substantive enough to expand into a full specification. This is a bet on the assistant's ability to elaborate.

That execution can follow directly from specification. The user assumes a linear workflow: write the spec, then execute it. In practice, software development—especially GPU kernel development—is highly iterative. The spec will inevitably be revised as the assistant encounters real hardware constraints, numerical precision issues, and integration complexities. The user's directive implicitly accepts this, treating the spec as a living document rather than a frozen contract.

That the CT200 resource constraints are manageable. The assistant had noted that CT200's GPUs are nearly full (~94 GB used out of ~98 GB), leaving only ~4 GB headroom for development. The user's directive to "execute" on CT200 assumes this is sufficient for kernel development and testing. This assumption would be tested in the very next message, when the assistant begins exploring the repository and compiling code.

Input Knowledge Required to Understand This Message

A reader needs substantial context to grasp the significance of this seven-word message:

The technical problem: DDTree speculative decoding requires a verify attention kernel that can handle a custom tree mask. On sm_120 GPUs, all optimized MLA kernels are unavailable, forcing the system to use Triton's JIT-compiled implementation, which achieves only ~14 GB/s bandwidth.

The architectural constraint: sm_120 is an Ada-like architecture (not Hopper or full Blackwell-DC), meaning it lacks wgmma, TMA, and tcgen05 instructions. It has only ~100 KB of shared memory and relies on mma.sync + cp.async. This makes porting existing kernels impossible—they must be rewritten.

The integration strategy: The assistant plans to subclass TritonAttnBackend and override only the DDTree verify method, minimizing risk while keeping prefill (which already performs well at 50–90% tensor utilization) on Triton.

The defragmentation tiers: Tier 0 enables free-list sorting for better allocation contiguity; Tier 1 implements background relocation of live KV slots under a copy budget.

The conversation history: The user had already answered four strategic questions in a previous interaction, and the assistant had incorporated those answers into the refined plan.

Output Knowledge Created by This Message

The immediate output was the assistant's shift into execution mode. In message [msg 12215], the assistant responds: "I'm in build mode now. I'll write the per-phase spec doc, then execute starting with Phase 0/1. First let me ground the spec in the actual code — reading the existing kernel, C-ABI, harness, and build scripts."

It then begins exploring the repository structure, reading source files, and laying the groundwork for the specification. The user's directive transforms the assistant's behavior from analytical/reporting to constructive/building.

The downstream output, traced through the rest of the session, is substantial. The assistant goes on to:

• Write the formal plan document plans/0002-sm120-verify-kernel-defrag.md
• Implement the sm_120 flash MLA verify kernel in verify_attn_flash.cu
• Make the kernel CUDA graph capture-safe
• Achieve a 3–6× decode speedup over Triton across all context lengths
• Implement Tier 0 defragmentation via monkeypatching the allocator
• Identify MoE imbalance as the remaining bottleneck All of this flows from the user's directive to formalize and execute.

The Thinking Process Visible in the Assistant's Response

The assistant's reasoning in message [msg 12215] reveals how it interprets the user's directive. It immediately recognizes that "Write down a plan/spec for each phase, then execute" means it should stop deliberating and start building. The phrase "I'm in build mode now" is a conscious mental shift. It also recognizes that the spec must be grounded in actual code—it cannot be written from memory or abstract reasoning. Hence the first action is to read the existing source files: verify_attn.cu, verify_attn.cuh, the test harness, the build scripts, and the C API wrapper.

The assistant also demonstrates an understanding of the spec's purpose: it is not merely a documentation exercise but a tool for execution. The spec will define what each phase produces, what validation gates must be passed, and what risks must be managed. It will serve as the reference point for the entire implementation effort.

Conclusion

The message "Write down a plan/spec for each phase, then execute" is a masterclass in concise technical direction. In seven words, the user accomplishes what many managers struggle to do in paragraphs: affirm the plan, demand formalization, and command execution. It works because it arrives at precisely the right moment—after the assistant has done the investigative work, after the strategic decisions have been made, after the refined plan has been presented. Any earlier, and it would have short-circuited necessary analysis. Any later, and it would have allowed the assistant to stall in a loop of planning and permission-seeking.

The message also reveals something about the relationship between the user and the assistant. The user trusts the assistant's technical judgment enough to give a high-level directive without micromanaging the details. The assistant, in turn, understands that "write down a plan/spec" is not busywork but a necessary step in the engineering process—one that creates a shared artifact for tracking progress, validating decisions, and communicating with future readers of the codebase.

In the end, this seven-word message is the hinge on which the entire optimization effort turns. Before it, the conversation is about analysis and planning. After it, the conversation is about building, testing, and measuring. It is the moment when the team—user and assistant together—commits to a course of action and begins the hard work of making it real.