The Three-Word Steering Signal: How a User Prioritized Optimization Directions in a High-Performance ML Inference Session

Introduction

In the middle of an intensive optimization session for deploying the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a pivotal moment arrives. The assistant has just delivered a comprehensive diagnostic summary, laying out the performance bottlenecks in stark numerical detail: the GPUs are drawing only 235 watts out of a 600-watt TDP, the CUTLASS FP4 kernels are achieving merely 0.04% of theoretical peak during actual decode, and two critical larger tile configurations are crashing during initialization. The assistant then poses an open-ended question: "Want me to dig into any of these directions specifically?" The user's response is a masterclass in concise, high-bandwidth direction-setting:

Fix tiles, then try 2/3

This three-word message — supplemented by the numerical shorthand "2/3" — carries an enormous amount of decision-making weight. It prioritizes, sequences, and scopes the next phase of work in a single breath. Understanding why this message was written, what assumptions it encodes, and what knowledge it both consumes and produces reveals the essence of how human-AI collaboration functions in high-stakes technical optimization.

The Context: A Deep Diagnostic Culmination

To understand the user's message, one must first appreciate the context that produced it. The assistant had just completed an exhaustive investigation into why the GLM-5-NVFP4 model — a Mixture-of-Experts architecture with 256 experts and FP4 quantization — was underperforming on Blackwell GPUs. The investigation had uncovered several critical findings:

The compute-bound reality. Through a TP4+PP2 versus TP8 benchmark comparison, the assistant confirmed the model was compute-bound rather than communication-bound. This ruled out allreduce latency as the primary bottleneck and focused attention squarely on the FP4 GEMM kernel efficiency.

The power utilization gap. During inference, the GPUs were drawing only ~235W out of a 600W TDP — just 39%. This was actually lower than earlier measurements, suggesting something was throttling compute utilization even further.

The tile configuration dead end. The CUTLASS FP4 kernels on SM120 (Blackwell's compute architecture) only had working tile configurations of 128×128×128 and 128×128×256. The larger tiles — M128×N256 and M256×N128 — failed during initializeMoeGroupedGemm with an internal error. These larger tiles would have improved performance by processing larger blocks, increasing data reuse and compute density.

The fundamental small-batch problem. During decode, each of the 8 activated experts per token saw only ~32 tokens at 1024 concurrency. These tiny GEMMs (roughly 32×2048×6144) had an arithmetic intensity of approximately 16 ops/byte — about 72× below what's needed to be compute-bound on a device with 1,597 GB/s memory bandwidth and 1,850 TFLOPS dense FP4 peak.

The assistant synthesized these findings into three proposed directions for further investigation:

Fix the failing larger tiles — potentially improving small-batch GEMM efficiency by allowing better work distribution across SMs
Increase effective batch size per expert — through parameters like num_continuous_decode_steps or restructuring MoE dispatch
Try different kernel paths — specifically cuBLASLt, which reportedly worked better for small FP4 GEMMs on SM120 The assistant then asked the open-ended question: "Want me to dig into any of these directions specifically?"

The Message: A Decision in Three Words

The user's response — "Fix tiles, then try 2/3" — is remarkable for its density of meaning. Let us parse it carefully.

"Fix tiles" refers to direction #1: resolve the CUTLASS tile initialization failures for M128×N256 and M256×N128. This is prioritized first, suggesting the user believes that fixing the tiles is a prerequisite or enabler for the other approaches.

"then try 2/3" is the shorthand. "2/3" means "directions 2 and 3" — increase effective batch size per expert, and try cuBLASLt. The slash notation is a concise way of saying "both of the remaining options."

The sequencing is deliberate: fix first, then try. This implies a belief that fixing the tiles might change the baseline against which the other two approaches should be evaluated, or that the tile fix is lower-risk and should be attempted before branching into multiple parallel investigations.

Why This Message Was Written: Reasoning and Motivation

The user's motivation for writing this message stems from several factors:

Decision fatigue avoidance. The assistant presented three open-ended directions with no prioritization. Without guidance, the assistant might have pursued them in an arbitrary order, spent time analyzing trade-offs, or asked further clarifying questions. The user preempted this by providing a clear priority and sequence.

Bandwidth optimization. The user is operating at a high level of abstraction. They have absorbed the assistant's detailed analysis and synthesized it into a simple action plan. By communicating in this compressed form, they maximize the assistant's autonomy while maintaining strategic control.

Confidence in the diagnosis. The user accepts the assistant's framing of the three directions as the correct set of options. There is no request for additional data, no skepticism about the analysis, and no suggestion of alternative approaches. This implies trust in the assistant's diagnostic work and a shared mental model of the problem space.

Urgency. The session has been running for many rounds across multiple segments. The user wants to maintain momentum. A long, deliberative response would slow things down. The three-word response keeps the pace high.

How Decisions Were Made: The Implicit Prioritization Framework

The user's decision to prioritize tile fixing first reveals an implicit reasoning framework:

Prerequisite ordering. Fixing tiles may be necessary before the other approaches can be properly evaluated. If larger tiles enable better work distribution across SMs, then the effective batch size per expert (direction 2) might already improve without any other changes. Similarly, if cuBLASLt (direction 3) is tested before tiles are fixed, the comparison would be against a suboptimal CUTLASS baseline.

Risk ordering. Tile fixing is a surgical intervention — modifying or patching specific kernel configurations. It has a well-defined scope and clear success criteria (the tiles initialize without error). Directions 2 and 3 are broader: direction 2 involves server parameter tuning which could have unpredictable interactions, and direction 3 involves switching kernel backends which could introduce new failure modes. The user may be applying a "fix what's broken first, then optimize" heuristic.

Information value. Fixing the tiles first generates information that informs the subsequent approaches. If the tiles fix dramatically improves performance, directions 2 and 3 may become less important or may need to be re-evaluated against a new baseline.

Assumptions Made by the User

The user's message rests on several assumptions:

That the tiles are fixable. The assistant had traced the failure to a shared memory limitation — the larger tiles required more than the 99KB opt-in shared memory per block on SM120. The user assumes this can be worked around, perhaps by reducing pipeline stages, adjusting tile dimensions, or modifying the CUTLASS kernel schedule. This is not guaranteed.

That the three directions are independent. The user assumes that fixing tiles, increasing batch size, and trying cuBLASLt are separable efforts that can be pursued sequentially. If they interact in complex ways (e.g., cuBLASLt doesn't support the larger tiles), the sequencing could waste effort.

That directions 2 and 3 are worth trying. The user does not question whether these are the right directions. They accept the assistant's framing as complete and correct. There is no request for evidence that cuBLASLt is actually faster for small FP4 GEMMs on SM120, or that increasing batch size is feasible within the KV cache budget.

That "2/3" is unambiguous. The user assumes the assistant will correctly parse "2/3" as "directions 2 and 3" rather than, say, "two-thirds of something" or "option 2 out of 3." In the context of the conversation, this is a safe assumption, but it's worth noting how much rests on shared context.

Input Knowledge Required to Understand This Message

To parse the user's three words correctly, the assistant (and any reader of the conversation) needs:

Knowledge of the three numbered directions. The assistant had just listed them in the previous message. Without that context, "Fix tiles, then try 2/3" would be incomprehensible.

Knowledge of what "tiles" means in this context. The user is referring to CUTLASS GEMM tile configurations — the M×N×K dimensions that determine how matrix multiplication work is partitioned across thread blocks. This requires understanding of GPU kernel programming, CUTLASS, and the specific SM120 architecture.

Knowledge of the failure mode. The user knows that the M128×N256 and M256×N128 tiles fail during initialization, and that this is a shared memory limitation. The message assumes this shared understanding.

Knowledge of the optimization landscape. The user understands that FP4 GEMM efficiency is the primary bottleneck, that per-expert batch sizes are tiny during decode, and that different kernel backends (CUTLASS vs. cuBLASLt) may have different performance characteristics for small matrices.

Output Knowledge Created by This Message

The message creates several forms of knowledge:

A prioritized action plan. The assistant now has a clear sequence of work: fix tiles first, then investigate directions 2 and 3. This eliminates decision overhead and allows the assistant to proceed autonomously.

A scope boundary. The user implicitly approves the three directions as the complete set of next steps. The assistant should not, for example, start investigating expert parallelism or other approaches not in the list.

A success criterion. The tile fix has a clear success criterion: the larger tiles initialize without error. The assistant can report back when this is achieved and then proceed to the next phase.

A communication protocol. The user has demonstrated that short, high-bandwidth responses are acceptable and expected. This sets the tone for the remainder of the session.

The Thinking Process Visible in the Message

While the user's message is only three words, the thinking process behind it is visible through what is not said:

The user did not ask for more information. They did not request clarification on why the tiles fail, how difficult the fix might be, or what the expected performance impact would be. This suggests the user either already understands these details (having followed the assistant's analysis) or trusts the assistant to figure them out.

The user did not suggest alternatives. They did not propose a fourth direction, question the assistant's analysis, or express doubt about any of the three options. This suggests alignment with the assistant's diagnostic conclusions.

The user did not specify how to fix the tiles. They did not prescribe a particular approach — modifying shared memory allocation, reducing pipeline stages, patching the CUTLASS kernel schedule, or any other technique. This delegates the technical implementation to the assistant while retaining strategic control over priorities.

The user used the shorthand "2/3" rather than spelling out the directions. This reveals a comfort with compressed communication and an expectation that the assistant can resolve the reference. It also implies a mental model where the three directions are a numbered list in shared working memory.

What Happened Next

The assistant immediately acted on the user's direction. In the following message ([msg 906]), it acknowledged the plan by updating its todo list: "Fix failing CUTLASS M128x256 and M256x128 tiles for SM120" was marked as in progress, and directions 2 and 3 were set to pending. The assistant then began investigating the tile initialization failure by examining the CUTLASS launcher code, tracing the error to a shared memory allocation failure during gemm.initialize().

Over the subsequent messages, the assistant confirmed that the SM120 architecture has 99KB of opt-in shared memory per block, and that the larger tiles exceed this limit. This led to a deeper investigation into whether the tile configurations could be modified, the pipeline stages reduced, or the kernel schedule changed to fit within the shared memory budget.

Conclusion

The message "Fix tiles, then try 2/3" is a textbook example of effective human-AI collaboration in a technical domain. It demonstrates how a user can maintain strategic control over a complex optimization session while delegating tactical implementation to an AI assistant. The message's brevity is not a sign of low effort but of high-bandwidth communication built on shared context, mutual trust, and a well-developed mental model of the problem space. In just three words and a numerical shorthand, the user sequenced work, set priorities, defined scope, and launched the next phase of investigation — a remarkable compression of decision-making that would require paragraphs to express in a less collaborative setting.