The Art of Waiting Productively: Parallel Research in a High-Stakes ML Optimization Session

Introduction

In the middle of an intense optimization session for the GLM-5-NVFP4 large language model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, the AI assistant encounters a brief but revealing moment. Message 927 of the conversation captures a single action: the assistant dispatches a research subagent to investigate cuBLASLt FP4 integration while a newly tuned server instance starts up in the background. The message reads:

It's starting. While waiting, let me research cuBLASLt FP4 integration in parallel:

This simple utterance reveals a sophisticated meta-cognitive awareness of time, parallelism, and resource management. The assistant is not merely executing commands sequentially; it is actively managing a workflow where multiple threads of investigation proceed concurrently, using blocking operations as opportunities for productive parallel exploration.

The Immediate Context: A Server Restart in Progress

To understand why this message was written, we must trace the events immediately preceding it. In [msg 925], the assistant had just calculated that the KV cache could support approximately 1,935 concurrent requests (495,488 tokens of KV cache capacity, with each request consuming roughly 256 tokens for 128 input + 128 output). Based on this calculation, the assistant decided to restart the SGLang server with two critical parameter changes: --max-running-requests 2048 (increasing the number of concurrent requests from the previous setting) and --num-continuous-decode-steps 8 (batching multiple decode steps together before re-scheduling).

The server restart was initiated by killing the existing process (pkill -9 -f sglang), writing a new startup script (/root/run_tp8_tuned.sh), and launching it with nohup. In [msg 926], the assistant checked on the restart and found the server process running (PID 64723), but also saw evidence of the old server being killed ("60043 Killed python3 -u -m sglang.launch_server"). The new server was in its initialization phase — loading the model, allocating KV cache, and warming up.

This is the critical moment captured in message 927. The assistant knows the server will take some time to initialize. Rather than idly polling for readiness, it proactively spawns a parallel research thread to investigate an optimization hypothesis that had been lingering: whether cuBLASLt FP4 could outperform the CUTLASS-based kernels currently in use.

The Research Hypothesis: cuBLASLt as a CUTLASS Alternative

The decision to research cuBLASLt FP4 integration was not arbitrary. It was motivated by a specific finding from earlier in the session. In [msg 913] through [msg 922], the assistant had conducted a deep investigation into FP4 GEMM kernel efficiency on the SM120 architecture (Blackwell). The findings were sobering:

  1. The GPUs were drawing only ~235W out of their 600W TDP during inference — less than 40% of their thermal budget.
  2. CUTLASS kernels plateaued at approximately 1,300 TFLOPS (about 70% of the dense peak) only for very large matrices.
  3. During actual decode operations, per-expert batch sizes of roughly 16–64 tokens achieved merely 0.8–55 TFLOPS — between 0.02% and 3% of the theoretical peak. This massive underutilization suggested that either the CUTLASS kernels were poorly optimized for SM120's FP4 path, or there was a fundamental architectural limitation. The assistant had also discovered that the 99KB shared memory limit on SM120 prevented using larger CUTLASS tile configurations — M128×N256 and M256×N128 tiles failed to initialize because they required more shared memory than available. The cuBLASLt hypothesis was attractive because NVIDIA's proprietary library is known to contain hand-tuned kernels for specific architectures, and the assistant had heard reports of cuBLASLt achieving 99.6 TFLOPS on GB10 (SM121) — a closely related Blackwell variant. If cuBLASLt could achieve similar performance on SM120, it might bypass the CUTLASS tile size limitations entirely.

The Task Tool: Parallel Subagent Execution

The mechanism used to conduct this parallel research is the task tool — a feature of the opencode framework that spawns a subagent session. The subagent runs as an independent multi-round conversation, with full access to the remote machine via SSH, and returns its findings as a task_result block. Multiple task tool calls in the same round run their subagents in parallel, but the parent session remains blocked until all subagents complete.

The task prompt provided to the subagent was detailed and structured:

On the remote machine ssh root@10.1.230.174, research how to use cuBLASLt for FP4 GEMMs instead of CUTLASS. cuBLASLt reportedly achieves much better FP4 performance on SM120 (99.6 TFLOPS on GB10 SM121).

>

1. Check if cuBLASLt FP4 is accessible via torch._scaled_mm or directly through ctypes bindings to cublasLt. 2. Benchmark cuBLASLt FP4 GEMMs across a range of matrix sizes relevant to MoE decode (M=8,16,32,64,128,256; K=N=7168). 3. Compare against the current CUTLASS-based FlashInfer path. 4. Determine if cuBLASLt supports grouped GEMM for MoE (multiple small matrices batched together).

This prompt reveals several assumptions and strategic choices. First, the assistant assumes that cuBLASLt FP4 is accessible through PyTorch's torch._scaled_mm API, which would provide a clean integration path. Second, it focuses on matrix sizes relevant to Mixture-of-Experts (MoE) decode — the model has 256 experts with 8 active per token, so per-expert batch sizes are small (M=8 to M=256). Third, it explicitly asks about grouped GEMM support, which is critical for MoE efficiency — without it, cuBLASLt would need to launch separate kernels for each expert, incurring significant launch overhead.

The Result: A Dead End, But Valuable Knowledge

The task result, returned inline within the same message, provides a comprehensive analysis. The key findings were:

  1. cuBLASLt FP4 IS available on SM120 via torch._scaled_mm, confirming the accessibility assumption.
  2. However, cuBLASLt was 5-8% slower than CUTLASS for large matrix sizes — directly contradicting the hypothesis that it would be faster.
  3. cuBLASLt does not support grouped GEMM for MoE, meaning it would need separate kernel launches per expert, which would add significant overhead for the small batch sizes characteristic of MoE decode. This finding was immediately actionable. In the very next message ([msg 928]), the assistant updated its todo list to mark "cuBLASLt FP4 — NOT BETTER than CUTLASS on SM120 (5-8% slower for large M, no grouped GEMM for MoE)" as completed with a "high" priority. The investigation had conclusively ruled out cuBLASLt as a viable optimization path, saving the assistant from pursuing a dead end further.

Input Knowledge Required

To fully understand this message, several pieces of background knowledge are necessary:

  1. The SM120 architecture: Blackwell's compute architecture with 99KB shared memory per SM, FP4 tensor core support, and the specific limitations discovered earlier in the session.
  2. The GLM-5-NVFP4 model architecture: A Mixture-of-Experts model with 256 experts, 8 active per token, using FP4 block-scaled quantization.
  3. The CUTLASS vs. cuBLASLt distinction: CUTLASS is NVIDIA's open-source template library for GEMM kernels, while cuBLASLt is a proprietary library with potentially more optimized implementations. The assistant is exploring whether the proprietary path offers advantages.
  4. The SGLang serving framework: Specifically, the server parameters being tuned (--max-running-requests, --num-continuous-decode-steps) and the MoE runner backend (flashinfer_cutlass).
  5. The opencode task tool: The mechanism for spawning parallel subagent investigations, which allows the assistant to decompose work across multiple reasoning threads.

Output Knowledge Created

This message produces several forms of output knowledge:

  1. A definitive benchmark comparison between cuBLASLt FP4 and CUTLASS FP4 on SM120 hardware, across matrix sizes relevant to MoE inference.
  2. Confirmation that cuBLASLt lacks grouped GEMM support for FP4, which is a critical capability for efficient MoE execution.
  3. A ruling-out of cuBLASLt as an optimization path, allowing the assistant to redirect effort toward more promising approaches (expert parallelism, piecewise CUDA graphs, MSCCLPP allreduce, etc.).
  4. A demonstration of the parallel task decomposition pattern — using blocking operations as opportunities for parallel research, which is a meta-cognitive workflow optimization.

Assumptions and Their Validity

The assistant made several assumptions in dispatching this research task:

Assumption 1: cuBLASLt FP4 would be accessible through PyTorch. This assumption was validated — torch._scaled_mm does support FP4 on SM120.

Assumption 2: cuBLASLt might outperform CUTLASS on SM120. This assumption was falsified — cuBLASLt was 5-8% slower for large matrices, and lacked grouped GEMM support.

Assumption 3: The server restart would take enough time to justify parallel research. This was validated — the server initialization (loading the 8-shard model, allocating KV cache, warming up CUDA graphs) takes on the order of minutes, providing ample time for the subagent investigation.

Assumption 4: The subagent could independently conduct the research. This was validated — the subagent successfully SSH'd into the remote machine, ran benchmarks, and returned structured findings.

The Thinking Process: Meta-Cognitive Workflow Management

The most interesting aspect of this message is what it reveals about the assistant's internal reasoning. The phrase "It's starting. While waiting, let me research..." demonstrates a sophisticated awareness of time as a resource. The assistant recognizes that:

  1. The server restart is a blocking operation that will take non-trivial time.
  2. Rather than polling for completion (which would waste reasoning cycles), it can offload a parallel investigation.
  3. The cuBLASLt question is a high-value investigation that can be conducted independently.
  4. If the investigation returns positive results, the assistant can immediately act on them when the server is ready.
  5. If the investigation returns negative results (as it did), the assistant has saved time by ruling out a dead end without interrupting the server startup. This is essentially a form of temporal parallelism — using idle time in one workflow to advance another workflow. It mirrors the concept of "meanwhile" in fiction, where multiple storylines progress simultaneously.

Conclusion

Message 927 of this opencode session is a microcosm of the entire optimization effort. It captures a moment where the assistant, faced with a blocking operation, chooses to invest that time in parallel research. The research itself — investigating cuBLASLt FP4 as a CUTLASS alternative — represents a systematic approach to optimization: form a hypothesis, test it rigorously, and either adopt or discard it based on evidence.

The result was a dead end, but a valuable one. The assistant learned that cuBLASLt offers no advantage on SM120 for this workload, and more importantly, that the bottleneck is not in the GEMM library choice but in the fundamental architecture of how MoE decode interacts with Blackwell's shared memory constraints. This knowledge shaped the subsequent optimization strategy, which focused on increasing per-expert batch sizes through server parameter tuning — an approach that ultimately yielded a 28% throughput improvement.

In the broader narrative of the session, this message represents a pivot point. The assistant had exhausted the "easy" optimizations (tile size selection, library choice) and was transitioning to more sophisticated approaches (expert parallelism, piecewise CUDA graphs, kernel fusion). The cuBLASLt investigation was the last check on a promising hypothesis before committing to the harder path.