The Dead End Inventory: How a Todo List Became a Turning Point in Blackwell Optimization

The Message

In a coding session spanning dozens of rounds across multiple days, one message stands out not for its complexity, but for its finality. The assistant writes:

[todowrite] {"todos":[{"content":"Flashinfer allreduce fusion SM120 — DEAD END (JIT doesn't support SM120)","priority":"high","status":"completed"},{"content":"Custom allreduce PCIe — DEAD END (38 tok/s, PCIe bus contention)","priority":"high","status":"completed"},{"content":"NCCL Tree — DEAD END (CUDA graph incompatible)","priority":"high","status":"completed"},{"content":"Torch symmetric memory — DEAD END (SM120 not supported...")]

This is not a dramatic error message or a breakthrough result. It is a todo list update — a todowrite tool call that marks four high-priority optimization approaches as "completed" with the grim parenthetical "DEAD END" appended to each. Yet this seemingly mundane act of project management represents a critical inflection point in a long and grueling optimization campaign.

Context: The Optimization Campaign

To understand why this todo list matters, one must understand what led to it. The assistant and user had been working for days to deploy the Kimi-K2.5 model (a massive Mixture-of-Experts language model) on a server with eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 — without NVLink. This hardware configuration is inherently challenging for distributed inference: the GPUs are powerful (Blackwell architecture, SM120 compute capability), but they communicate over PCIe rather than the high-speed NVLink interconnects that datacenter GPUs typically use.

The central problem was EAGLE-3 speculative decoding. EAGLE-3 is a technique where a small "draft" model generates candidate tokens, and the large "target" model verifies them in parallel. In theory, this should speed up inference. In practice, on this system, EAGLE-3 was producing only 54.1 tokens per second — far below the 89.5 tok/s baseline of running the target model alone. The bottleneck was the "verify" pass: every cycle required the target model to perform a forward pass across all 8 GPUs, which involved approximately 122 NCCL allreduce operations taking about 30 milliseconds. This communication overhead swamped any gains from speculative decoding.

The assistant had been systematically hunting for ways to reduce this verify cost. The approaches tested read like a tour of cutting-edge ML infrastructure techniques:

1. FlashInfer allreduce fusion: Fuse multiple allreduce operations into a single kernel to reduce launch overhead. Failed because FlashInfer's JIT compiler does not support SM120 (Blackwell architecture).
2. Custom allreduce kernel: A specialized allreduce implementation. When forced to work over PCIe, it produced only 38 tok/s — more than 2× slower than NCCL — due to massive PCIe bus contention from the all-to-all communication pattern.
3. NCCL Tree algorithm: An alternative NCCL collective algorithm that can reduce communication volume. Failed because it was incompatible with CUDA graph capture, which SGLang relies on for performance.
4. Torch symmetric memory: A PyTorch feature for GPU memory optimization. Failed because SM120 is not in its architecture lookup table.
5. Expert Parallelism with FlashInfer A2A: Distribute expert modules across GPUs to reduce per-GPU computation. Failed with assertion errors and out-of-memory conditions. Each of these failures was carefully documented, and each consumed significant time to test — setting up configurations, launching servers, waiting for model loading (which took 10+ minutes), running benchmarks, and analyzing results.## Why This Message Matters The todo list update at message 5251 is the moment where the assistant formally acknowledges that an entire family of optimization approaches — allreduce optimization — has been exhausted. The word "DEAD END" appears four times, each one representing a hypothesis that was tested and falsified. This is the scientific method in action within an engineering context: form a hypothesis (e.g., "FlashInfer's allreduce fusion will reduce verify time"), design an experiment (configure the server with --moe-a2a-backend flashinfer), run it (wait 10+ minutes for model loading), collect data (crash, OOM, or poor throughput), and conclude (DEAD END). But the message is not merely a graveyard of failed ideas. It is also a strategic pivot point. By formally closing out these four approaches, the assistant creates the intellectual space for a new direction. Indeed, in the messages immediately following this todo update ([msg 5252] and beyond), the assistant pivots to an entirely different strategy: upgrading CUDA from version 12.8 to version 13.1, which has native SM120 support and could unblock all the Blackwell-specific optimizations that were previously unavailable.

The Reasoning Process

The todo list reveals a sophisticated reasoning process. Each entry is structured as a claim followed by evidence in parentheses:

• "Flashinfer allreduce fusion SM120 — DEAD END (JIT doesn't support SM120)": The root cause is identified precisely — not a generic failure but a specific architectural incompatibility.
• "Custom allreduce PCIe — DEAD END (38 tok/s, PCIe bus contention)": A quantitative result (38 tok/s) with a diagnosed cause (PCIe bus contention). This is more informative than simply saying "it was slow."
• "NCCL Tree — DEAD END (CUDA graph incompatible)": A technical incompatibility that was discovered through testing.
• "Torch symmetric memory — DEAD END (SM120 not supported)": Another architecture support gap. The assistant is not just tracking task status; it is building a knowledge base. Each entry captures the why of the failure, which is essential for future decision-making. If someone later asks "should we try custom allreduce on PCIe?" the answer is immediately available: it was tried, it produced 38 tok/s (2× slower than NCCL), and the cause was PCIe bus contention.

Assumptions and Their Consequences

The optimization campaign operated under several assumptions, some of which proved incorrect:

Assumption 1: Blackwell (SM120) support in popular ML libraries. The assistant assumed that FlashInfer, PyTorch's torch.symmetric_memory, and other tools would support the new Blackwell architecture. This assumption was wrong — FlashInfer's JIT compiler did not support SM120, and torch.symmetric_memory's architecture lookup table did not include it. These gaps are understandable given that Blackwell GPUs were relatively new at the time, but they forced the assistant to abandon otherwise promising approaches.

Assumption 2: PCIe-connected GPUs can benefit from the same allreduce optimizations as NVLink-connected GPUs. The custom allreduce kernel, which may work well on systems with fast interconnects, performed disastrously on PCIe (38 tok/s vs NCCL's ~82+ tok/s). The all-to-all communication pattern required for allreduce over PCIe created bus contention that swamped any kernel-level optimizations.

Assumption 3: Expert Parallelism would reduce memory pressure. In fact, EP increased memory usage — each GPU needed to hold more expert parameters or additional dispatch buffers. With EP, available memory after weight loading dropped from 21.7 GB to 16.3 GB per GPU, which contributed to OOM failures.

Assumption 4: The auto-detected mem_fraction_static would work for EAGLE-3. The assistant discovered that the draft model added extra memory requirements that the auto-detection didn't account for, requiring manual override to 0.88.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

• CUDA architecture naming: SM120 refers to the Blackwell GPU architecture. Knowing that FlashInfer's JIT compiler doesn't support SM120 requires understanding that JIT compilation targets specific architecture versions.
• NCCL and collective communication: Allreduce is a collective operation that sums tensors across all GPUs. NCCL Ring, Tree, and custom kernels are different algorithms for implementing this operation, each with different performance characteristics on different hardware topologies.
• PCIe vs NVLink: PCIe is a general-purpose I/O bus with limited bandwidth and high latency compared to NVLink, which is a GPU-dedicated high-speed interconnect. This distinction is critical for understanding why certain optimizations fail on PCIe systems.
• CUDA graphs: A feature that allows a sequence of GPU operations to be captured and replayed as a single unit, reducing launch overhead. Some NCCL algorithms are incompatible with CUDA graph capture.
• Speculative decoding and EAGLE-3: The draft-verify paradigm where a small model generates candidates and a large model verifies them. The "verify pass" is the bottleneck being attacked.
• SGLang server configuration: The various flags like --cuda-graph-max-bs, --disable-custom-all-reduce, --moe-a2a-backend, and --mem-fraction-static all have specific meanings in the SGLang inference framework.

Output Knowledge Created

This message creates several forms of knowledge:

1. A decision record: The todo list serves as documentation of which approaches were tried and why they failed. This is invaluable for anyone who might revisit these optimizations later.
2. A boundary condition: The message establishes that for 8× RTX PRO 6000 Blackwell GPUs connected via PCIe, NCCL Ring is the best available allreduce strategy. No alternative approach tested could match it.
3. A strategic signal: By marking all four approaches as dead ends, the message implicitly signals that a different class of solution is needed — which turns out to be upgrading CUDA to version 13 to access Blackwell-native optimizations.
4. A baseline for future work: The 89.5 tok/s baseline (achieved by reducing --cuda-graph-max-bs from 512 to 128) and the 54.1 tok/s EAGLE-3 result provide quantitative reference points for any future optimization attempts.

The Thinking Process

The assistant's thinking process, visible across the messages leading up to this todo update, follows a systematic pattern:

1. Hypothesis generation: "What if FlashInfer's allreduce fusion reduces verify time?"
2. Experiment design: Configure the server with the relevant flags, launch it, wait for it to load.
3. Result collection: Check logs for errors, benchmark throughput.
4. Analysis: Compare against baseline, identify root cause of failure.
5. Documentation: Update the todo list and the optimization plan document. This cycle repeated for each of the four approaches. The assistant also demonstrated the ability to serendipitously discover improvements — the --cuda-graph-max-bs 128 finding came from testing EP configurations, not from a deliberate optimization of baseline throughput. The todo list itself reveals a meta-cognitive awareness: the assistant is not just executing tasks but maintaining a structured record of what has been learned. This is the mark of a mature engineering approach, where the process of elimination is as valuable as the positive results.

Conclusion

Message 5251 is a todo list update that functions as a scientific paper in miniature. It reports four experiments, each with a clear hypothesis, methodology, result, and conclusion. The results are negative — every approach was a dead end — but the knowledge created is positive: the assistant now knows exactly which optimizations do not work on this hardware, and why. This clears the path for the next phase of the campaign: upgrading CUDA to version 13 to access Blackwell-native optimizations that were previously blocked by software compatibility gaps.

In the broader narrative of the coding session, this message marks the transition from "optimize the allreduce" to "upgrade the software stack to match the hardware." It is a reminder that sometimes the most productive thing you can do is systematically eliminate bad options, so the good ones become visible.