The Baseline Checkpoint: Recovering from a Failed Allreduce Fusion Experiment on SM120

Introduction

In the course of a marathon optimization session for GLM-5-NVFP4 inference on 8× RTX PRO 6000 Blackwell GPUs, message [msg 805] arrives as a quiet but critical inflection point. After a bold but failed attempt to enable FlashInfer's allreduce fusion on the SM120 architecture—an experiment that cratered throughput from ~1,867 tok/s to a dismal 236 tok/s and dropped GPU power draw to a mere 125W—the assistant has reverted its changes, restarted the server, and now issues this message to confirm that the baseline is restored. The message is deceptively short: two shell commands, one launched in the background, and a table of nvidia-smi readings. But beneath this sparse surface lies a dense layer of engineering reasoning—a deliberate checkpoint after failure, a parallel monitoring strategy, and a quiet celebration that the recovery worked. This article examines message [msg 805] in depth: why it was written, what decisions it embodies, what assumptions it carries, and what knowledge it creates for the ongoing optimization effort.

The Context of Failure and Recovery

To understand message [msg 805], one must understand what immediately preceded it. In messages [msg 780] through [msg 801], the assistant had identified that FlashInfer's allreduce fusion kernels explicitly excluded SM120 (the architecture of the RTX PRO 6000) via a __CUDA_ARCH__ >= 900 && __CUDA_ARCH__ < 1200 gate. The assistant patched this gate to include SM120, cleared the JIT cache, and restarted the server with --enable-flashinfer-allreduce-fusion. The server started successfully—the JIT compilation worked, the IPC handles were allocated across all 8 ranks—but when benchmarked, the results were catastrophic: 236 tok/s total throughput and GPU power draw of only ~125W per card, compared to ~1,867 tok/s and ~250W before the fusion was enabled.

The assistant correctly diagnosed the problem in [msg 801]: the cudaGridDependencySynchronize() primitive used in the allreduce kernel likely behaves differently or is unsupported on SM120, causing excessive serialization or deadlock-like stalls. The assistant then reverted the changes in [msg 802], removing SM120 from the architecture support checks in both communicator.py and server_args.py, and restarted the server without the fusion flag in [msg 803]. By [msg 804], the server was back up—the restart was fast thanks to cached autotune results from the previous run.

Message [msg 805] is the next logical step: verify the recovery.

Why This Message Was Written

The primary motivation for [msg 805] is validation. The assistant has just performed a surgical revert of a failed experiment, and needs to confirm three things before proceeding:

  1. The server is actually serving requests correctly. Running a benchmark of 512 prompts with 256 input tokens and 128 output tokens at infinite request rate will stress the server and reveal any lingering issues from the revert—corrupted state, misconfigured parameters, or residual JIT cache problems.
  2. GPU utilization and power have returned to expected levels. The power draw of ~125W during the failed allreduce fusion run was a clear red flag—it indicated the GPUs were barely working. The assistant needs to see power draw return to the ~250W+ range that characterized the healthy baseline. The fact that the assistant checks power during the benchmark (after a 30-second sleep) rather than waiting for the benchmark to finish shows a desire for real-time visibility into hardware behavior.
  3. The baseline is solid enough to serve as a jumping-off point for the next optimization attempt. The assistant's todo list in [msg 799] already includes "Run benchmark sweep with allreduce fusion" as in-progress, but after the fusion failure, the plan needs to pivot. Before pivoting, the assistant must know exactly where it stands. There is also a subtle motivational layer: the message opens with "Good, quick restart thanks to cached autotune." This is the assistant acknowledging a small win—the autotune cache from the failed fusion run survived the revert and sped up the restart. In a long optimization session filled with setbacks (NaN crashes, PCIe bottlenecks, virtualization issues), small wins matter for maintaining momentum.

The Parallel Monitoring Strategy

The message executes two commands in parallel: a benchmark launched in the background with &, and a power check after a 30-second sleep. This is a deliberate design choice, not an accident. The assistant could have run the benchmark first and checked power afterward, but that would miss the transient behavior during peak load. By overlapping them, the assistant gets a snapshot of GPU power draw while the server is under maximum stress from 512 concurrent requests.

The choice of 512 prompts (double the 256 used in the previous benchmark at [msg 800]) is also significant. The assistant is stress-testing more aggressively this time, perhaps to ensure the revert didn't introduce any scaling issues. The --request-rate inf flag ensures the server is saturated, giving a true measure of peak throughput and power.

The nvidia-smi query is also more detailed than before. In [msg 800], the assistant queried only utilization.gpu and power.draw. Here, it adds clocks.current.sm, revealing the SM clock speeds of 2280–2325 MHz. This additional metric gives insight into whether the GPUs are thermally throttling or running at their boost clocks. The fact that all 8 GPUs are within a narrow range (2280–2325 MHz) and at 97–98% utilization suggests healthy operation.

What the Results Reveal

The nvidia-smi output in [msg 805] shows:

| GPU | Utilization | Power Draw | SM Clock | |-----|-------------|------------|----------| | 0 | 97% | 326.62 W | 2295 MHz | | 1 | 98% | 333.66 W | 2310 MHz | | 2 | 98% | 318.89 W | 2295 MHz | | 3 | 98% | 327.91 W | 2325 MHz | | 4 | 98% | 338.03 W | 2317 MHz | | 5 | 98% | 333.00 W | 2317 MHz | | 6 | 98% | 334.01 W | 2317 MHz | | 7 | 98% | 317.38 W | 2280 MHz |

These numbers are significantly better than the pre-fusion baseline of ~250W. Power draw has jumped to ~317–338W per GPU—an improvement of roughly 30%. This is unexpected and important. The assistant did not change any performance-critical parameters between the pre-fusion run and this recovery; the only difference is that the server was restarted. Several hypotheses could explain the improvement:

Assumptions Embedded in the Message

Message [msg 805] rests on several assumptions, some explicit and some implicit:

The benchmark will produce valid results. The assistant assumes that running bench_serving with 512 prompts at infinite request rate will complete successfully and produce throughput numbers comparable to earlier runs. The benchmark is launched in the background, and the assistant does not wait for its output before checking power—it trusts that the benchmark is running correctly.

Power draw is a proxy for performance. The assistant has been treating GPU power draw as a key health metric throughout the session. While higher power generally correlates with higher throughput for compute-bound workloads, the relationship is not linear, and power can be high even when the workload is inefficient (e.g., due to memory stalls or synchronization overhead). The assistant implicitly assumes that ~320W is "good" and ~125W was "bad," which is reasonable but not rigorous.

The revert was complete. The assistant reverted two files (communicator.py and server_args.py) but did not revert the change to the FlashInfer header file trtllm_allreduce.cuh (the __CUDA_ARCH__ gate patch from [msg 781]). That header file still has the SM120-inclusive gate. However, since the server is not using --enable-flashinfer-allreduce-fusion, the allreduce kernels are never invoked, so the patched header is dormant. The assistant is assuming this dormant patch causes no harm—a reasonable assumption, but one that could bite later if a different code path triggers the allreduce kernel.

Cached autotune is safe. The assistant celebrates the "cached autotune" that sped up the restart, but this cache was built during the failed fusion run, which included profiling of allreduce fusion kernels that may have left behind suboptimal or corrupted tuning records. The assistant assumes the cache is benign for the non-fusion path.

Input Knowledge Required

To fully understand [msg 805], a reader needs awareness of:

Output Knowledge Created

Message [msg 805] produces several pieces of actionable knowledge:

  1. The server is healthy post-revert. No crashes, no errors, the server is serving requests at high utilization.
  2. GPU power draw has improved to ~320–338W. This is a meaningful improvement over the ~250W baseline, suggesting that the server restart or autotune cache maturation has unlocked additional GPU utilization. This is new information—the assistant did not expect this improvement.
  3. SM clocks are at 2280–2325 MHz across all 8 GPUs. This confirms the GPUs are running at or near their boost clocks, with no thermal throttling. The narrow distribution (45 MHz spread) suggests good thermal balance across the cards.
  4. The baseline is re-established. The assistant can now proceed to alternative optimization strategies (NCCL tuning, MoE backend switching, overlap scheduling, continuous decode steps) with confidence that the starting point is solid.
  5. The allreduce fusion path on SM120 is a dead end (for now). The failed experiment and successful revert confirm that the FlashInfer allreduce fusion kernels are incompatible with SM120's synchronization primitives. This is a durable piece of knowledge that will inform future architecture decisions.

The Thinking Process Visible in the Message

Although the message itself is short, the assistant's reasoning is visible in its structure and choices:

"Good, quick restart thanks to cached autotune." — The assistant is processing the restart time as a signal. A fast restart means the JIT cache survived the revert, which means the autotuner's work on the MoE kernels (separate from the allreduce fusion kernels) is preserved. This is a relief—the assistant was worried the failed fusion experiment might have corrupted the cache.

Running the benchmark in the background. — The assistant wants to check power during the benchmark, not after. This shows an understanding that peak power draw occurs under sustained load, and that waiting for the benchmark to finish would miss the transient. The 30-second sleep is calibrated to give the benchmark time to ramp up to full concurrency.

Adding clocks.current.sm to the query. — The assistant is deepening its diagnostic toolkit. Earlier queries only checked utilization and power. Now it wants to know if the GPUs are clock-throttled. This suggests the assistant is building a mental model of the GPU's performance state that includes thermal and power constraints.

Not waiting for benchmark output. — The message ends with the nvidia-smi output and does not include the benchmark results. The assistant will process those in a subsequent message. This is a deliberate separation of concerns: first confirm the hardware is healthy (power, clocks, utilization), then evaluate the software performance (throughput, latency). The assistant is building its diagnosis bottom-up.

Conclusion

Message [msg 805] is a masterclass in disciplined engineering under pressure. After a failed experiment that cut throughput by nearly 8×, the assistant does not panic, does not thrash, and does not rush into the next idea. Instead, it methodically reverts, restarts, and verifies. The message captures a moment of recovery and recalibration—a checkpoint that says "the system is healthy, the baseline is solid, and we can now try the next approach." The improved power draw of ~320–338W is an unexpected gift from the recovery, a reminder that even failed experiments can leave behind useful artifacts (matured autotune caches, thermal equilibration). For the reader following this optimization journey, [msg 805] is the quiet breath before the next dive—a moment of stability that makes the subsequent exploration possible.