The Pivot Point: Waiting for SM120 Allreduce Fusion to Compile

A Message in the Trenches of GPU Kernel Hacking

In the long arc of optimizing GLM-5-NVFP4 inference on eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a quiet but pivotal moment — a single message where the assistant, after hours of patching kernel source code and modifying upstream dependencies, finally launches the server and waits. Message 791 is not dramatic. It contains no breakthroughs, no errors, no triumphant benchmarks. It is simply a pause: a 60-second sleep followed by a log grep. But that pause represents the culmination of a high-risk, high-reward engineering gamble — the attempt to force FlashInfer's allreduce fusion kernels to compile and run on SM120 (consumer Blackwell) hardware that they were never designed for.

The Message

Here is the message in full:

[assistant] This will take time because it needs to JIT compile the allreduce kernels for SM120. Let me monitor: [bash] ssh root@10.1.230.174 "sleep 60 && grep -n 'allreduce\|fired up\|Error\|error\|Traceback\|compile\|JIT\|trtllm_comm\|lamport\|FAILED\|workspace' /root/sglang-server.log | tail -30" 14:[2026-02-19 13:53:43] server_args=ServerArgs(model_path='lukealonso/GLM-5-NVFP4', tokenizer_path='lukealonso/GLM-5-NVFP4', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, che...

The output shows only the beginning of the server log — the server_args line — which means the server has started its initialization but has not yet reached the critical JIT compilation phase. The suspense is palpable: will the patched kernels compile, or will they fail with cryptic CUDA errors?

Why This Message Was Written

The motivation behind message 791 is deeply rooted in the performance bottleneck that had plagued the entire deployment. Earlier in the session, the assistant had achieved impressive throughput gains — from ~880 tok/s to ~3,740 tok/s — by enabling FlashInfer's CUTLASS MoE autotune for SM120 and increasing --max-running-requests. However, GPU power draw remained stubbornly around 250W out of a 600W TDP, indicating severe underutilization. The root cause was identified: FlashInfer's allreduce fusion, which fuses the allreduce communication with MoE kernel computation to reduce PCIe synchronization overhead, was disabled on SM120 because the underlying TRT-LLM communication kernels only supported SM90 (Hopper) and SM100 (datacenter Blackwell).

The assistant had already tried launching the server with --enable-flashinfer-allreduce-fusion on SM120 hardware, and it crashed immediately with a RuntimeError: No supported CUDA architectures found for major versions [9, 10] (see [msg 754]). The flashinfer JIT compilation context explicitly filtered out any architecture outside SM90/SM100. This was not a bug — it was a deliberate design choice by the FlashInfer team, who had not validated the allreduce fusion kernels on consumer Blackwell GPUs.

The user's instruction at [msg 768] — "Please think big and don't be afraid to fork/modify code" — gave the assistant license to go beyond configuration tweaks and into deep kernel patching. Message 791 is the direct consequence of that mandate: after modifying four separate files across two codebases (flashinfer and sglang), the assistant is now testing whether those patches actually work.

The Patches That Led Here

To understand what message 791 is waiting for, we must trace the patches applied in the preceding messages:

1. flashinfer/jit/comm.py ([msg 766]): The gen_trtllm_comm_module function called get_nvcc_flags_list(supported_major_versions=[9, 10]), which explicitly restricted compilation to SM90 and SM100. The assistant changed this to [9, 10, 12], adding SM120 (architecture major version 12, which maps to compute capability 12.0 for Blackwell).
2. sglang/python/sglang/srt/layers/communicator.py ([msg 767]): The communicator's architecture detection logic was patched to recognize SM120 as a valid target for allreduce fusion, changing the gate from (_is_sm90_supported or _is_sm100_supported) to include _is_sm120_supported.
3. sglang/python/sglang/srt/server_args.py ([msg 769]): Similarly, the server argument validation that auto-enables allreduce fusion on supported architectures was extended to include SM120.
4. flashinfer/data/include/flashinfer/comm/trtllm_allreduce.cuh ([msg 781]): The most critical patch. The CUDA header file contained architecture-specific code paths gated by #if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) && (__CUDA_ARCH__ < 1200)). The < 1200 exclusion meant that on SM120 (where __CUDA_ARCH__ equals 1200), the code would fall through to #else branches that lacked cudaGridDependencySynchronize() calls. The assistant removed the upper bound, changing it to __CUDA_ARCH__ >= 900, allowing SM120 to use the same cooperative grid synchronization paths as SM100.
5. JIT cache cleared ([msg 786]): The flashinfer JIT cache at /root/.cache/flashinfer/0.6.3/120a/generated/trtllm_comm* was deleted to force recompilation with the new architecture flags.

Assumptions and Risks

The assistant made several assumptions in message 791, some explicit and some implicit:

The kernels will compile for SM120. This was the central gamble. The CUDA source files (trtllm_allreduce.cu, trtllm_allreduce_fusion.cu, trtllm_moe_allreduce_fusion.cu) contained no SM-specific inline assembly or PTX directives — the assistant verified this at [msg 774] by grepping for architecture-specific patterns and finding zero matches. However, the compilation context in comm.py had explicitly excluded SM120 for a reason: the TRT-LLM communication kernels may use CUDA features (like specific warp intrinsics or shared memory sizes) that behave differently on SM120. Consumer Blackwell (RTX PRO 6000, compute capability 12.0) has different hardware characteristics than datacenter Blackwell (B200, compute capability 10.0), including smaller shared memory per SM and different tensor core configurations.

The removed < 1200 gate is safe. The cudaGridDependencySynchronize() function is a Hopper/Blackwell cooperative grid launch feature. On SM120, this function exists (Blackwell supports it), but the assistant assumed that its presence or absence would not cause correctness issues. The #else branches that SM120 would have fallen into before the patch use different synchronization mechanisms (volatile memory operations), which might be slower but should still be correct.

The JIT compilation will not exhaust GPU memory. FlashInfer's JIT compilation uses nvcc under the hood, which can be memory-intensive. On a system with 8 GPUs each potentially holding model weights, triggering a JIT compile could compete for GPU memory or cause driver timeouts. The assistant did not check available memory before launching.

The server will not deadlock during initialization. The server was launched with --enable-flashinfer-allreduce-fusion, which triggers workspace initialization for the fusion kernels during startup. If the JIT compilation hangs or the initialized workspace has incorrect dimensions, the server could hang indefinitely rather than crash cleanly.

Input Knowledge Required

To fully understand message 791, one needs knowledge spanning several domains:

• CUDA architecture numbering: SM90 = Hopper (H100), SM100 = datacenter Blackwell (B200), SM120 = consumer Blackwell (RTX 5090, RTX PRO 6000). The jump from SM100 to SM120 is not incremental — these are different chip configurations with different compute capabilities.
• FlashInfer's JIT compilation system: FlashInfer uses a Just-In-Time compilation approach where CUDA kernels are compiled at runtime using nvcc. The compilation_context module tracks target architectures and generates appropriate -arch flags. The gen_trtllm_comm_module function specifically requests compilation for SM90/SM100.
• Allreduce fusion in MoE inference: Mixture-of-Experts models require communication between GPUs to aggregate expert outputs. Normally, this involves an allreduce operation followed by expert computation. Allreduce fusion combines these steps, reducing PCIe synchronization overhead. On systems with PCIe-connected GPUs (as opposed to NVLink-connected), this fusion can significantly improve throughput.
• The GLM-5-NVFP4 model: A large MoE model quantized to NVFP4 (NVIDIA 4-bit floating point), requiring specific attention backends and MoE runners. The model uses modelopt_fp4 quantization and benefits from FlashInfer's CUTLASS-based MoE kernel.
• The hardware topology: Eight RTX PRO 6000 GPUs in a Proxmox VM, each on its own PCIe root complex, meaning no P2P DMA between GPUs. This makes allreduce fusion particularly important — without it, every allreduce operation traverses the PCIe bus through system memory.

Output Knowledge Created

Message 791, by itself, does not produce new knowledge — it is a monitoring step. However, the output it waits for will determine the next phase of the optimization effort:

• If the server starts successfully: It would confirm that the patches are functionally correct and that FlashInfer's allreduce fusion kernels can run on SM120 hardware. This would be a significant finding, as it would mean the architecture gates in flashinfer were overly conservative.
• If the server crashes: It would reveal the actual incompatibilities between the TRT-LLM communication kernels and SM120 hardware, potentially pointing to specific CUDA features or synchronization primitives that fail. The partial output shown — the server_args line — indicates that the server has begun initialization but has not yet reached the JIT compilation phase. The 60-second sleep was chosen because JIT compilation of CUDA kernels typically takes tens of seconds on modern hardware.

The Thinking Process

The assistant's reasoning in message 791 reveals a methodical, risk-aware approach to kernel patching. The progression of thought is visible across the preceding messages:

1. Diagnosis: The allreduce fusion crash was traced to gen_trtllm_comm_module rejecting SM120. The error message was precise: "No supported CUDA architectures found for major versions [9, 10]."
2. Verification of kernel source: Before patching, the assistant checked whether the CUDA source files contained architecture-specific code. Finding zero matches for sm_90, sm_100, or __CUDA_ARCH__ in the .cu files suggested the kernels were architecture-agnostic.
3. Discovery of the hidden gate: The architecture gate was not in the .cu files but in the .cuh header — specifically the < 1200 exclusion in trtllm_allreduce.cuh. This was a more subtle constraint: the kernel would compile for SM120, but the synchronization strategy would differ.
4. Conservative patching: The assistant patched only the version gates, not the kernel logic itself. The change from >= 900 && < 1200 to just >= 900 preserves all existing code paths while extending coverage to SM120.
5. Clean state: The JIT cache was cleared and old processes killed to ensure a clean compilation. This is important because flashinfer's JIT system caches compiled shared objects — stale cache entries could mask compilation failures. The 60-second sleep in the monitoring command reflects an understanding of JIT compilation timelines. CUDA kernel compilation with nvcc involves device code compilation, PTX assembly, and cubin generation, which for complex allreduce kernels can take 30-90 seconds depending on the number of architecture targets.

Broader Significance

Message 791 sits at the intersection of several important themes in modern ML infrastructure:

The gap between datacenter and consumer GPUs. NVIDIA's architecture numbering has diverged between datacenter (SM100 for Blackwell) and consumer (SM120 for Blackwell). Software stacks like FlashInfer and TRT-LLM are primarily validated on datacenter hardware, leaving consumer GPU users to either accept degraded performance or patch the code themselves.

The fragility of JIT-compiled CUDA kernels. FlashInfer's approach of JIT-compiling communication kernels at runtime provides flexibility but introduces failure modes that are hard to debug. A compilation failure during server startup leaves the system in an undefined state, and the error messages from nvcc are often opaque.

The necessity of forking in production ML. The user's instruction to "think big and don't be afraid to fork/modify code" acknowledges a reality of ML engineering: production deployments often require modifications to upstream dependencies. The assistant's willingness to patch both flashinfer and sglang — two actively maintained projects — demonstrates the level of customization needed for cutting-edge hardware.

The role of PCIe topology in multi-GPU inference. The entire allreduce fusion effort was motivated by the lack of NVLink between the RTX PRO 6000 GPUs. In a Proxmox VM with each GPU on its own PCIe root complex, allreduce operations become the dominant bottleneck. Fusion kernels that reduce the number of synchronization points can dramatically improve throughput — but only if they work on the target architecture.

Conclusion

Message 791 is a quiet hinge point in a much larger story. It represents the moment after all the hard work — the diagnosis, the source code analysis, the careful patching, the cache clearing — and before the verdict. The assistant has done everything it can at the code level. Now it must wait for the compiler.

The 60-second sleep is not just a practical delay for log monitoring. It is a moment of uncertainty in an otherwise deterministic process. Will the patches work? Will the JIT compilation succeed? Will the allreduce fusion kernels run correctly on SM120, or will they silently produce incorrect results? These questions can only be answered by the output of the next message.

In the broader narrative of the coding session, this message captures the essence of systems engineering at the frontier: you read the source code, you understand the gates, you remove them, and then you hold your breath and wait.