The Health Check That Confirmed a Breakthrough: Finding a Working Backend for Blackwell FP4 Inference

The Message

[assistant] [bash] ssh root@10.1.230.174 'for i in $(seq 1 60); do curl -s http://127.0.0.1:30000/health > /dev/null 2>&1 && echo "healthy at attempt $i" && break || echo "not ready ($i)"; sleep 5; done'
healthy at attempt 1

At first glance, this message appears trivial: a simple health check against an SGLang inference server, succeeding on the very first attempt. But in the context of the preceding hour of debugging, this single line represents a critical inflection point — the moment when a long sequence of failed backend configurations finally gave way to a working deployment. The message is a bash one-liner that loops up to 60 times, polling the /health endpoint of a locally running SGLang server every 5 seconds, and it reports "healthy at attempt 1." The server was ready before the health check even began.

The Context: A Gauntlet of Failed Backends

To understand why this message matters, one must appreciate what came before it. The assistant was in the midst of deploying the Qwen3.5-397B-A17B-NVFP4 model — a 397-billion-parameter Mixture-of-Experts (MoE) model quantized to NVFP4 (NVIDIA FP4) — on an 8× RTX PRO 6000 Blackwell GPU system. This was not a straightforward deployment. The Blackwell architecture (compute capability SM120) was brand new, and the software ecosystem was still catching up. The assistant had already spent hours building custom kernels, patching SGLang for SM120 support, and upgrading the entire CUDA/PyTorch stack to nightly versions.

The immediate predecessor to this message was a systematic search for a working backend configuration. The assistant had tested:

1. flashinfer_trtllm for MoE (message 5960): This backend uses NVIDIA's TensorRT-LLM fused MoE kernels. It crashed immediately with an OOM (out of memory) error because the TRT-LLM path requires weight shuffling that temporarily doubles memory usage. More fundamentally, the TRT-LLM MoE kernels were compiled for SM100 (Hopper-generation GPUs) and did not support SM120 at all.
2. flashinfer_cutedsl for MoE (message 5963): This backend uses NVIDIA's CUTE DSL framework, which JIT-compiles kernels for the exact GPU architecture. It loaded successfully and even began CUDA graph capture — a promising sign. But when the smoke test ran (message 5966), the model produced garbage output: repeated exclamation marks (!!!!), the hallmark of corrupted computation. The cutedsl backend had the same SM120 kernel issue as the default auto backend.
3. flashinfer_trtllm for dense FP4 GEMM (message 5969): The assistant tried using TRT-LLM only for the dense (non-MoE) FP4 matrix multiplications while keeping MoE on flashinfer_cutlass. This also crashed immediately — the dense FP4 TRT-LLM kernels were equally SM100-only. Each failure was informative. Each crash or garbage output narrowed the search space. The assistant was methodically eliminating options, building a mental map of which backends supported SM120 and which did not. The pattern was clear: trtllm and cutedsl backends, while potentially faster on supported hardware, were compiled or designed for SM100 and did not function correctly on Blackwell.

The Decision: Going with flashinfer_cutlass for Everything

Message 5972 shows the assistant's final attempt before the subject message. The launch command used:

--moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cutlass

This was a conservative choice. The flashinfer_cutlass backend uses NVIDIA's CUTLASS library, which is a more mature, well-tested kernel template library. Unlike TRT-LLM's hand-tuned kernels or CUTE DSL's JIT compilation, CUTLASS generates kernels at compile time using well-established CUDA C++ templates. The assistant had already confirmed in earlier testing (message 5946 and surrounding context) that flashinfer_cutlass worked correctly for FP4 on SM120 — it was the only backend that produced correct output.

The decision to use cutlass for both MoE and dense FP4 GEMM was a pragmatic tradeoff. The assistant knew that trtllm would likely be faster on SM100, and cutedsl might eventually be faster once its JIT kernels were fixed for SM120. But on Blackwell hardware in early March 2026, cutlass was the only option that actually worked. Speed is irrelevant if the output is garbage.

Why the Health Check Succeeded Immediately

The health check returning "healthy at attempt 1" is significant for several reasons. First, it means the server loaded and initialized within seconds — remarkably fast for a 397B-parameter model spread across 8 GPUs. The previous launch with flashinfer_cutedsl (message 5963) had taken long enough that the bash tool timed out after 120 seconds, yet the server still started successfully in the background. This time, with cutlass, the server was ready almost instantly.

Second, the immediate success validated the assistant's hypothesis that cutlass was the correct backend for SM120. If there had been a compatibility issue — a missing kernel, a tensor shape mismatch, or a CUDA graph capture failure — the health check would have returned nothing or an error. Instead, it returned immediately.

Third, the health check's success set the stage for the smoke test that followed in message 5974. That test confirmed correct output: "OK" for a simple completion, "323" for a math problem, and a full reasoning trace for a thinking-mode query. The model was not just running — it was running correctly.

Assumptions and Knowledge Required

To understand this message fully, one must know several things:

• What a health check means in SGLang: The /health endpoint returns a 200 OK response when the server has finished loading the model, capturing CUDA graphs, and is ready to accept inference requests. A successful health check is the server's stamp of approval that initialization is complete.
• The significance of backend selection: SGLang supports multiple MoE runner backends (flashinfer_trtllm, flashinfer_cutlass, flashinfer_cutedsl, triton, deep_gemm) and FP4 GEMM backends (flashinfer_cudnn, flashinfer_trtllm, flashinfer_cutlass, flashinfer_cutedsl). Each backend uses different kernel implementations with different performance characteristics and hardware support.
• The SM120 compute capability: Blackwell GPUs report compute capability 12.0 (SM120), which is distinct from Hopper's SM90 and the rumored SM100. Many kernels compiled for SM100 do not run on SM120, and JIT-compiled kernels may produce incorrect results.
• The model architecture: Qwen3.5-397B-A17B-NVFP4 is a 397B-parameter MoE model with 17B active parameters per token, quantized to NVFP4 format. It requires specialized FP4 GEMM kernels and MoE routing kernels that must support both the quantization format and the target GPU architecture. The assistant made a key assumption: that flashinfer_cutlass would work for both MoE and dense FP4 GEMM on SM120. This assumption was based on earlier testing (messages 5945-5956) where the assistant had examined the FlashInfer source code and confirmed that cutlass backend support was implemented for SM120. The assumption proved correct.

What This Message Created

This message produced critical output knowledge: the server was healthy and ready. This knowledge enabled the assistant to proceed to the smoke test (message 5974) and, ultimately, to the production deployment with systemd service configuration. The health check was the gate that had to be passed before any further work could continue.

More broadly, this message confirmed that the flashinfer_cutlass backend was the correct choice for Blackwell FP4 inference — a finding that would inform the final production configuration. The assistant would later add --kv-cache-dtype bf16 to fix an FP8 KV cache accuracy issue and deploy the server as a systemd service, but the fundamental backend choice was validated by this single health check.

The Thinking Process

The reasoning visible in the surrounding messages reveals a methodical debugging approach. The assistant did not randomly try backends; it systematically worked through the options, testing each one and analyzing the failure mode:

• trtllm → crash (OOM + SM100-only kernels)
• cutedsl → garbage output (JIT kernels produce incorrect results on SM120)
• trtllm for dense only → crash (same SM100 limitation)
• cutlass for both → success Each test was a hypothesis: "Will this backend work on SM120?" Each failure provided information about why it didn't work. The assistant read source code, checked kernel dispatch logic, and examined error messages to understand the root cause. By the time the health check succeeded, the assistant had built a comprehensive understanding of the SM120 backend landscape — knowledge that would be valuable for future Blackwell deployments. The health check itself was a simple polling loop with a 5-second interval and a 60-attempt maximum (5 minutes total). The fact that it succeeded on the first attempt, before any polling delay, means the server initialization was nearly instantaneous. This was a welcome surprise after the multi-minute CUDA graph captures of previous launches.

Conclusion

A single health check that returns "healthy at attempt 1" might seem like the most mundane possible message in a coding session. But in the context of this conversation — after hours of build failures, kernel crashes, and garbage outputs — it represents a genuine breakthrough. The assistant had found the combination of backends that worked on Blackwell hardware, validated it, and was ready to move forward. Sometimes the most important messages are the quietest ones.