The Missing Header: Diagnosing a CUDA Graph Capture Failure on Blackwell SM120

The Message

ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'grep -i \"error\\|failed\\|FAILED\" /workspace/sglang_logs/sglang_gpu0.log | grep -v \"Possible\\|Open an\\|disable\" | tail -5'" 2>&1
Exception: Capture cuda graph failed: ninja exited with status 1
FAILED: cuda_0.o 
/root/venv/lib/python3.12/site-packages/nvidia/cu13/bin/../include/cuda_fp16.h:4500:10: fatal error: nv/target: No such file or directory
ninja: build stopped: subcommand failed.
[2026-05-18 21:08:53] Received sigquit from a child process. It usually means the child failed.

This message, from an extended debugging session deploying the GLM-5-NVFP4 model using SGLang on NVIDIA's desktop Blackwell (SM120) architecture, captures a pivotal diagnostic moment. After a long chain of environment fixes—installing matching CUDA 13.2 headers, creating symlinks for libcudart.so and libcuda.so stubs, and clearing the flashinfer JIT cache—the assistant's latest SGLang launch attempt has just crashed. The assistant now runs a targeted grep to extract the root cause from the server log, and what it finds reveals a new, distinct failure mode: the CUDA graph capture compilation cannot find the nv/target header, a critical component of NVIDIA's CCCL (CUDA C++ Core Libraries).

The Reasoning Behind the Diagnostic Command

The assistant's decision to run this specific grep command is not arbitrary—it reflects a careful, hypothesis-driven debugging strategy. The previous launch attempt (see [msg 9529]) timed out after 15 seconds, and a subsequent check (see [msg 9530]) revealed that the server had crashed with a "Received sigquit" message, indicating a child process failure. However, the log excerpt shown in [msg 9530] only displayed the generic suggestion text ("Possible solutions: 1. set --mem-fraction-static to a smaller value..."), which is unhelpful boilerplate printed by SGLang when CUDA graph capture fails.

The assistant recognized that the real error lay deeper in the log, buried among ninja build output and compiler diagnostics. The grep command is carefully crafted: it searches for lines containing "error", "failed", or "FAILED" (case-insensitive), then filters out the unhelpful suggestion lines containing "Possible", "Open an", or "disable". The tail -5 limits output to the last five matches, capturing the most recent failure. This is a surgical extraction designed to surface the compiler-level root cause while ignoring the generic error handling output.

The assistant's thinking process is visible in the structure of this command. It knows from the previous check that the server crashed during CUDA graph capture—the suggestion text in [msg 9530] explicitly mentions CUDA graph options. But the suggestion text is a catch-all error handler; it doesn't reveal why the graph capture failed. The assistant needs the actual compiler error to understand whether this is a memory issue (as the suggestions imply), a missing library, an include path problem, or something else entirely. The grep targets FAILED specifically because ninja (the build system used by torch's CUDA graph compilation) outputs FAILED: lines for each compilation unit that fails.

What the Error Reveals

The output delivers a precise diagnosis. The error chain is:

  1. Top-level exception: Exception: Capture cuda graph failed: ninja exited with status 1 — confirming the CUDA graph capture mechanism crashed.
  2. Failed compilation unit: FAILED: cuda_0.o — a specific object file in the graph capture compilation pipeline.
  3. Root compiler error: /root/venv/lib/python3.12/site-packages/nvidia/cu13/bin/../include/cuda_fp16.h:4500:10: fatal error: nv/target: No such file or directory — the CUDA toolkit's cuda_fp16.h header, at line 4500, attempts to include nv/target and cannot find it.
  4. Build system status: ninja: build stopped: subcommand failed. — the build aborted.
  5. Process death: Received sigquit from a child process — the SIGQUIT signal was sent to the parent, likely by the child process that encountered the fatal error. The critical detail is the include path. The error shows /root/venv/lib/python3.12/site-packages/nvidia/cu13/bin/../include/cuda_fp16.h, which resolves to the CUDA toolkit headers installed via the nvidia-cu13 pip package. This is the same package that the assistant upgraded in [msg 9520] from version 13.0 to 13.2 to resolve the earlier flashinfer JIT compilation issue. However, the CUDA graph capture compilation is a separate compilation pipeline from flashinfer's JIT, and it appears to use a different include path configuration.

Input Knowledge Required

To fully understand this message, one needs knowledge of several interconnected systems:

CUDA Graph Capture in PyTorch/SGLang: CUDA graphs allow capturing a sequence of GPU operations and replaying them with reduced launch overhead. In SGLang, this is used to optimize the prefill and decode kernels. The graph capture process compiles CUDA kernels on-the-fly using torch.compile or a similar mechanism, invoking nvcc and ninja under the hood.

CCCL and the nv/target Header: The nv/target header is part of NVIDIA's CCCL library, which provides C++ standard library extensions and target-specific macros for CUDA. It defines __NV_TARGET__ and related macros that allow code to be conditionally compiled for different GPU architectures. This header is typically bundled with the CUDA toolkit or installed as a separate package. Its absence indicates that the CUDA graph compilation's include paths do not cover the CCCL installation.

The Pip-Based CUDA Installation: The environment uses NVIDIA's pip-distributed CUDA packages (nvidia-cu13, nvidia-cuda-runtime, nvidia-cuda-nvcc, etc.) rather than a system-wide CUDA installation. These packages install headers and libraries into the Python virtual environment's site-packages/nvidia/cu13/ directory. The assistant had previously upgraded several of these packages to version 13.2 to match the nvcc compiler version, but the CCCL headers (which include nv/target) may not be part of the nvidia-cu13 package—they may come from a different package like nvidia-cccl or may be bundled with flashinfer instead.

SM120 Architecture Specifics: The desktop Blackwell (SM120) architecture is relatively new at the time of this session. Pre-compiled CUDA kernels for SM120 are scarce—flashinfer has no pre-compiled cubins for this architecture, and JIT compilation is required. The CUDA graph capture compilation faces similar challenges: it must compile kernels for SM120 from source, and any missing headers or incompatible toolchain components will cause failures.

Assumptions and Potential Mistakes

The assistant makes several implicit assumptions in this message:

  1. That the error is in the log file: The assistant assumes the SGLang server wrote detailed compilation errors to the log before crashing. This is a reasonable assumption given that the server uses ninja for compilation, which typically outputs error messages to stderr, and SGLang captures stderr into its log.
  2. That the grep filter is sufficient: The grep -v "Possible\|Open an\|disable" filter is designed to remove the generic suggestion text, but it could also filter out legitimate error messages that happen to contain these words. The assistant prioritizes removing noise over completeness.
  3. That the CUDA graph compilation uses the same include paths as flashinfer JIT: The assistant had just fixed flashinfer's JIT compilation by upgrading CUDA headers and creating symlinks. The assumption that the CUDA graph capture would also work after these fixes proved incorrect—the graph capture compilation has a different include path configuration that doesn't include the CCCL headers.
  4. That the nv/target header should be present in the CUDA toolkit headers directory: The error shows that cuda_fp16.h at line 4500 includes nv/target. The assistant may have assumed that upgrading nvidia-cuda-runtime to 13.2 would bring in all necessary headers, but nv/target is a CCCL header that may reside in a separate package or location. The mistake here is not in the diagnostic command itself—that is well-crafted—but in the earlier assumption that fixing the flashinfer JIT include paths would also fix CUDA graph capture. The two compilation pipelines have different dependency structures: flashinfer bundles its own CCCL headers and uses them via -I include paths, while the CUDA graph capture compilation relies on the system/toolkit include paths, which don't include CCCL.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. A confirmed failure mode: CUDA graph capture on SM120 fails with a missing nv/target header. This is a distinct error from the earlier flashinfer JIT compilation issues.
  2. A specific file and line number: The error originates from cuda_fp16.h:4500, which includes nv/target. This gives a precise location to investigate.
  3. A clear next debugging direction: The assistant now knows it needs to either (a) install the CCCL headers so they are findable by the CUDA graph compilation, (b) override the include paths for the graph capture compilation, or (c) disable CUDA graph capture entirely (as the suggestion text mentions).
  4. Confirmation that the earlier fixes were incomplete: The symlinks for libcudart.so and libcuda.so and the header version upgrade solved the flashinfer JIT linking issue but did not address the CUDA graph capture compilation.

The Broader Significance

This message exemplifies the layered complexity of deploying modern ML inference engines on cutting-edge hardware. Each component—SGLang, flashinfer, PyTorch, CUDA toolkit, CCCL—has its own compilation pipeline, include path configuration, and version requirements. On a new architecture like SM120, where pre-compiled binaries are unavailable, every component must compile from source, and each compilation step can fail for different reasons.

The assistant's approach here—running a targeted diagnostic command to extract the specific compiler error from a noisy log—is a textbook debugging technique. Rather than guessing at the cause or blindly trying workarounds, the assistant isolates the exact error message and uses it to inform the next action. The nv/target header error points directly to a missing CCCL dependency in the CUDA graph capture compilation pipeline, which is a fundamentally different issue from the flashinfer JIT problems that occupied the previous messages.

This message also highlights the tension between pip-based CUDA installations and traditional CUDA development workflows. NVIDIA's pip packages provide convenience for simple use cases, but they fragment the CUDA installation across multiple packages (nvidia-cu13, nvidia-cuda-runtime, nvidia-cuda-nvcc, nvidia-cccl, etc.), and the include path resolution becomes complex when different compilation pipelines expect different directory structures. A system-wide CUDA installation would place all headers in a single include/ directory, avoiding this fragmentation, but would require a multi-gigabyte download and system-level installation.

The assistant's next step, informed by this diagnostic, would likely be to either locate and add the CCCL include path to the CUDA graph compilation, or to disable CUDA graph capture as a workaround. Either way, the precise error message extracted in this message provides the necessary direction.