The Blackwell Compatibility Check: How One Message Determined the Fate of GPU Proving on an RTX 5070 Ti

In the course of building a production-grade SNARK proving daemon for Filecoin, a single question threatened to derail an entire afternoon of work: would the CUDA backend actually run on the RTX 5070 Ti (Blackwell architecture, compute capability 12.0) sitting in the test machine? Message [msg 201] captures the moment when the assistant pivoted from "it compiles" to "will it run?" — a brief but critical investigation into GPU architecture compatibility that would determine whether the entire Phase 0 validation could proceed.

The Context: A Pipeline Waiting for a GPU

The conversation leading up to message [msg 201] had been a marathon of systems engineering. Over the course of several hours, the assistant had designed and implemented the full Phase 0 scaffold of cuzk — a pipelined SNARK proving daemon for Filecoin — creating six Rust crates, a gRPC API with eight RPCs, a priority scheduler, and a real prover module wired to filecoin-proofs-api. The workspace compiled cleanly with --no-default-features (CPU-only mode), and the end-to-end gRPC pipeline had been validated with a proof submission that failed only because the 45 GiB SRS parameters weren't in the expected location.

By message [msg 195], the parameters had been copied to /data/zk/params/, the workspace compiled without warnings, and the hardware was confirmed: an NVIDIA RTX 5070 Ti with 16 GiB VRAM, CUDA 13.1, compute capability 12.0 — Blackwell. The next step was clear: build with --features cuda-supraseal and run a real proof.

But there was a lurking uncertainty. The RTX 5070 Ti is a Blackwell GPU (sm_120), and the CUDA code paths in the Filecoin proof pipeline were written years earlier, targeting older architectures. Would the precompiled fatbin kernels in ec-gpu-gen support Blackwell? Would sppark's JIT compilation handle it? These questions had to be answered before investing time in a CUDA build that might fail at link time or, worse, at runtime with a cryptic cudaErrorInvalidDevice or PTX JIT compilation failed.

The Investigation: Two CUDA Paths, Two Build Systems

Message [msg 201] is deceptively brief — just two find commands and a few lines of commentary — but it represents a systematic investigation of two separate CUDA compilation paths. The assistant's reasoning, visible in the surrounding messages, reveals a careful triage of the problem.

The first path is supraseal-c2, which uses sppark for GPU acceleration. The assistant had already examined supraseal-c2-0.1.2/build.rs in message [msg 200] and seen that it delegates to sppark::build::ccmd(), which returns an nvcc command. The key question was whether sppark's build script filters GPU architectures — if it hardcodes -arch=sm_70 or -arch=sm_80, the Blackwell GPU would be unsupported.

The first command in message [msg 201] digs into this:

find ~/.cargo/registry/src -path "*/sppark*/build.rs" 2>/dev/null | sort | tail -1 | xargs cat 2>/dev/null | head -80

This finds the most recent version of sppark's build script and prints its first 80 lines. The output (visible in the conversation data) reveals that sppark's build system uses cc::Build with .cuda(true), which lets nvcc choose the default architecture for the current GPU. There's no explicit -arch flag filtering — nvcc will JIT-compile for whatever GPU it detects. This is the key insight: by delegating to nvcc at compile time rather than using precompiled fatbins, sppark naturally supports any CUDA-capable GPU, including Blackwell.

The second path is bellperson's native CUDA backend, which uses ec-gpu-gen and rust-gpu-tools. This path is more concerning because it relies on precompiled PTX or fatbin kernels. The assistant's second command probes this:

find ~/.cargo/registry/src -path "*/ec-gpu-gen*" -name "*.rs" 2>/dev/null | xargs grep -l "arch\|compute\|sm_" 2>/dev/null | head -5

This searches for architecture-related filtering in ec-gpu-gen source files. The results point to files like multiexp.rs, multiexp_cpu.rs, and source.rs — suggesting that architecture filtering exists but may be in the kernel generation code rather than the build system itself.

The Reasoning: Why This Matters

The assistant's thinking, visible in the surrounding messages, reveals a nuanced understanding of the CUDA compilation ecosystem:

Precompiled vs. JIT-compiled kernels: The bellperson path uses rust-gpu-tools → rustacuda which loads precompiled fatbin kernels. If those fatbins were compiled for sm_70 (Turing) or sm_80 (Ampere), they would not run on sm_120 (Blackwell). The rust-gpu-tools library does query compute capability (as seen in message [msg 203] where get_compute_capability() is found), but it uses this for device selection, not for kernel compatibility verification.
nvcc default architecture behavior: With CUDA 13.1 and no explicit -arch flag, nvcc compiles for the default architecture of the host GPU. On a Blackwell system, this would be sm_120. The sppark path benefits from this because it compiles CUDA source files at build time rather than loading precompiled binaries.
The risk of silent failure: A build that succeeds but produces kernels that don't run on the target GPU is worse than a build that fails — it wastes time and creates confusion. The assistant's investigation was motivated by a desire to avoid this scenario.

Assumptions and Their Validity

The assistant makes several assumptions in this message:

Assumption 1: sppark delegates entirely to nvcc without architecture filtering. This is validated by reading the build script output — the cc::Build configuration doesn't set -arch flags, meaning nvcc will use its default behavior.

Assumption 2: CUDA 13.1's nvcc can generate code for sm_120. This is reasonable given that CUDA 13.1 ships with driver support for Blackwell (the nvidia-smi output in message [msg 195] confirms the driver is 590.48.01, which supports Blackwell). However, the assistant doesn't explicitly verify that nvcc can target sm_120 — it relies on the fact that the driver and CUDA toolkit are from the same generation.

Assumption 3: The ec-gpu-gen path is riskier than the sppark path. This is correct — precompiled fatbins are architecture-specific, while JIT compilation is architecture-agnostic. The assistant's decision to try the sppark path first (message [msg 204]) reflects this understanding.

The Outcome: A Successful Build

The investigation in message [msg 201] directly informed the decision in message [msg 204] to proceed with cargo build --workspace --features cuda-supraseal --release. The build succeeded without errors or warnings, producing a 21 MiB daemon binary and a 5.2 MiB bench binary (message [msg 205]). This was the green light needed to proceed with the end-to-end validation.

The subsequent proof runs (messages [msg 220] through [msg 236]) confirmed that the Blackwell GPU was fully supported. The first proof completed in 116.8 seconds, including the ~15-second SRS load from disk, and the second proof (with SRS cached) completed in 92.8 seconds — a 20.5% improvement. Both proofs passed internal verification, and the daemon correctly tracked metrics.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with CUDA compute capabilities and architecture naming (sm_70, sm_80, sm_120); understanding of the difference between JIT-compiled CUDA kernels and precompiled fatbins; knowledge of how cc::Build with .cuda(true) works in Rust build scripts; and awareness of the Filecoin proof pipeline's two GPU backends (supraseal via sppark, and bellperson native via ec-gpu-gen).

Output knowledge created by this message is the confirmation that the sppark build system does not filter GPU architectures, meaning the supraseal-c2 CUDA backend should work on any GPU supported by the installed CUDA toolkit. This is a non-trivial finding — it means that the cuzk proving daemon can run on modern GPUs without requiring updates to the upstream CUDA kernel code.

The Broader Significance

This message exemplifies a pattern that recurs throughout the cuzk development session: the assistant systematically eliminates uncertainty before committing to expensive operations. Building with CUDA features takes several minutes of compilation time, and a failed build would waste time and erode confidence. By investing a few seconds in examining build scripts, the assistant avoided this risk.

More importantly, the investigation reveals the architectural layering of the Filecoin proof pipeline. The fact that supraseal-c2 uses sppark's ccmd() function — which returns a raw nvcc command — means that GPU compatibility is determined by the CUDA toolkit version, not by any intermediate library. This is a design strength: it means that as NVIDIA releases new GPU architectures, the proof pipeline can support them simply by updating the CUDA toolkit, without waiting for ec-gpu-gen or rust-gpu-tools to release updated fatbins.

The message also highlights the value of reading source code rather than relying on documentation or assumptions. The assistant could have assumed that "supraseal supports CUDA" and proceeded with the build, but instead verified the mechanism by which it supports CUDA. This attention to the build system's internals is characteristic of the session's approach to every component it touches — from the SRS parameter cache to the gRPC message size limits to the serialization format of C1 outputs.

In the end, message [msg 201] is a small but pivotal moment in a much larger engineering effort. It's the kind of investigation that, if omitted, could lead to hours of debugging a "CUDA error: no kernel image is available for execution on the device" at runtime. By asking the right question at the right time — "will this actually run on Blackwell?" — the assistant saved time and ensured that the subsequent validation produced meaningful results.