The Blackwell Checkpoint: Validating GPU Architecture Compatibility in the cuzk Proving Pipeline

A Pivotal Moment in the cuzk Phase 0 Validation

In the middle of an intense coding session to build the cuzk proving daemon — a pipelined SNARK proving engine for Filecoin proof generation — the assistant arrives at a critical juncture. Message 198 is deceptively short, but it represents a moment of hardware-software validation that could have derailed the entire Phase 0 effort. The message reads:

Good. The RTX 5070 Ti is a Blackwell-class GPU (sm_120). Let me check if supraseal supports it, or if we need the regular bellperson CUDA backend.

`` nvcc --version 2>&1; echo "---"; nvidia-smi --query-gpu=compute_cap --format=csv 2>&1 nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2025 NVIDIA Corporation Built on Tue_Dec_16_07:23:41_PM_PST_2025 Cuda compilation tools, release 13.1, V13.1.115 Build cuda_13.1.r13.1/compiler.37061995_0 --- compute_cap 12.0 ``

This message is the culmination of a chain of reasoning that begins with a simple question: "Will the GPU acceleration work on this machine?" and branches into a deep investigation of CUDA compilation pipelines, precompiled kernel compatibility, and the architectural differences between two competing GPU proving backends.

The Context: Why This Question Matters

To understand why this message is so important, we must reconstruct the situation. The assistant and user have been building the cuzk proving daemon — a Rust workspace with six crates that implements a gRPC-based SNARK proving service for Filecoin's proof-of-replication (PoRep) protocol. They have completed the scaffold: the gRPC API is defined, the engine with priority scheduler is wired, the prover module calls seal_commit_phase2 through filecoin-proofs-api, and the daemon can accept 51 MB C1 outputs over the wire. But all of this has been tested only with --no-default-features, meaning the CPU-only path. The next step — running a real end-to-end proof with GPU acceleration — requires building with CUDA support.

The machine has an RTX 5070 Ti, which is a Blackwell-architecture GPU (compute capability 12.0, architecture codename sm_120). Blackwell is NVIDIA's latest architecture, succeeding Hopper (sm_90) and Ada Lovelace (sm_89). It was announced in 2024 and the RTX 5070 Ti is a mid-range consumer card based on this architecture. The key question is whether the existing CUDA code in the proving pipeline supports this new architecture, or whether it will fail with "no compatible kernel" errors.

This is not a trivial question. The Filecoin proving pipeline has two CUDA backends, and they handle architecture compatibility very differently:

supraseal-c2 (the cuda-supraseal feature): Uses the sppark library from Supranational, which compiles CUDA kernels via nvcc at build time. Because nvcc JIT-compiles for the target architecture, this path is generally forward-compatible with new GPU architectures — as long as the CUDA toolkit version supports the new architecture.
bellperson (the cuda feature): Uses ec-gpu-gen and rust-gpu-tools, which rely on precompiled fatbin kernels. These are compiled ahead of time for specific compute capabilities. If the fatbin doesn't include a kernel for sm_120, the GPU will reject it. The assistant's reasoning, visible in the surrounding messages, shows an awareness of this distinction. In message 201, the assistant examines the sppark build system and notes that "sppark delegates to nvcc for CUDA compilation" and that it "doesn't explicitly filter architectures — it uses cc::Build with .cuda(true) which lets nvcc choose the default architecture." In message 202, the assistant examines rust-gpu-tools and finds that it queries compute capability at runtime, suggesting it might filter devices based on compatibility. The assistant is essentially performing a risk assessment: which backend is more likely to work on this brand-new GPU?## The Assumptions Embedded in the Question The message makes several implicit assumptions that are worth examining. First, the assistant assumes that one of the two backends will work — the question is framed as "which one to use," not "whether either works." This is a reasonable assumption given that CUDA 13.1 is a very recent toolkit (released with Blackwell support), but it's not guaranteed. The sppark library might have hardcoded architecture checks, or the ec-gpu-gen fatbins might have been compiled before Blackwell was supported. Second, the assistant assumes that supraseal is the preferred path if it works. The message says "Let me check if supraseal supports it, or if we need the regular bellperson CUDA backend," implying a hierarchy: try supraseal first, fall back to bellperson. This preference is rooted in earlier analysis from the session, where supraseal-c2 was identified as the more optimized GPU backend for the specific Groth16 proving workload, with custom CUDA kernels for NTT and MSM operations rather than relying on the more generic bellperson implementation. Third, the assistant assumes that nvcc JIT-compilation will handle Blackwell. The nvcc --version output shows CUDA 13.1, which was released with Blackwell support. But JIT-compilation requires the CUDA driver to support the architecture, and the sppark build script might pass architecture flags that restrict compilation. The assistant checks this by examining the sppark build script in subsequent messages.

The Knowledge Required to Understand This Message

To fully grasp what's happening here, one needs knowledge spanning several domains:

NVIDIA GPU architectures and compute capabilities: Understanding that sm_120 means Blackwell, that different CUDA toolkit versions support different architectures, and that precompiled kernels (fatbins) are architecture-specific while JIT-compiled kernels are more portable.
The Filecoin proving pipeline's CUDA backend architecture: Knowing that there are two paths — supraseal-c2 (via sppark) and bellperson (via ec-gpu-gen/rust-gpu-tools) — and that they handle GPU compilation differently.
The cuzk build system: Understanding how the cuda-supraseal feature flag propagates through cuzk-core → filecoin-proofs-api → supraseal-c2 → sppark, and how the cuda feature flag takes a different route through bellperson.
Rust build tooling: Knowing that cargo build --features cuda-supraseal will compile CUDA kernels as part of the build process, and that this requires nvcc to be on PATH.
The history of the session: Understanding that the team has been working toward this moment for hours — building the daemon, debugging parameter locations, fixing serialization issues — and that this GPU compatibility check is the last gate before the first real proof.

The Output Knowledge Created

This message produces several pieces of critical knowledge:

Confirmed Blackwell architecture: The nvidia-smi --query-gpu=compute_cap --format=csv output confirms compute capability 12.0, which is definitively Blackwell. This is important because it rules out any ambiguity about which architecture the GPU is — some early RTX 5070 Ti reviews suggested they might use a cut-down die with different compute capability.
CUDA 13.1 compatibility: The nvcc --version output shows CUDA 13.1, build cuda_13.1.r13.1, which is the latest CUDA toolkit. This is significant because CUDA 12.x did not support Blackwell natively (it required the R535+ driver with compatibility mode). CUDA 13.1 has full Blackwell support, meaning nvcc can generate code for sm_120.
The decision framework: The message establishes the criteria for choosing between supraseal and bellperson. The assistant will check whether sppark's build system can compile for Blackwell, and if not, will check whether ec-gpu-gen's precompiled kernels include sm_120 support.

What Happens Next: The Resolution

The subsequent messages (204-209) show the resolution. The assistant builds with --features cuda-supraseal and the build succeeds — no errors, no warnings. The binaries are produced: a 21 MB daemon and a 5.2 MB bench tool. The supraseal path works on Blackwell.

This is a significant validation. It means that the sppark library's nvcc-based compilation pipeline correctly handles the Blackwell architecture, and that CUDA 13.1's JIT compiler can generate appropriate machine code for the RTX 5070 Ti. The daemon is then started, a real PoRep C1 output is submitted, and the first proof completes in 116.8 seconds — including a ~15 second SRS parameter load from disk. A second proof with the SRS cached completes in 92.8 seconds, demonstrating a 20.5% improvement from SRS residency.

The Broader Significance

This message, though brief, represents a critical architectural decision point. The choice between supraseal and bellperson has implications beyond just "does it compile." Supraseal's custom CUDA kernels are optimized specifically for the Groth16 proving workload, with specialized NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) implementations that can exploit GPU parallelism more effectively than bellperson's more generic approach. By validating that supraseal works on Blackwell, the assistant ensures that the cuzk daemon will benefit from these optimizations.

Moreover, this validation has implications for the broader Filecoin ecosystem. The RTX 5070 Ti is a consumer-grade GPU that costs a fraction of the data center GPUs typically used for Filecoin proving (like the RTX 4090 or A100). If Blackwell consumer cards can efficiently run the proving pipeline, it could lower the barrier to entry for Filecoin storage providers, potentially increasing the network's storage capacity. The assistant's work here — validating that the proving pipeline works on this hardware — is a small but important step toward democratizing Filecoin proof generation.

Mistakes and Incorrect Assumptions

Looking critically at this message, one potential blind spot is the assumption that if supraseal compiles, it will run correctly. Build-time compilation success does not guarantee runtime correctness — there could be subtle numerical differences in how Blackwell handles certain operations, or the JIT-compiled kernels might use suboptimal instruction sequences. The assistant implicitly assumes that nvcc's JIT compilation will produce correct code, which is generally true but not guaranteed for a brand-new architecture.

Another assumption worth examining is that the bellperson fallback would work if supraseal didn't. The bellperson path uses precompiled fatbin kernels, and it's entirely possible that those fatbins don't include sm_120 support. If supraseal had failed, the bellperson path might also have failed, leaving the team with no working GPU backend. The assistant doesn't explicitly acknowledge this risk.

Finally, the message assumes that the GPU is the primary compute device for proving. This is true for the current pipeline, but the optimization proposals developed earlier in the session (particularly the Sequential Partition Synthesis and Persistent Prover Daemon) envision a more complex architecture where CPU and GPU work in concert. The GPU compatibility check is necessary but not sufficient for the full vision.

Conclusion

Message 198 is a textbook example of a "small question with large implications" moment in software engineering. The assistant's brief check of GPU compute capability — a single command that takes seconds to run — determines the entire trajectory of the validation phase. It answers the question "can we prove on this hardware?" and opens the door to the first real end-to-end proof. Without this validation, the team would be building and testing against an unknown hardware target, risking wasted effort if the GPU backend proved incompatible. The message demonstrates the importance of understanding the hardware-software interface, the value of knowing your build toolchain's architecture support, and the critical role of explicit validation at architectural boundaries.