The Blackwell Crossroads: A Technical Decision in GPU-Accelerated SNARK Proving

Introduction

In the course of building a high-performance SNARK proving daemon called cuzk — a system designed to pipeline Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol — an agent working on the project arrived at a critical juncture. The system had been designed, the workspace scaffolded, the gRPC API defined, and the parameter files downloaded. But before a single real proof could be generated, a fundamental question needed answering: which GPU compute backend would actually run on the available hardware?

Message 204 captures this decision point. It is a short but dense piece of technical reasoning, containing a single paragraph of analysis followed by a build command and its output. On its surface, it appears to be a routine "try this build flag" moment. But beneath that lies a sophisticated chain of inference about CUDA architecture compatibility, the differences between precompiled fatbin kernels and Just-In-Time (JIT) compilation, and the architectural layering of the Filecoin proof system's GPU acceleration stack. This article unpacks that message in detail, examining the reasoning, the assumptions, the knowledge required, and the knowledge produced.

The Message

Here is the message in full:

The bellperson CUDA path uses rust-gpu-tools → rustacuda which uses precompiled fatbin kernels. The question is whether those fatbin binaries support sm_120. This is a potential problem — older ec-gpu-gen might not generate PTX/fatbin for Blackwell.

Let me try the supraseal path first since sppark delegates to nvcc which should JIT-compile for the current architecture. Let me try building with cuda-supraseal:

[bash] cargo build --workspace --features cuda-supraseal --release 2>&1 | tail -40 Compiling hex v0.4.3 Compiling bellpepper-core v0.2.1 Compiling toml_datetime v0.6.11 Compiling serde_spanned v0.6.9 Compiling bincode v1.3.3 Compiling merkletree v0.23.0 Compiling axum v0.7.9 Compiling tracing-serde v0.2.0 Compiling tonic-build v0.12.3 Compiling rust-gpu-tools v0.7.2 Compiling pasta_curves v0.5.1 Compiling tracing-subscriber v0.3.22 Compiling toml_edit v0.22.27 Compiling blstrs v0.7.1 Compiling bellpepper v0.2.1 Compiling cuzk-pro...

WHY This Message Was Written: The Context and Motivation

To understand why this message exists, we must trace the arc of the session that produced it. The agent was in the middle of Segment 5 of a larger investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline. The preceding segments had accomplished an extraordinary amount: mapping the full call chain from Curio's Go layer through Rust FFI into C++/CUDA kernels, accounting for the ~200 GiB peak memory footprint, identifying nine structural bottlenecks, and proposing five optimization strategies including Sequential Partition Synthesis, a Persistent Prover Daemon, and Cross-Sector Batching.

Now, in Segment 5, the agent had moved from design to implementation. Phase 0 of the cuzk proving engine had been scaffolded: a Rust workspace with six crates, a gRPC protobuf API, a core engine with a priority scheduler, and a prover module wired to filecoin-proofs-api calls. The 32 GiB of parameter files had been fetched and placed at /data/zk/params/. A test C1 output (the intermediate result from the first phase of PoRep proving) sat at /data/32gbench/c1.json, weighing 51 MB.

The next step was the moment of truth: run a real end-to-end proof on actual GPU hardware. The machine had an NVIDIA RTX 5070 Ti — a Blackwell-architecture GPU with compute capability 12.0, paired with CUDA 13.1. This was cutting-edge hardware, and that was precisely the problem.

The agent's investigation in messages 197 through 203 had revealed a tension in the GPU acceleration stack. The Filecoin proof system has two paths for GPU-accelerated Groth16 proving:

The bellperson CUDA path, which uses rust-gpu-tools → rustacuda to load precompiled CUDA fatbin kernels. These kernels are compiled ahead-of-time and embedded as binary blobs. If the fatbins don't contain code for a given GPU architecture, they simply won't run.
The supraseal-c2 path, which uses sppark (Supranational's GPU acceleration library). This path delegates to nvcc at build time via a Cargo build script, compiling CUDA source files (.cu) on the fly. Because nvcc runs on the build machine, it can JIT-compile for the detected architecture — or more precisely, it can generate PTX (parallel thread execution) intermediate code that the CUDA driver JIT-compiles to device code at runtime. The agent had been probing this stack methodically. In message 198, they checked nvidia-smi --query-gpu=compute_cap and got 12.0 — Blackwell. In message 199, they searched for supraseal-c2 and bellperson build scripts. In message 200, they read both build.rs files. In message 201, they examined sppark's build system and found it didn't explicitly filter architectures — it used cc::Build with .cuda(true), letting nvcc choose the default. In messages 202 and 203, they dug into rust-gpu-tools to see if it filtered devices by compute capability. Message 204 is the synthesis of that investigation. It represents the moment the agent stopped researching and started acting — but the action was informed by a clear mental model of the two paths' differing compilation strategies.

HOW the Decision Was Made

The decision process visible in message 204 is a textbook example of reasoning under uncertainty with incomplete information. The agent had not definitively proven that the bellperson fatbin path would fail on Blackwell. They had identified a risk — "The question is whether those fatbin binaries support sm_120. This is a potential problem" — but not a certainty.

The reasoning can be reconstructed as follows:

Identify the failure mode: Precompiled fatbins may not contain architecture-specific code for sm_120 (Blackwell). If they were compiled before Blackwell GPUs existed (which is likely, given that ec-gpu-gen 0.7.0 and rust-gpu-tools 0.7.2 predate the Blackwell architecture), they simply won't have the necessary code paths.
Assess the alternative: The supraseal path uses nvcc JIT compilation. Even if the sppark build script doesn't explicitly target sm_120, nvcc on CUDA 13.1 will generate PTX that the driver can JIT-compile for the Blackwell architecture. This path is inherently more forward-compatible.
Make a risk-weighted choice: "Let me try the supraseal path first." This is not a final decision — it's a try-first heuristic. If the supraseal build succeeds and produces correct proofs, the problem is solved. If it fails, the agent can fall back to diagnosing the bellperson path. The choice minimizes the expected time to a working proof.
Execute immediately: The agent doesn't over-analyze. They run cargo build --workspace --features cuda-supraseal --release and show the output. The build is proceeding — dependencies like hex, bellpepper-core, toml_datetime, merkletree, axum, tonic-build, rust-gpu-tools, pasta_curves, blstrs, and bellpepper are compiling. The command is piped through tail -40, showing only the tail of the output, which suggests the build is long enough that the interesting part is near the end. This is a decision made with bounded rationality — the agent had gathered enough information to form a hypothesis, but not enough to prove it. The action itself would generate the missing evidence.

Assumptions Made

Several assumptions underpin the reasoning in message 204:

That nvcc on CUDA 13.1 can generate code for sm_120: This is a reasonable assumption given that CUDA 13.1 is a recent release that should support Blackwell. However, it's not guaranteed — the specific nvcc flags used by sppark's build script might constrain the target architecture.
That the supraseal-c2 crate's CUDA kernels are compatible with Blackwell at the PTX level: Even if nvcc can compile for sm_120, the CUDA source code itself might use features or idioms that don't translate well. The agent is implicitly trusting that Supranational's CUDA code is portable across architectures.
That the bellperson fatbin path would definitely fail or be harder to debug: The agent says "This is a potential problem" but doesn't quantify the probability. The decision to try supraseal first assumes the bellperson path has a non-trivial chance of failure.
That the cuda-supraseal feature flag is the correct one: The agent had previously read the workspace Cargo.toml and identified this feature. The assumption is that this feature correctly enables the supraseal GPU backend without conflicting with other features.
That a successful build implies a working runtime: The build output shows compilation succeeding, but this doesn't guarantee that the CUDA kernels will load and execute correctly at runtime. The agent will need to run an actual proof to validate this.

Potential Mistakes and Incorrect Assumptions

While the reasoning is sound, there are several ways it could be wrong:

The supraseal path might also fail on Blackwell: If sppark's CUDA kernels use architecture-specific intrinsics or assume a minimum compute capability that Blackwell satisfies but in an unexpected way, the JIT compilation might produce incorrect results or crash at runtime. The agent's assumption that "JIT-compile for the current architecture" is a panacea is slightly optimistic.
The bellperson fatbin path might actually work: The fatbins might include PTX code (architecture-independent intermediate representation) that the CUDA driver can JIT-compile for any architecture, including Blackwell. The agent assumes "precompiled fatbin" means "compiled for a specific architecture," but fatbins can contain multiple architecture targets or PTX fallbacks. This is a subtle but important technical distinction.
The build output is incomplete: The agent shows only the tail of the build output. If there were warnings or errors earlier, they would be missed. The tail -40 command is a pragmatic choice for a long build, but it risks hiding important information.
Feature flag conflicts: Building with --features cuda-supraseal might enable features that conflict with other parts of the workspace. The agent doesn't check for this explicitly.
The assumption that Blackwell (sm_120) is the problem: The agent fixates on architecture compatibility, but the real issue might be something else entirely — driver version, memory bandwidth, kernel launch parameters, or CUDA runtime library paths.

Input Knowledge Required

To fully understand message 204, a reader needs knowledge in several domains:

CUDA Architecture and Compilation Model: Understanding the difference between precompiled fatbin kernels (device code compiled ahead-of-time for specific architectures) and JIT compilation via PTX (intermediate code compiled by the driver at runtime). The concept of "compute capability" (sm_120 for Blackwell) and how it determines which GPU instructions are available.

The Filecoin Proof System Stack: Knowledge that Filecoin's PoRep uses a two-phase proving pipeline (C1 and C2), that C2 involves Groth16 proofs over the BLS12-381 curve, and that GPU acceleration is critical for performance. Understanding that there are two competing GPU backends: the older bellperson path (using ec-gpu-gen/rust-gpu-tools) and the newer supraseal-c2 path (using sppark from Supranational).

Rust Build System and Cargo Features: Understanding how Cargo features enable conditional compilation, how build scripts (build.rs) work, and how cc::Build with .cuda(true) invokes nvcc. Knowing that --features cuda-supraseal is a feature flag that must be defined in the workspace's Cargo.toml.

Hardware Context: The RTX 5070 Ti is a Blackwell-architecture GPU. Blackwell (compute capability 12.0) was released in late 2025, making it newer than most of the CUDA libraries in the dependency tree. This recency is the root cause of the compatibility concern.

Previous Investigation (Messages 197-203): The reader must know that the agent had already checked nvcc version, compute capability, build scripts for both bellperson and supraseal, and the rust-gpu-tools device filtering code. Message 204 is the conclusion of that investigation, not the beginning.

Output Knowledge Created

Message 204 produces several valuable pieces of knowledge:

A confirmed build path: The cargo build --workspace --features cuda-supraseal --release command succeeds (at least through the compilation stage shown). This establishes that the supraseal CUDA backend compiles on this system with CUDA 13.1 and an sm_120 GPU. Future developers on similar hardware can use this as a reference.
A decision record: The message documents why the supraseal path was chosen over the bellperson path. This is valuable for future debugging — if proofs fail, a developer can revisit this reasoning and try the alternative path.
A risk assessment: The message explicitly identifies the fatbin architecture compatibility risk. This is knowledge that might otherwise be tacit — the agent could have just tried supraseal without explaining why. By writing it down, the agent creates a permanent record of the reasoning.
Dependency compilation order: The build output shows the compilation order of dependencies, which can be useful for understanding the dependency graph and identifying potential build bottlenecks.
A baseline for further investigation: If the supraseal path fails at runtime, the agent now has a clear next step: investigate the bellperson fatbin path more deeply, possibly by checking what architectures the fatbins were compiled for or by rebuilding them with Blackwell support.

The Thinking Process: A Window into Engineering Judgment

What makes message 204 interesting is not the technical content itself, but what it reveals about the agent's thinking process. Let's reconstruct the mental model:

Step 1: Map the terrain. The agent knows there are two GPU paths. They've traced both through the codebase. They understand the compilation strategy of each.

Step 2: Identify the risk vector. The risk is architecture compatibility. The agent knows the GPU is Blackwell (sm_120), a very new architecture. They suspect the precompiled fatbins predate Blackwell support.

Step 3: Evaluate the alternatives probabilistically. The bellperson path has a known failure mode (fatbin incompatibility) with unknown probability. The supraseal path has a different failure mode (nvcc JIT compilation might also fail) but the probability seems lower because nvcc is designed to handle the current architecture.

Step 4: Choose the action with the best expected outcome. Try supraseal first. If it works, great. If it doesn't, the failure mode is likely easier to diagnose (nvcc error messages are relatively clear).

Step 5: Execute and observe. Run the build command. Show the output. Let the results speak.

This is not the thinking of someone who knows the answer. It's the thinking of someone who knows how to find the answer — by forming hypotheses, testing them, and letting the system's behavior guide the next step. The message is a snapshot of that process in motion.

Conclusion

Message 204 is a small but revealing moment in a much larger engineering effort. It captures the transition from investigation to action, from uncertainty to hypothesis testing. The agent's decision to try the supraseal CUDA path first, based on a reasoned assessment of the two backends' compilation strategies, is a textbook example of pragmatic engineering judgment.

But the message also illustrates something deeper about the nature of systems engineering: that the most critical decisions are often not about which algorithm to use or which architecture to adopt, but about which path to try first when the system's behavior is unknown. The agent could have spent hours proving that the bellperson fatbin path would fail on Blackwell. Instead, they spent minutes trying the alternative. That's not laziness — it's efficiency.

The build output at the bottom of the message tells us that the compilation is proceeding. The dependencies are compiling in order: hex, bellpepper-core, toml_datetime, merkletree, axum, tonic-build, rust-gpu-tools, pasta_curves, blstrs, bellpepper, cuzk-pro.... The ellipsis at the end of cuzk-pro... is itself a narrative device — it tells us the story isn't over. The build will complete, the proof will run, and the agent will learn whether their hypothesis was correct.

In the broader arc of the cuzk project, this message is a hinge point. Before it, the agent was investigating, reading code, and gathering information. After it, the agent will be running proofs, measuring performance, and hardening the system. Message 204 is the moment the project crossed from "will it work?" to "how fast is it?" — and that crossing required a decision, however small, about which CUDA backend to trust with the Blackwell GPU.