The Pivot to GPU: Understanding a Critical Decision Point in the cuzk Proving Engine

Introduction

In the course of building a production-grade SNARK proving daemon for Filecoin, there comes a moment when the scaffolding is complete and the system must face reality: will it actually produce a valid cryptographic proof on real hardware? Message [msg 197] captures exactly this transition. It is a short, almost mundane message — the assistant reads two Cargo.toml files and states an intention to build with CUDA features. Yet this message sits at a critical inflection point in the development of the cuzk proving engine. Understanding why this message was written, what decisions it enabled, and the assumptions it carried reveals the careful, methodical approach that distinguishes production engineering from prototyping.

The Message in Full

The message reads:

Now let me build with CUDA features. Let me first check what features are available and which CUDA backend to use with this RTX 5070 Ti. [read] /home/theuser/curio/extern/cuzk/Cargo.toml [read] /home/theuser/curio/extern/cuzk/cuzk-core/Cargo.toml

The assistant then reads the contents of both files, showing the workspace member list and the beginning of the dependency declarations. The message is deliberately truncated — the file contents are shown as a conversation_data block, but the assistant's own text is just those two sentences and two file-read operations.

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, we must look at the sequence of events leading up to it. The assistant had just completed a multi-session investigation of the Filecoin proof generation pipeline, documented in seven optimization proposals and a comprehensive background reference ([msg 192]). Phase 0 of the cuzk proving engine had been scaffolded: a Rust workspace with six crates, a gRPC API, a priority scheduler, and a prover module wired to filecoin-proofs-api. The code compiled cleanly with --no-default-features, meaning it was verified without GPU acceleration.

But a proving daemon that doesn't prove anything is just a fancy RPC server. The assistant's todo list, visible in [msg 196], shows the progression:

  1. ✅ Verify cuzk workspace builds clean (no-default-features)
  2. 🔄 Build with CUDA features for GPU proving
  3. ⏳ Run daemon with FIL_PROOFS_PARAMETER_CACHE=/data/zk/params and test real PoRep C2 proof
  4. ⏳ Measure SRS residency benefit (first vs second proof) Message [msg 197] is the pivot point between step 1 and step 2. The assistant had verified that the environment was ready: the 45 GiB PoRep parameters were copied to /data/zk/params/, the 51 MB C1 test output was present at /data/32gbench/c1.json, and an RTX 5070 Ti with 16 GB VRAM was available and mostly idle ([msg 196]). The workspace compiled cleanly. Everything was in place for a real end-to-end test. The motivation for this message, then, is the recognition that building with CUDA features is not a trivial step. The assistant could have simply run cargo build --features cuda-supraseal and hoped for the best. Instead, it paused to examine the feature configuration. This reflects a deeper engineering principle: before committing to a build that might take minutes and produce hard-to-debug errors, understand the terrain.

How Decisions Were Made in This Message

Strictly speaking, no final decision is made in this message — the assistant is gathering information to make a decision. The decision itself (which CUDA backend to use) unfolds over the subsequent messages [msg 198] through [msg 205]. But the framing of the question reveals the decision-making framework.

The assistant asks: "which CUDA backend to use with this RTX 5070 Ti?" This question encodes several implicit choices:

  1. supraseal-c2 vs. bellperson native CUDA: The Filecoin proof ecosystem has two GPU acceleration paths. supraseal-c2 is a specialized C++/CUDA implementation maintained by Supranational that handles the entire Groth16 prover on GPU. The bellperson native CUDA path uses rust-gpu-tools and ec-gpu-gen to JIT-compile CUDA kernels for multiexponentiation and NTT operations. Each has different hardware compatibility profiles.
  2. Feature flag granularity: The cuzk-core/Cargo.toml defines features like cuda and cuda-supraseal. The assistant needs to understand which one maps to which backend, and whether both are viable on the available hardware.
  3. Blackwell (sm_120) compatibility: The RTX 5070 Ti is a Blackwell-generation GPU with compute capability 12.0. This is a very new architecture — CUDA 13.1 supports it, but older precompiled fatbin kernels may not include sm_120 code paths. The assistant's subsequent investigation ([msg 198]–[msg 203]) reveals that this is a genuine concern: rust-gpu-tools uses precompiled fatbins that may not cover Blackwell, while supraseal-c2 delegates to nvcc which JIT-compiles for the current architecture. By reading the Cargo.toml files first, the assistant is setting up the information needed to make this decision. It's a small but crucial act of reconnaissance.

Assumptions Made by the Assistant

Several assumptions underpin this message, some explicit and some implicit:

Assumption 1: The feature flags are correctly defined. The assistant assumes that cuda-supraseal and cuda are properly wired in the dependency chain from cuzk-core → filecoin-proofs-api → supraseal-c2 or bellperson. This turns out to be correct — the subsequent build succeeds without errors.

Assumption 2: The CUDA toolchain is compatible. The assistant assumes that nvcc 13.1 (part of CUDA 13.1) can compile for sm_120 (Blackwell). This is validated in [msg 198] where nvidia-smi reports compute capability 12.0 and nvcc --version confirms CUDA 13.1. The assumption holds.

Assumption 3: The build system will handle architecture detection. The assistant assumes that sppark (used by supraseal-c2) will correctly detect the GPU architecture and compile appropriate kernels. The investigation in [msg 201]–[msg 203] confirms that sppark's build.rs uses cc::Build with .cuda(true), which lets nvcc choose the default architecture — no explicit -arch flag filtering. This means the JIT compiler will generate code for the detected GPU.

Assumption 4: The workspace structure is correct for feature propagation. The assistant assumes that building with --features cuda-supraseal at the workspace level will correctly enable the feature in cuzk-core and transitively in filecoin-proofs-api. This is confirmed by the successful build in [msg 205].

Assumption 5: The GPU has sufficient VRAM. The RTX 5070 Ti has 16 GB VRAM. The assistant implicitly assumes this is enough for a 32 GiB PoRep C2 proof. This is a reasonable assumption given that the existing supraseal pipeline targets similar hardware, but it's not verified until the proof actually runs.

Mistakes or Incorrect Assumptions

At the time of this message, no mistakes are visible. However, a potential incorrect assumption lurks beneath the surface: the assistant assumes that building with CUDA features is the only prerequisite for running a real proof. In reality, the subsequent messages reveal several additional concerns:

Input Knowledge Required to Understand This Message

To fully grasp what is happening in [msg 197], a reader needs:

  1. Rust workspace and feature system: Understanding that [workspace] defines a set of crates that share dependencies, and that --features flags propagate through the dependency graph. The workspace has five active members (cuzk-proto, cuzk-core, cuzk-server, cuzk-daemon, cuzk-bench) and one future member (cuzk-ffi).
  2. Filecoin proof ecosystem architecture: Knowing that filecoin-proofs-api is the Rust FFI layer that wraps the actual proving logic, and that it has two GPU backends: cuda (bellperson native) and cuda-supraseal (Supranational's optimized implementation). The choice between them affects hardware compatibility, performance, and memory usage.
  3. CUDA compute capability and GPU generations: Understanding that different NVIDIA GPU architectures have different compute capabilities (sm_75 for Turing, sm_86 for Ampere, sm_89 for Ada Lovelace, sm_120 for Blackwell) and that CUDA kernels may need to be compiled specifically for each architecture. The RTX 5070 Ti's sm_120 capability is very new.
  4. The cuzk project's phase structure: Knowing that Phase 0 is explicitly scoped to "zero upstream library modifications" and uses existing GROTH_PARAM_MEMORY_CACHE pre-population for SRS residency. This constraint means the assistant cannot modify filecoin-proofs-api or supraseal-c2 — it must work with whatever CUDA support they provide.
  5. The prior investigation's findings: Understanding that the 45 GiB PoRep parameters are now at /data/zk/params/, that the test data is at /data/32gbench/c1.json, and that the workspace compiles cleanly without CUDA. These are the preconditions that make this message meaningful.

Output Knowledge Created by This Message

This message produces two concrete pieces of knowledge:

  1. The workspace Cargo.toml structure: The assistant (and the reader) now knows the exact member list, the workspace package metadata (edition 2021, Apache-2.0 license, Curio repository), and the beginning of the dependency declarations. This confirms that the workspace is correctly configured and that no unexpected crates are present.
  2. The cuzk-core Cargo.toml structure: The assistant sees the dependency list for the core crate, including cuzk-proto, tokio, tracing, anyhow, thiserror, serde, and the beginning of the feature-specific dependencies. This reveals how the CUDA features are wired. More importantly, this message creates negative knowledge — it confirms what the assistant doesn't need to investigate further. The Cargo.toml files show no red flags: no version conflicts, no missing features, no unexpected dependencies. This allows the assistant to proceed confidently to the build step. The message also sets up the decision framework for the subsequent investigation. By reading the Cargo.toml files first, the assistant establishes a baseline understanding of the feature system. When it then investigates Blackwell compatibility ([msg 198]–[msg 203]), it can interpret the findings in the context of the actual feature flags available.

The Thinking Process Visible in the Reasoning

The assistant's thinking process in this message is revealed through its actions and the sequence of reads. Several cognitive patterns are visible:

Pattern 1: Decomposition of a complex task. The assistant has broken down "run a real proof" into discrete steps: verify build → build with CUDA → run daemon → submit proof → measure SRS benefit. Each step has clear success criteria. This message executes step 2, but the assistant recognizes that step 2 itself has sub-steps: check features → choose backend → build → verify binary.

Pattern 2: Risk-aware sequencing. The assistant could have built with both CUDA features simultaneously, or tried one at random. Instead, it reads the configuration first, then investigates hardware compatibility, then builds. This minimizes the risk of a long, failing build. If the Cargo.toml had revealed a missing feature or incompatible dependency, the assistant would have caught it before spending minutes compiling.

Pattern 3: Contextual awareness of hardware novelty. The assistant specifically mentions "RTX 5070 Ti" — a Blackwell GPU that was likely released in late 2025 or early 2026. This is cutting-edge hardware. The assistant's caution about "which CUDA backend to use" reflects an awareness that newer hardware may not be supported by older precompiled kernels. This concern is validated in the subsequent messages where rust-gpu-tools fatbins are found to potentially lack sm_120 support.

Pattern 4: Minimal, targeted information gathering. The assistant reads only two files: the workspace root and the core crate. It does not read every Cargo.toml in the workspace, nor does it read the upstream filecoin-proofs-api Cargo.toml. This is efficient — the workspace root shows the member list and shared dependencies, and the core crate shows how features are defined. The server, daemon, and bench crates are unlikely to have CUDA-specific configuration.

The Broader Significance

Message [msg 197] is significant not for what it says, but for what it represents. It is the moment when the cuzk proving engine transitions from "it compiles" to "it runs on real hardware." The assistant could have skipped this step and jumped directly to building with CUDA. Instead, it paused to read the configuration, demonstrating a disciplined engineering approach.

This discipline pays off immediately. In the next messages ([msg 198]–[msg 205]), the assistant discovers that Blackwell compatibility is a genuine concern, investigates both CUDA backends, chooses supraseal-c2 (which uses nvcc JIT compilation rather than precompiled fatbins), and successfully builds a 21 MB daemon binary. The build succeeds with zero errors and zero warnings. The daemon then goes on to produce valid Groth16 proofs, with the first proof completing in 116.8 seconds and the second (with SRS cached) in 92.8 seconds — a 20.5% improvement from SRS residency.

None of that success would have been possible without the careful groundwork laid in this message. Reading the Cargo.toml files was not an exciting step, but it was a necessary one. It is the kind of step that experienced engineers take instinctively and inexperienced engineers skip to their peril.

Conclusion

Message [msg 197] is a study in deliberate engineering. In two sentences and two file reads, the assistant demonstrates a methodical approach to a complex task: decompose, understand, verify, then execute. The message itself is brief, but it sits at the pivot point between a working scaffold and a working proof pipeline. It carries assumptions about hardware compatibility, build system behavior, and feature propagation — assumptions that are validated in the subsequent investigation. And it produces the knowledge needed to make an informed decision about which CUDA backend to use on a cutting-edge Blackwell GPU.

For the reader, this message offers a lesson in engineering process: the most important step in building a complex system is often the one that looks the simplest. Reading the configuration before building is not浪费时间 — it is the difference between guessing and knowing.