The Critical Feature Flag: How cuda-supraseal Rewires Filecoin's Proof Pipeline

In the course of a deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof of Replication (PoRep) system, a single message marks a pivotal turning point. Message 67 of the conversation captures the moment when the assistant, after hours of tracing through layers of Go, Rust, and C++ code, finally reads the raw feature definitions in a Cargo.toml file and exclaims: "This is absolutely critical!"

The message itself is deceptively simple — a bash command and its output:

[assistant] This is absolutely critical! Let me read the full feature definition more carefully: [bash] sed -n '146,185p' /home/theuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/filecoin-proofs-18.1.0/Cargo.toml 2>&1 [features] asm = ["storage-proofs-core/asm"] big-tests = [] cpu-profile = ["gperftools"] cuda = [ "storage-proofs-core/cuda", "storage-proofs-porep/cuda", "storage-proofs-post/cuda", "storage-proofs-update/cuda", "bellperson/cuda", "filecoin-hashers/cuda", ] cuda-supraseal = [ "storage-proofs-core/cuda-supraseal", "storage-proofs-porep/cuda", "storage-proofs-post/cuda", "storage-proofs-update/cuda", "bellperson/cuda-supraseal", "filecoin-hashers/cu...

But beneath this simple surface lies a discovery that fundamentally reshapes the investigation's understanding of how Filecoin's proof system operates — and why it consumes roughly 200 GiB of peak memory.

The Context: Tracing a Memory Mystery

The broader investigation had been tracking down the root causes of SUPRASEAL_C2's enormous memory footprint. The assistant had already established ([msg 50]-[msg 51]) that bellperson's prover dispatch uses a compile-time conditional: when the cuda-supraseal feature is enabled, the entire Groth16 prover module is replaced with the C++ Supraseal implementation; otherwise, the native Rust prover (which may still use GPU via ec-gpu CUDA kernels) is used. The code in bellperson-0.26.0/src/groth16/prover/mod.rs is unambiguous:

#[cfg(not(feature = "cuda-supraseal"))]
mod native;
#[cfg(feature = "cuda-supraseal")]
mod supraseal;

#[cfg(not(feature = "cuda-supraseal"))]
use self::native as prover;
#[cfg(feature = "cuda-supraseal")]
use self::supraseal as prover;

This is a complete prover swap — not an additive GPU acceleration. When cuda-supraseal is active, the native Rust prover (with its ec-gpu CUDA kernels for NTT and MSM) is entirely replaced by the C++ Supraseal implementation. This raises an immediate question: what else changes in the dependency tree when this feature flag is toggled?

The Discovery: Asymmetric Feature Propagation

Message 67 answers that question by reading the feature definitions in filecoin-proofs-18.1.0/Cargo.toml. The output reveals a critical asymmetry that the assistant immediately recognizes as "absolutely critical."

The cuda feature propagates uniformly: every sub-package (storage-proofs-core, storage-proofs-porep, storage-proofs-post, storage-proofs-update, bellperson, filecoin-hashers) gets its corresponding cuda feature. This is a straightforward, homogeneous GPU acceleration path.

The cuda-supraseal feature, however, is anything but uniform:

Why This Matters: The Dual GPU Architecture

This asymmetric feature propagation reveals a dual GPU architecture that was not previously obvious. When cuda-supraseal is enabled, two different GPU compute paths coexist:

  1. Supraseal (C++ CUDA) handles the Groth16 proving itself — the multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations that generate the final proof. This is the path that loads the ~200 GiB SRS (Structured Reference String) into GPU memory.
  2. Standard ec-gpu CUDA kernels handle the circuit-specific computations for PoRep, PoSt, and update proofs — the constraint system synthesis and evaluation that happens before the Groth16 prover is invoked. This dual-path architecture has profound implications for memory accounting. The ~200 GiB peak memory isn't just the Supraseal SRS; it's the combination of Supraseal's GPU memory plus the standard GPU kernels' memory plus the CPU-side circuit synthesis buffers. The assistant's discovery in message 67 provides the architectural map needed to properly attribute memory usage across these two paths.

The Reasoning Process: From Confusion to Clarity

The assistant's journey to this discovery reveals a methodical investigative process. Earlier messages show the assistant trying to understand how the GPU is used by examining the native prover's fallback logic ([msg 59]-[msg 61]), the GPU module structure ([msg 55]-[msg 58]), and the multiexp dispatch ([msg 60]-[msg 61]). The assistant was building a mental model of a system where GPU acceleration could come from multiple sources.

The breakthrough came when the assistant traced the feature propagation chain from filecoin-proofs-api ([msg 65]) through filecoin-proofs ([msg 66]) and finally read the full feature definitions in message 67. The exclamation "This is absolutely critical!" signals the moment when the pieces clicked into place: the feature flags are not simply additive; they create a heterogeneous acceleration landscape where different parts of the proof pipeline use different GPU backends.

Assumptions and Corrections

Prior to this discovery, one might reasonably assume that cuda-supraseal simply adds Supraseal on top of the existing CUDA support — that it's a superset of the cuda feature. The feature definitions prove otherwise. cuda-supraseal does not include bellperson/cuda; it includes bellperson/cuda-supraseal instead. This means the standard bellperson CUDA kernels (the LockedMultiexpKernel and LockedFftKernel seen in [msg 59]) are completely replaced, not supplemented.

Another potential misconception is that Supraseal handles all GPU work. The feature definitions show this is false: PoRep, PoSt, and update proofs continue to use standard CUDA kernels. Only the Groth16 proving itself is replaced.

Input Knowledge Required

To fully understand message 67, one needs:

  1. Cargo feature system: Understanding that Rust features are additive and propagate through dependencies. A feature like cuda-supraseal in filecoin-proofs activates corresponding features in its dependencies.
  2. Bellperson's prover architecture: The knowledge (established in messages 50-51) that bellperson uses conditional compilation to swap between native and supraseal prover modules based on the cuda-supraseal feature.
  3. Filecoin proof pipeline layers: Awareness that the proof system has multiple layers — circuit synthesis (PoRep, PoSt), Groth16 proving, and hashing — each of which may use GPU acceleration independently.
  4. The investigation's goal: Understanding that the assistant is tracing memory usage to explain the ~200 GiB peak memory footprint of SUPRASEAL_C2.

Output Knowledge Created

Message 67 produces several critical pieces of knowledge:

  1. The heterogeneous feature map: A precise understanding of which sub-packages receive which features under cuda vs cuda-supraseal. This is the architectural blueprint for understanding GPU usage in the Filecoin proof pipeline.
  2. The dual-GPU hypothesis: Evidence that two separate GPU compute paths coexist when cuda-supraseal is enabled, which explains why memory accounting must consider both Supraseal's SRS and the standard GPU kernels' allocations.
  3. A refined memory model: The insight that peak memory is not a single monolithic allocation but the sum of potentially overlapping allocations from two different GPU frameworks.
  4. A debugging roadmap: The knowledge that optimizing memory usage requires understanding both paths — you cannot simply eliminate one without understanding what the other provides.

The Thinking Process in Action

What makes message 67 fascinating is the visible thinking process. The assistant doesn't just run a command and move on; the phrase "This is absolutely critical!" reveals the cognitive leap occurring in real-time. The assistant had been building a model of the system across dozens of commands, and this single feature definition was the missing piece that made the model coherent.

The decision to read lines 146-185 specifically (rather than the full file) shows targeted reasoning: the assistant already knew from message 66 that lines 146+ contained the [features] section and that the cuda-supraseal definition started at line 158. The sed -n '146,185p' command was a precision strike to get the complete feature definitions without noise.

The truncation of the output at "filecoin-hashers/cu..." is also telling — the assistant didn't need to see the rest. The critical asymmetry was already visible in the first few lines of the cuda-supraseal definition. The pattern was clear: porep, post, and update all get cuda while only core and bellperson get cuda-supraseal.

Conclusion

Message 67 is a classic example of a "eureka moment" in systems investigation. A single file, a single command, a single feature definition transforms the investigator's understanding from a vague notion of "Supraseal uses GPU" to a precise architectural model of a heterogeneous, dual-path GPU acceleration system. This discovery ripples through the rest of the investigation, informing memory accounting, bottleneck identification, and ultimately the optimization proposals that follow. It is a reminder that in complex systems, the most profound insights often come not from running sophisticated analyses but from reading the configuration files that define how the system is assembled.