The GPU Module Dispatch: Tracing Bellperson's Conditional Compilation at the Heart of Filecoin Proving

Introduction

In the depths of a sprawling investigation into Filecoin's proof generation pipeline, a single bash command reveals a critical architectural seam. At message index 56, the assistant executes:

sed -n '1,40p' /home/theuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bellperson-0.26.0/src/gpu/mod.rs 2>&1

This command prints the first forty lines of a Rust module file. On its surface, it is a trivial file read. But in the context of the broader investigation—a deep-dive into the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) and related proof types—this message is a deliberate probe into the conditional compilation machinery that determines whether proof computations run on a GPU or fall back to the CPU. Understanding this boundary is essential to the investigation's goals: mapping the ~200 GiB peak memory footprint, identifying structural bottlenecks, and designing a pipelined proving daemon architecture.

The Context: A Hunt for the Proving Dispatch

To understand why this particular message was written, one must trace the investigation's trajectory through the preceding messages. The session began with an exploration of parameter file sizes for SnapDeals, WindowPoSt, and WinningPoSt circuits ([msg 33]–[msg 38]). The assistant discovered that the Curio task scheduler declares resource requirements such as Ram: 50 << 30 (50 GiB) for SnapDeals proving tasks ([msg 38]), hinting at the enormous memory footprint that motivated the investigation.

The critical turning point came when the assistant shifted focus from what resources are needed to how the proving code is dispatched. In messages 39 through 55, the assistant traced the bellperson library's prover selection logic. The key discovery was in groth16/prover/mod.rs ([msg 50]–[msg 51]):

#[cfg(not(feature = "cuda-supraseal"))]
mod native;
#[cfg(feature = "cuda-supraseal")]
mod supraseal;

This conditional compilation block is the architectural fork: when the cuda-supraseal feature is enabled, bellperson delegates all proof generation to the C++ SupraSeal backend via FFI. When it is not enabled, the native Rust prover handles everything. The native prover itself has its own GPU acceleration path via CUDA or OpenCL features, but the supraseal path bypasses that entirely.

This discovery raised an immediate question: How does the native prover decide between GPU and CPU execution? The answer lies in the gpu/mod.rs file that the assistant now reads.

What the Message Reveals: The GPU Module's Conditional Structure

The forty lines printed by the command show the top of bellperson-0.26.0/src/gpu/mod.rs:

mod error;

pub use self::error::*;

#[cfg(any(feature = "cuda", feature = "opencl"))]
mod locks;

#[cfg(any(feature = "cuda", feature = "opencl"))]
pub use self::locks::*;

#[cfg(any(feature = "cuda", feature = "opencl"))]
mod multiexp;

#[cfg(any(feature = "cuda", feature = "opencl"))]
pub use self::multiexp::CpuGpuMultiexpKernel;

#[cfg(not(any(feature = "cuda", feature = "opencl")))]
mod nogpu;

#[cfg(not(any(feature = "cuda", feature = "opencl")))]
pub use self::nogpu::*;

// This is a hack...

This is a textbook example of Rust's conditional compilation for hardware-accelerated computation. The module defines two mutually exclusive code paths:

GPU path (when cuda or opencl feature is enabled): It brings in locks (for GPU resource synchronization) and multiexp (providing CpuGpuMultiexpKernel for multi-exponentiation that can run on either CPU or GPU). This path assumes the presence of GPU runtime libraries and kernel code.
CPU-only path (when neither cuda nor opencl is enabled): It brings in nogpu, a fallback module that presumably provides pure CPU implementations of the same operations. The comment "This is a hack..." (truncated at line 40) hints at the engineering tension: the trait system needs to present a uniform interface regardless of whether GPU support is compiled in, but the underlying implementations differ radically. This is a common pattern in cryptographic libraries where the same mathematical operations must be available on diverse hardware.

The Reasoning Behind the Probe

The assistant wrote this command for a specific investigative reason. Having established that the cuda-supraseal feature flag is the master switch between the C++ SupraSeal backend and the native Rust prover, the next logical question is: Within the native prover, what determines GPU usage? The answer is the cuda and opencl features, which control the gpu module's behavior.

But the investigation's deeper purpose is to understand the memory implications of these choices. The native prover's GPU path uses CpuGpuMultiexpKernel for multi-exponentiation and presumably GPU-accelerated NTT (Number Theoretic Transform). These operations are memory-intensive, and the GPU path may have different memory allocation patterns than the CPU fallback. The supraseal path, being a completely separate C++ implementation, has its own memory management entirely.

By reading the gpu/mod.rs header, the assistant is mapping the decision tree:

bellperson create_proof()
    └─ feature "cuda-supraseal"?
        ├─ YES → supraseal (C++ FFI, separate memory model)
        └─ NO  → native (Rust)
                  └─ feature "cuda" or "opencl"?
                      ├─ YES → gpu::CpuGpuMultiexpKernel
                      └─ NO  → gpu::nogpu (CPU-only)

This decision tree directly impacts the memory footprint analysis that is the investigation's core concern. The ~200 GiB peak memory might arise from the supraseal path, the native GPU path, or both—but in different ways.

Assumptions and Knowledge Requirements

The assistant makes several assumptions in writing this message. First, that the gpu/mod.rs file is the correct location for the GPU/CPU dispatch logic—an assumption validated by the grep results in [msg 55] which found references to BELLMAN_NO_GPU, LockedFftKernel, and LockedMultiexpKernel in this file. Second, that reading only the first forty lines is sufficient to understand the module's structure—a reasonable heuristic for a Rust module file where cfg attributes appear at the top. Third, that the nogpu module provides CPU-only implementations of the same interfaces—an inference from the naming and the cfg(not(...)) guard.

To fully understand this message, the reader needs knowledge of: Rust's conditional compilation system (#[cfg(...)]), the bellperson library's role in Filecoin proving, the distinction between the cuda-supraseal feature and the cuda/opencl features, and the broader context of the investigation into memory bottlenecks. The reader must also understand that this is one piece of a larger puzzle—the assistant is not just reading a file for its own sake but building a mental model of the entire proof generation pipeline.

The Thinking Process Visible in the Message

The message itself is terse—a single bash command—but the thinking process is visible in the sequence of messages that precede it. In [msg 55], the assistant ran a grep command searching for BELLMAN_NO_GPU, LockedFftKernel, LockedMultiexpKernel, and other GPU-related identifiers in gpu/mod.rs. That grep returned references to conditional compilation blocks. Now, in message 56, the assistant reads the actual source to see the structure firsthand.

The progression shows a methodical investigative approach:

Discover the fork ([msg 50]–[msg 51]): Find that cuda-supraseal selects between native and supraseal provers.
Read the supraseal prover ([msg 52]–[msg 54]): Understand how the C++ backend is invoked.
Probe the GPU module ([msg 55]): Grep for GPU-related identifiers to locate the dispatch logic.
Read the GPU module header ([msg 56]): Confirm the conditional compilation structure. This is classic reverse-engineering: follow the call chain, identify decision points, and read the relevant source files to understand the branching logic. The assistant is constructing a map of the proving pipeline, and each message adds a new layer of detail.

Output Knowledge Created

This message produces concrete knowledge: the conditional compilation structure of bellperson's GPU module. Specifically, it confirms that:

The gpu module has two mutually exclusive sub-module trees: one for GPU-enabled builds (locks + multiexp) and one for CPU-only builds (nogpu).
The CpuGpuMultiexpKernel type is the GPU-accelerated multi-exponentiation kernel, available only when CUDA or OpenCL is enabled.
The nogpu module provides fallback implementations for builds without GPU support. This knowledge feeds directly into the investigation's output documents. The background reference document mapping the call chain can now include the GPU module as a decision point. The bottleneck analysis can distinguish between memory usage patterns in the GPU vs CPU paths. The optimization proposals can target specific modules: for example, the nogpu path might be eliminated entirely if all production deployments have GPUs, or the CpuGpuMultiexpKernel could be optimized for the specific GPU architectures used in Filecoin mining.

Potential Missteps and Limitations

The message has a minor limitation: it only reads the first forty lines. The file likely contains additional conditional compilation blocks, trait definitions, and function signatures that are relevant but not shown. The assistant would need to read more of the file to understand, for example, how CpuGpuMultiexpKernel is implemented, how the nogpu module provides equivalent functionality, and how the "hack" mentioned in the comment works around trait system limitations.

However, this is not a mistake—it is a deliberate scoping decision. The assistant is building understanding incrementally. Reading the full file at once would risk information overload. By reading the header first, the assistant establishes the module's structure before diving into implementation details in subsequent messages.

Conclusion

Message 56 is a small but essential step in a much larger investigation. It captures the moment when the assistant confirms the conditional compilation boundary between GPU-accelerated and CPU-only proving within bellperson's native prover. This boundary is one of the key architectural seams that the investigation seeks to understand, document, and ultimately optimize.

The message exemplifies the investigative method: trace the call chain, identify decision points, read the source, and build a mental model incrementally. Each bash command is a probe that extracts a piece of the puzzle. The forty lines of gpu/mod.rs are not just code—they are evidence of design decisions about hardware abstraction, performance portability, and the trade-offs between GPU acceleration and CPU fallback. In the context of a ~200 GiB memory footprint investigation, understanding these decisions is the first step toward knowing where the memory goes and how to reduce it.