Tracing the GPU Fallback Chain: A Pivotal Investigation into Bellperson's Native Prover

Introduction

In the middle of a deep-dive investigation into Filecoin's SUPRASEAL_C2 Groth16 proof generation pipeline, a single message captures a critical moment of discovery. At message index 59 in the conversation, the assistant executes a targeted grep command against the bellperson library's native prover source code, searching for the GPU fallback logic. This seemingly simple command — searching for keywords like multiexp, fft_kern, gpu, GpuError, Disabled, fallback, lock, and BELLMAN_NO_GPU — represents a turning point in the investigation. The agent is no longer exploring the high-level architecture or parameter sizes; it is now tracing the precise mechanism by which the proving system decides whether to use GPU acceleration or fall back to CPU-only computation.

Context: The Investigation Leading Up to This Moment

To understand why message 59 matters, one must appreciate the journey that preceded it. The investigation began with a broad exploration of Filecoin proof parameters — the .params and .vk files that define circuit sizes for different proof types. The agent discovered that SnapDeals (empty sector update) .params files reached 626 MB on disk, while WindowPoSt and WinningPoSt parameters were significantly smaller ([msg 36]). This led to examining Curio task resource definitions, where the agent found that WdPost tasks requested 25 GB of RAM and UpdateProve tasks requested 50 GB ([msg 38]). These numbers hinted at the massive memory footprint of Groth16 proving.

The investigation then pivoted to the critical question: how does bellperson decide between the SupraSeal C++ prover and the standard GPU-accelerated Rust prover? The agent traced through bellperson's prover/mod.rs and discovered a compile-time dispatch mechanism (<msg id=50-51>):

#[cfg(not(feature = "cuda-supraseal"))]
mod native;
#[cfg(feature = "cuda-supraseal")]
mod supraseal;

#[cfg(not(feature = "cuda-supraseal"))]
use self::native as prover;
#[cfg(feature = "cuda-supraseal")]
use self::supraseal as prover;

This was a pivotal finding: the choice between SupraSeal and the native Rust prover is not a runtime decision. It is baked in at compile time via the cuda-supraseal Cargo feature flag. When that feature is enabled, bellperson links against the supraseal-c2 C++ library and uses its batched proving pipeline. When disabled, it falls through to the native Rust prover, which itself can optionally use GPU acceleration via the ec-gpu-gen library's CUDA or OpenCL backends.

But this raised a further question: what happens within the native prover when GPU acceleration is unavailable? Does it crash? Does it silently fall back to CPU? Message 59 is the agent's attempt to answer that question.

The Message Itself: A Targeted Search

The agent executes:

grep -n "multiexp\|fft_kern\|gpu\|GPU\|cpu\|CPU\|GpuError\|Disabled\|fallback\|lock\|BELLMAN_NO_GPU" \
  /home/theuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bellperson-0.26.0/src/groth16/prover/native.rs \
  2>&1 | head -50

The search terms are carefully chosen. Each one targets a different aspect of the GPU fallback architecture:

multiexp / fft_kern: These are the two core computational kernels in Groth16 proving. Multi-exponentiation (MSM) and Number Theoretic Transform (NTT/FFT) are the operations that benefit most from GPU acceleration. The agent wants to see how these are dispatched.
gpu / GPU / cpu / CPU: Broad searches for any GPU or CPU path references.
GpuError: Error handling for GPU failures — a critical part of any fallback mechanism.
Disabled: Likely refers to a GPU-disabled state or a GpuDisabled error type.
fallback: The explicit fallback mechanism, if one exists.
lock: The priority lock mechanism for GPU access, which the agent had seen referenced in earlier exploration.
BELLMAN_NO_GPU: The environment variable that users can set to disable GPU acceleration at runtime — a key part of the fallback story.

What the Search Revealed

The grep output shows the skeleton of the native prover's GPU interaction:

10:use ec_gpu_gen::{
11:    multiexp_cpu::FullDensity,
26:use crate::gpu::PriorityLock;
29:    gpu::{GpuError, GpuName, LockedFftKernel, LockedMultiexpKernel},
30:    multiexp::multiexp,
94:    let prio_lock = if priority {
95:        trace!("acquiring priority lock");
96:        Some(PriorityLock::lock())
114:        let mut fft_kern = Some(LockedFftKernel::new(priority));
116:            a_s.push(execute_fft(worker, prover, &mut fft_kern)?);
121:    let mut multiexp_g1_kern = LockedMultiexpKer...

Line 10-11 shows that ec_gpu_gen::multiexp_cpu::FullDensity is imported — this is the CPU-based multi-exponentiation implementation, always available as a fallback. Line 26 imports PriorityLock, a synchronization mechanism for GPU access. Line 29 imports the GPU kernel types (LockedFftKernel, LockedMultiexpKernel) alongside GpuError — the error type that would signal GPU unavailability. Line 30 imports the multiexp function, which (as the agent would later discover in message 61) attempts GPU acceleration and falls back to CPU on error.

The priority lock mechanism (lines 94-96) is particularly interesting. It suggests that GPU access in the native prover is mediated by a priority-based locking system — higher-priority tasks can acquire the GPU preferentially. This is relevant to Filecoin's Curio scheduler, where different proof tasks (WinningPoSt, WindowPoSt, SnapDeals) may have different priority levels.

The Thinking Process Visible in This Message

Message 59 reveals a methodical, systematic approach to understanding a complex software system. The agent is not guessing or hypothesizing; it is reading source code directly. The thinking process can be reconstructed as follows:

The agent has already identified the two prover paths (native vs. supraseal) and the compile-time dispatch mechanism. Now it needs to understand the internal architecture of the native prover.
The agent recognizes that the native prover itself has a GPU sub-system. The imports in native.rs reference LockedFftKernel, LockedMultiexpKernel, and GpuError from crate::gpu. This means there is a GPU abstraction layer within bellperson itself, separate from the SupraSeal integration.
The agent wants to trace the fallback chain. If GPU initialization fails, or if BELLMAN_NO_GPU is set, what happens? Does the prover crash with GpuError, or does it silently fall back to CPU-based NTT and MSM?
The search terms are chosen to reveal this fallback chain. GpuError would appear in error handling paths. Disabled might be a variant of GpuError. fallback would be an explicit fallback mechanism. BELLMAN_NO_GPU is the runtime disable flag.
The results show the key types but not the complete fallback logic. The agent sees GpuError is imported, LockedFftKernel and LockedMultiexpKernel are used, and the priority lock mechanism exists. But the grep output is truncated at head -50 and cuts off at line 121. The agent will need to continue investigating — which it does in subsequent messages (60-61) by examining multiexp.rs and gpu/mod.rs.

Assumptions and Potential Misconceptions

The agent makes several assumptions in this message:

That the native prover has a well-defined GPU fallback mechanism. This is a reasonable assumption given the existence of GpuError and the conditional compilation in gpu/mod.rs (which the agent had seen in message 56). However, the agent may be assuming the fallback is graceful when in reality it might be a crash-and-retry pattern.
That BELLMAN_NO_GPU is the primary runtime control. This is correct — it is a well-known environment variable in the bellman/bellperson ecosystem. But the agent may not yet know whether it's checked at proof time or at initialization time.
That the priority lock mechanism is relevant to GPU fallback. The agent seems to be treating the priority lock as part of the fallback story, when it may be purely a concurrency control mechanism unrelated to CPU fallback.
That the fallback chain is linear. The agent may be expecting a simple if-else chain (try GPU, catch error, fall back to CPU). In reality, the fallback may be more nuanced — the GPU kernels might be optional at compile time (via cfg flags), with the nogpu.rs polyfill providing CPU implementations when GPU support is not compiled in.

Input Knowledge Required to Understand This Message

To fully grasp what message 59 is doing, a reader needs:

Knowledge of Groth16 proving: Understanding that the core computational steps are NTT (FFT) and MSM (multi-exponentiation), and that these are the operations that benefit from GPU acceleration.
Knowledge of the bellperson architecture: Understanding that bellperson has two prover backends — a native Rust implementation and a SupraSeal C++ integration — selected at compile time via feature flags.
Knowledge of ec-gpu-gen: Understanding that this library provides GPU-accelerated NTT and MSM kernels with a CPU fallback, and that it is the GPU layer used by bellperson's native prover.
Knowledge of Filecoin proof types: Understanding that different proof types (WinningPoSt, WindowPoSt, SnapDeals) have different circuit sizes and resource requirements, which motivates the need for GPU acceleration and fallback mechanisms.
Familiarity with Rust conditional compilation: Understanding how #[cfg(feature = "...")] works and how the nogpu.rs polyfill provides CPU implementations when GPU features are disabled.

Output Knowledge Created by This Message

Message 59 produces several concrete findings:

The native prover imports both CPU and GPU multiexp implementations. The presence of multiexp_cpu::FullDensity alongside LockedMultiexpKernel confirms that both paths are available.
The native prover uses a priority lock for GPU access. The PriorityLock::lock() call suggests that GPU access is mediated by a priority system, which is relevant to Curio's task scheduling.
The native prover creates LockedFftKernel and LockedMultiexpKernel instances. These are the GPU-accelerated kernel wrappers. The Locked prefix suggests they acquire the GPU lock on creation and release it on destruction.
GpuError is imported and used in the error handling path. This is the error type that would signal GPU failures, potentially triggering a fallback.
The grep output is truncated. The agent only sees the first 50 lines of output, and the critical line 121 is cut off. This creates a knowledge gap that the agent will need to fill with further investigation.

The Broader Significance

Message 59 is significant because it represents the moment when the investigation shifts from what the system does to how it does it. The earlier messages established the high-level architecture — the parameter files, the task resource requirements, the compile-time dispatch between native and supraseal provers. Message 59 digs into the implementation details of the native prover's GPU interaction, which is essential for understanding:

How memory is managed: GPU kernels typically require device memory allocation. Understanding the GPU fallback chain is necessary for accurate memory accounting.
How performance varies: The GPU vs. CPU decision has enormous performance implications. A proof that takes seconds on GPU might take minutes on CPU.
How the system behaves on different hardware: Cloud instances may or may not have GPUs. Understanding the fallback mechanism is critical for deployment planning.
How the priority system works: In Curio's task scheduler, different proof types compete for GPU resources. The priority lock mechanism mediates this competition. The message also demonstrates a methodological principle: when investigating a complex system, trace the code paths. Don't rely on documentation or assumptions — read the source code. The agent's use of grep to find relevant lines, followed by targeted reads of specific files, is a textbook approach to software archaeology.

Conclusion

Message 59 is a small but crucial step in a larger investigation. It captures the agent's systematic effort to understand the GPU fallback chain in bellperson's native prover — a chain that determines whether Filecoin proofs run on GPU or CPU, and how gracefully the system degrades when GPU acceleration is unavailable. The search terms reveal the agent's mental model of the fallback architecture: a priority-locked GPU kernel system with GpuError handling and a BELLMAN_NO_GPU environment variable override. The truncated output creates a cliffhanger that drives the subsequent investigation into multiexp.rs and gpu/mod.rs. In the broader context of the SUPRASEAL_C2 deep-dive, this message is one of the key pieces that will inform the agent's understanding of the complete proof generation pipeline — from Curio's Go task layer through Rust FFI into C++/CUDA kernels, with all the GPU fallback paths accounted for.