The Static That Wasn't: Diagnosing and Fixing a C++ Initialization Order Bug in a Rust–CUDA Proving Pipeline

A Single Line That Tells a Deeper Story

The message at index 1938 in this coding session is deceptively simple:

[assistant] [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu

A single line, an edit command, no fanfare. But behind this unassuming message lies a critical debugging journey that spanned multiple rounds of investigation, spanning process initialization semantics, C++ static construction order, Rust FFI linkage, and the subtle ways that environment variables interact with dynamically loaded libraries. This edit was the culmination of a hard-won insight: that a thread pool, declared as a C++ static global, was being initialized before the Rust main() function had a chance to configure it, silently sabotaging an entire thread-isolation strategy.

The Motivation: Why Thread Isolation Mattered

To understand why this edit was necessary, we must step back to the broader context of the session. The assistant had been working on a high-performance Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) protocol, implemented as a hybrid Rust–C++–CUDA system called cuzk. The pipeline had a structural problem: GPU utilization was poor because CPU-side synthesis (the process of constructing the circuit and generating witness data) and GPU-side proving (the cryptographic heavy lifting) were competing for the same pool of CPU cores.

Earlier benchmarking had revealed a critical bottleneck. When the assistant implemented parallel synthesis (allowing multiple proofs to be synthesized concurrently using a tokio::sync::Semaphore), GPU utilization jumped to 99.3% — but overall throughput improved only modestly, from ~45.3s to ~42.2s per proof. The root cause was CPU resource contention: the GPU prover's b_g2_msm step (a multi-scalar multiplication on the G2 curve) was itself CPU-intensive, and running two full 10-partition syntheses simultaneously starved it of cores.

The proposed solution was thread pool partitioning: reserve a dedicated set of CPU cores for the GPU prover's b_g2_msm work, while allowing the remaining cores to handle synthesis. This required two independent thread pools: a Rayon-based pool for Rust-side synthesis, and a C++ thread_pool_t (from the sppark library) for the GPU-side preprocessing and MSM operations.

The Implementation: Setting the Stage for a Bug

The assistant had already implemented most of the plumbing. In message 1907, they modified groth16_cuda.cu to read a CUZK_GPU_THREADS environment variable that would control the size of the C++ thread pool:

static thread_pool_t groth16_pool([]() -> unsigned int {
    const char* env = getenv("CUZK_GPU_THREADS");
    ...
});

In messages 1909–1911, they added a gpu_threads configuration field to the Rust config struct and wired the daemon's main() function to call std::env::set_var("CUZK_GPU_THREADS", ...) before the engine started. The plan seemed sound: set the environment variable early in main(), and the C++ static constructor would read it during library initialization.

The daemon crashed immediately on the first test run (message 1925). The process died silently, leaving no error message beyond the startup logs. The assistant's initial reaction was confusion: "Daemon died. The issue is likely that the C++ static groth16_pool is initialized at library load time (before main()), so CUZK_GPU_THREADS set in main() is too late."

The Diagnosis: A Subtle Initialization Order Problem

This diagnosis required a deep understanding of process initialization semantics across language boundaries. The assistant walked through the reasoning carefully in messages 1930–1933.

The C++ static variable groth16_pool uses a lambda to read getenv("CUZK_GPU_THREADS") at construction time. In a pure C++ program, static storage duration objects are initialized before main() begins execution — this is guaranteed by the C++ standard. However, the situation is more nuanced when the C++ code is compiled into a Rust binary.

The supraseal-c2 library is linked statically at compile time (not loaded via dlopen). This means its C++ static constructors run during the dynamic linker's initialization phase, which occurs before Rust's main() function is called. At that point, std::env::set_var in Rust hasn't executed yet, so getenv("CUZK_GPU_THREADS") returns NULL, and the thread pool defaults to using all available CPUs.

The assistant confirmed this experimentally. When they launched the daemon with CUZK_GPU_THREADS=32 set as a shell environment variable (message 1930), it ran successfully. When they launched it without the environment variable (message 1932), relying on the Rust set_var call in main(), the daemon died. The difference was clear: the environment variable needed to exist before the process started, not be set during its execution.

This is a classic instance of what computer scientists call a static initialization order fiasco — a well-known hazard in C++ where the order of initialization across translation units is undefined. Here, the fiasco manifested across language boundaries: Rust's initialization (which happens in main()) cannot influence C++ static initialization (which happens before main()).

The Decision: Lazy Initialization via std::call_once

The assistant considered two solutions:

  1. Shell-level environment variable: Require users to set CUZK_GPU_THREADS before launching the daemon. This works but is inconvenient and error-prone — it pushes configuration responsibility onto the operator and makes the daemon's behavior dependent on external state.
  2. Lazy initialization: Replace the static constructor with a function that lazily initializes the pool on first use, using std::call_once to ensure thread safety. This allows the environment variable to be set at any point before the pool is actually needed — which could be after Rust's main() has configured it. The assistant chose option 2, and message 1934 was the first edit to implement this change. The new code pattern was:
static thread_pool_t* groth16_pool_ptr = nullptr;
static std::once_flag  groth16_pool_init_flag;

static thread_pool_t& get_groth16_pool() {
    std::call_once(groth16_pool_init_flag, []() {
        const char* env = getenv("CUZK_GPU_THREADS");
        unsigned int num_threads = /* ... parse env or default ... */;
        groth16_pool_ptr = new thread_pool_t(num_threads);
    });
    return *groth16_pool_ptr;
}

This pattern defers pool creation until the first call to get_groth16_pool(), which happens during actual proving work — well after Rust's main() has had a chance to call set_var. The std::call_once primitive guarantees that the initialization lambda runs exactly once, even under concurrent access from multiple threads.

Message 1938: The Pivot Point

This brings us to message 1938. The assistant had already:

The Assumptions That Failed

This debugging episode reveals several assumptions that proved incorrect:

Assumption 1: std::env::set_var in Rust modifies the process environment before C++ statics run. This is false when the C++ code is statically linked. The C++ static constructors execute during the linker's initialization phase, which precedes main(). The Rust runtime itself is initialized during this phase, but Rust's main() body — including any set_var calls — hasn't executed yet.

Assumption 2: The daemon crash was a fluke. The assistant initially wondered if the daemon was killed by a previous process (message 1931: "The real issue with the previous attempt was likely that the daemon process was killed before it could start"). Systematic testing disproved this: the daemon consistently died without the env var and consistently lived with it.

Assumption 3: A static constructor with a lambda is equivalent to lazy initialization. The original code used a lambda to compute the pool size, but because it was still a static constructor, the lambda ran at library load time. The lambda's deferred computation pattern was misleading — it looked like lazy evaluation but wasn't lazy initialization.

Input Knowledge Required

To understand this message and the surrounding debugging session, one needs:

Output Knowledge Created

This message and the surrounding edits produced:

The Broader Significance

This episode is a textbook example of a cross-language initialization order bug — a class of problem that becomes increasingly common as systems grow to span multiple programming languages. The Rust ecosystem's growing use of C/C++ libraries via FFI makes this a recurring pattern: Rust code tries to configure a C/C++ component, only to discover that the C/C++ component initialized itself before Rust had a chance to speak.

The fix — replacing static initialization with lazy initialization — is a well-known pattern in C++ itself, where it's used to avoid the static initialization order fiasco within a single language. Here, the same pattern solves an analogous problem across languages. The std::call_once primitive, introduced in C++11, provides a clean, thread-safe mechanism for deferred initialization that integrates naturally with both Rust's startup sequence and the proving pipeline's runtime.

Message 1938, for all its brevity, represents the moment when a subtle cross-language initialization bug was definitively fixed — not by adding complexity, but by applying a well-understood pattern from one language to solve a problem that emerged at the boundary between two.