The Static Initialization Trap: Debugging C++/Rust Initialization Order in a CUDA Proving Engine

The Message

[assistant] [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu Edit applied successfully.

This message, <msg id=1942>, is deceptively brief. On its surface, it is merely a confirmation that an edit tool applied changes to a CUDA/C++ source file. But to understand why this particular edit matters—and why it was the final edit in a chain of five modifications to the same file—requires reconstructing the debugging journey that led here. The message represents the last step in a fix for a subtle cross-language initialization ordering bug that had caused the entire proving daemon to crash on startup.

The Context: Thread Pool Partitioning for GPU Proving

The broader session was focused on optimizing the cuzk SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) protocol. The proving pipeline involves two major CPU-bound phases running alongside GPU computation: synthesis (constructing the circuit witness for 10 PoRep partitions, taking ~38s total) and GPU proving (the Groth16 proof itself, taking ~27s). Earlier benchmarks had revealed a structural GPU idle gap: because synthesis was strictly sequential, the GPU sat idle for ~12s between proof cycles waiting for the next batch of synthesized circuits.

The assistant's response was to implement parallel synthesis—allowing multiple proofs to be synthesized concurrently using a tokio::sync::Semaphore ([msg 1906]). This successfully saturated GPU utilization to 99.3%, but it introduced a new problem: CPU contention. The GPU prover's most CPU-intensive step, b_g2_msm (a multi-scalar multiplication on the G2 curve), runs on a C++ CPU thread pool embedded in the CUDA code. When parallel synthesis was running simultaneously, the CPU threads used by b_g2_msm competed with synthesis threads, inflating both synthesis and GPU times.

The solution was thread pool partitioning: limit the GPU prover's C++ thread pool to a subset of CPU cores (e.g., 32 threads on a 96-core machine), reserving the remaining cores for synthesis. This required modifying the CUDA/C++ code in groth16_cuda.cu to read a CUZK_GPU_THREADS environment variable controlling the pool size ([msg 1907]).

The Bug: Static Initialization Order

The initial implementation used a C++ static constructor to initialize the thread pool:

static thread_pool_t groth16_pool([]() -> unsigned int {
    const char* env = getenv("CUZK_GPU_THREADS");
    // ...
});

The Rust daemon's main() function would set CUZK_GPU_THREADS via std::env::set_var() before any proving work began ([msg 1911]). The assumption was straightforward: set the environment variable early in main(), and the C++ code would read it when the thread pool was first accessed.

This assumption was wrong, and the daemon crashed on startup ([msg 1930]). The assistant's debugging process reveals a careful chain of reasoning:

1. Observation: The daemon died when CUZK_GPU_THREADS was set from Rust's main(), but worked when set as an actual environment variable before process launch (<msg id=1930-1931>).
2. Hypothesis: The C++ static constructor runs at library load time, before Rust's main() executes. Since the supraseal-c2 library is linked at compile time (not dynamically loaded via dlopen), its static initializers run during process startup, before any Rust code has a chance to call setenv().
3. Verification: The assistant confirmed this by launching the daemon with CUZK_GPU_THREADS=32 prepended to the command—the environment variable was inherited from the shell's environment, which exists before any code runs. The daemon started successfully ([msg 1931]).
4. Root cause confirmed: std::env::set_var() in Rust calls the POSIX setenv() function, which modifies the process environment at runtime. But C++ static constructors for statically-linked libraries execute during the ELF initialization phase (.init_array / .init sections), which runs before main(). By the time Rust's main() executes setenv(), the C++ static has already called getenv() and captured the (empty) value. This is a classic initialization order fiasco, but one that is particularly insidious in mixed-language projects. The C++ standard guarantees that static objects within a single translation unit are initialized in declaration order, but says nothing about when they initialize relative to Rust's runtime. The Rust reference does not document the ordering of C++ static initializers relative to Rust main(), making this an undocumented behavioral dependency.

The Fix: Lazy Initialization

The assistant's fix was to replace the static constructor with a lazy-initialized singleton using std::call_once ([msg 1934]):

static thread_pool_t* groth16_pool_ptr = nullptr;
static std::once_flag  groth16_pool_init_flag;

static thread_pool_t& get_groth16_pool() {
    std::call_once(groth16_pool_init_flag, []() {
        const char* env = getenv("CUZK_GPU_THREADS");
        unsigned int num_threads = ...;
        groth16_pool_ptr = new thread_pool_t(num_threads);
    });
    return *groth16_pool_ptr;
}

This defers thread pool construction to the first call of get_groth16_pool(), which happens during actual proving work—well after Rust's main() has set the environment variable. The std::call_once primitive ensures thread-safe single initialization without the overhead of a mutex on the fast path.

However, this change introduced a mechanical problem: every reference to the old groth16_pool variable (used via groth16_pool.par_map(...) or &groth16_pool) needed to be replaced with get_groth16_pool().par_map(...) or &get_groth16_pool(). The assistant used grep to locate all 13 occurrences across the file (<msg id=1935-1936>), then applied a series of edits to replace them.

Message 1942: The Final Edit

Message <msg id=1942> is the last of these replacement edits. It follows two prior edits in messages <msg id=1940> and <msg id=1941> that performed bulk replacements of groth16_pool. → get_groth16_pool(). and &groth16_pool → &get_groth16_pool(). By the time this message is sent, all references have been updated, the lazy initialization pattern is complete, and the fix is ready for compilation and testing.

The brevity of the message—a single line confirming the edit tool's success—belies the depth of reasoning that preceded it. The assistant had to:

1. Diagnose a crash with no error message other than "DAEMON DIED"
2. Formulate a hypothesis about C++ static initialization ordering vs Rust runtime startup
3. Test the hypothesis by comparing env-var-preset vs env-var-set-in-main behavior
4. Design a fix using lazy initialization with std::call_once
5. Implement the fix across 13 reference sites in a CUDA/C++ file

Assumptions and Mistakes

The primary mistake was the assumption that std::env::set_var() in Rust's main() would execute before C++ static constructors. This assumption is reasonable—it follows the intuitive model that main() is the program's entry point. But it fails to account for the C++ runtime model, where static constructors in statically-linked libraries execute during the loader's initialization phase, before main() is ever called.

This is not a bug in Rust or C++ per se, but a mismatch in expectations between the two language runtimes. The Rust documentation for std::env::set_var does not warn about this interaction, and the C++ standard does not define its initialization order relative to foreign language runtimes. The only defense is experience with mixed-language projects and an understanding of how ELF binaries handle initialization.

A secondary assumption was that the daemon's crash was due to the env-var timing issue specifically. The assistant initially considered other possibilities (e.g., the process being killed by a previous command), but the controlled experiment with the env-var-preset launch confirmed the hypothesis.

Input Knowledge Required

To understand this message and its context, a reader needs:

• C++ static initialization model: Knowledge that static objects with non-trivial constructors are initialized during program startup, before main(), and that for statically-linked libraries this happens during the ELF .init_array processing.
• Rust FFI and process model: Understanding that Rust's main() is not the true process entry point—the Rust runtime performs its own initialization before main(), but this still runs after C++ static constructors for linked libraries.
• POSIX environment semantics: getenv() and setenv() operate on a per-process environment block; setenv() can be called at any time, but any code that calls getenv() before setenv() will see the old value.
• CUDA proving pipeline: The groth16_pool is a CPU thread pool used for b_g2_msm preprocessing and parallel Pippenger MSM computations in the Groth16 prover. It is distinct from the GPU's CUDA kernel launches.
• Rayon and thread pool partitioning: The broader optimization strategy of partitioning CPU cores between synthesis (Rust/rayon) and GPU proving (C++ thread pool) to avoid contention.

Output Knowledge Created

This message, as the final edit in the lazy initialization refactor, produces:

• A working thread pool partitioning mechanism: The CUZK_GPU_THREADS environment variable now correctly controls the GPU prover's CPU thread pool size, because the pool is initialized lazily after Rust's main() has set the variable.
• A reusable pattern for cross-language configuration: The lazy initialization with std::call_once pattern can be applied to any C++ resource that depends on configuration set by Rust code during startup.
• A documented initialization ordering constraint: The code now explicitly handles the C++-before-Rust initialization order, making the dependency visible to future maintainers.

The Broader Significance

This fix was a prerequisite for the entire thread pool partitioning strategy. Without it, the GPU prover would either use all CPU cores (defeating the purpose of partitioning) or crash on startup. The successful implementation of lazy initialization enabled the subsequent benchmarks that measured the impact of thread isolation on throughput (<msg id=1922-1928>), ultimately demonstrating that while thread pool partitioning could prevent catastrophic contention, the fundamental bottleneck had shifted from GPU idle time to absolute CPU synthesis time—a finding that drove the design of subsequent optimization phases.

In the larger narrative of the cuzk proving engine's evolution, this message represents a moment of debugging clarity: a subtle cross-language initialization bug, diagnosed through careful experimentation, fixed with a well-understood pattern, and resolved with a single line of tool output confirming "Edit applied successfully."