The Static Initialization Trap: Fixing Thread Pool References in a GPU Proving Pipeline

Message Overview

The subject message, <msg id=1937>, is a brief but pivotal moment in a debugging session about thread pool initialization order in the cuzk SNARK proving engine. In it, the assistant states:

I need to replace all groth16_pool. and &groth16_pool references with get_groth16_pool(). and &get_groth16_pool():

It then issues a read tool call to inspect the file /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu, specifically seeking all occurrences of the old static variable groth16_pool that must be updated to use a new lazy-initialized accessor function.

On its surface, this is a mechanical refactoring step—find-and-replace across a CUDA source file. But the message represents the culmination of a subtle debugging journey that revealed a fundamental mismatch between C++ static initialization semantics and Rust's runtime environment setup. Understanding why this simple rename was necessary requires unpacking the entire thread pool isolation problem that preceded it.

The Problem: CPU Thread Contention in a GPU Proving Pipeline

The broader context is the optimization of Filecoin's Proof-of-Replication (PoRep) Groth16 proof generation. The cuzk proving engine has two major phases that compete for CPU resources:

  1. Synthesis (CPU-bound): Circuit synthesis using Rust's rayon parallel framework, which spawns work across all available CPU cores.
  2. GPU-side preprocessing (also CPU-bound): Operations like b_g2_msm (a multi-scalar multiplication on the G2 curve) that run on the CPU to prepare data for GPU computation. These operations use a C++ thread pool called groth16_pool, implemented via Supranational's thread_pool_t class. When both phases run concurrently—as they do when the engine attempts to overlap synthesis with GPU proving—the two thread pools compete for the same physical cores. The rayon pool (used for synthesis) and the groth16_pool (used for GPU preprocessing) would collectively try to use all 192 logical threads (96 cores × 2 hyperthreads) on the target machine, causing severe CPU contention that degraded overall throughput. The solution, as conceived in the preceding messages, was to partition the CPU cores: allocate a subset of cores (e.g., 64) to synthesis via rayon's ThreadPoolBuilder, and a separate subset (e.g., 32) to the C++ groth16_pool via a new CUZK_GPU_THREADS environment variable. This would prevent the two pools from stepping on each other and allow true overlap between synthesis and GPU work.

The Static Initialization Trap

The assistant implemented this partitioning in two steps:

  1. Rust side (cuzk-daemon/src/main.rs, <msg id=1911>): Called std::env::set_var("CUZK_GPU_THREADS", "32") and configured the rayon global pool with rayon::ThreadPoolBuilder::new().num_threads(64).build_global().
  2. C++ side (groth16_cuda.cu, <msg id=1907>): Modified the static groth16_pool declaration to read CUZK_GPU_THREADS from the environment using getenv(). The assumption was that std::env::set_var("CUZK_GPU_THREADS", ...) in Rust's main() would execute before the C++ static initializer called getenv(). This assumption proved incorrect. When the assistant tested the isolated config (<msg id=1929>), the daemon died silently. Further investigation (<msg id=1930>) revealed the root cause: C++ static objects with dynamic initializers are constructed before main() begins execution. In C++, the static groth16_pool variable—declared as:
static thread_pool_t groth16_pool([]() -> unsigned int {
    const char* env = getenv("CUZK_GPU_THREADS");
    return env ? (unsigned int)atoi(env) : 0;
}());

—runs its initializer lambda during static initialization, which occurs at program startup, before any main() function (whether C, C++, or Rust's main()). Since the Rust runtime's std::env::set_var call happens during main(), the getenv() in the C++ static initializer sees a NULL return and defaults to 0 (all CPUs).

This is a well-known cross-language initialization ordering hazard. The C++ standard guarantees that static objects in a translation unit are initialized in order of their declaration, but the relative ordering between C++ static initialization and Rust's runtime setup is undefined and platform-dependent. On Linux with dynamic linking (.so files), C++ statics in shared libraries are initialized when the library is loaded via dlopen or at process startup for linked libraries. For a library linked at compile time (as supraseal-c2 is with cuzk-daemon), the static initialization happens during the dynamic linker's startup phase, well before main().

The Solution: Lazy Initialization

The assistant's response to this discovery was to replace the static-initialized groth16_pool with a lazily-initialized alternative (<msg id=1934>). The new code uses a raw pointer combined with std::call_once:

static thread_pool_t* groth16_pool_ptr = nullptr;
static std::once_flag  groth16_pool_init_flag;

static thread_pool_t& get_groth16_pool() {
    std::call_once(groth16_pool_init_flag, []() {
        const char* env = getenv("CUZK_GPU_THREADS");
        unsigned int num_threads = env ? (unsigned int)atoi(env) : 0;
        groth16_pool_ptr = new thread_pool_t(num_threads);
    });
    return *groth16_pool_ptr;
}

This approach defers pool construction to the first call to get_groth16_pool(), which happens during actual GPU proving operations—well after Rust's main() has had a chance to call std::env::set_var(). The std::call_once ensures thread-safe single initialization, and the raw pointer (leaked intentionally, as the pool lives for the process lifetime) avoids the static initialization order fiasco entirely.

The Subject Message: Mechanical Refactoring with Deep Significance

This brings us to <msg id=1937>. Having introduced get_groth16_pool() as the replacement for the direct groth16_pool static variable, the assistant must now update every call site in the CUDA file. The message states the intent clearly: replace groth16_pool. with get_groth16_pool(). and &groth16_pool with &get_groth16_pool().

The read tool call that follows is not merely informational—it is a systematic inventory. The assistant needs to find every occurrence to ensure no stale references remain. A single missed reference would cause a compilation error (since groth16_pool no longer exists as a variable name) or, worse, a link-time error if the old symbol is referenced somewhere unexpected.

The grep results from <msg id=1935> show five call sites:

  1. Line 199: groth16_pool.par_map(num_circuits, [&](size_t c) { — preprocessing step
  2. Line 347: groth16_pool.par_map(num_circuits, [&](size_t c) { — another parallel map
  3. Line 543: groth16_pool.par_map(num_circuits, [&](size_t c) { — yet another parallel map
  4. Line 559: true, &groth16_pool); — passing pool reference to a function Each of these must be updated. The par_map calls are the primary mechanism by which the C++ code parallelizes work across circuits during GPU preprocessing. The &groth16_pool at line 559 is passed as an argument to a function that accepts a thread_pool_t&, likely for a single-circuit path that uses the pool differently.

Assumptions and Their Corrections

Several assumptions were made and corrected during this sequence:

Assumption 1: std::env::set_var executes before C++ static initialization. This was the critical error. The assistant assumed that because the Rust main() function is the program's entry point, its first statements execute before any C++ static constructors. In reality, C++ static initialization happens during the dynamic linker's startup phase, which completes before main() is called. This is a subtle but important detail of mixed-language programs.

Assumption 2: The daemon crash was due to a different issue. When the daemon failed to start with the isolated config (<msg id=1929>), the assistant initially suspected a process management issue (the daemon being killed by a previous command). Only after testing with the environment variable pre-set (CUZK_GPU_THREADS=32 prefix) did the initialization order issue become clear.

Assumption 3: A static lambda initializer would work. The original modification (<msg id=1907>) used a lambda invoked directly in the static declaration. This is valid C++ and works correctly when the environment variable is already set. The problem was not with the C++ code itself but with the timing of when the environment variable was set relative to when the static initializer ran.

Input Knowledge Required

To understand this message, one needs knowledge of:

  1. C++ static initialization order: The rule that dynamic initialization of static objects happens before main(), and the implications for cross-language programs.
  2. Rust's std::env::set_var: That it modifies the process environment at runtime via setenv(), not at link time.
  3. The cuzk proving engine architecture: How synthesis (rayon) and GPU preprocessing (groth16_pool) compete for CPU resources, and why partitioning is necessary.
  4. The thread_pool_t class: Supranational's thread pool implementation used in the sppark library for parallelizing MSM and other operations.
  5. The std::call_once pattern: A thread-safe lazy initialization mechanism in C++11 that avoids the static initialization order problem.

Output Knowledge Created

This message and its surrounding context produced several important knowledge artifacts:

  1. A corrected initialization strategy: The lazy-initialized get_groth16_pool() pattern that defers pool construction until first use, after Rust's environment setup.
  2. A complete inventory of call sites: The grep results documenting all five locations where groth16_pool is referenced, providing a checklist for the refactoring.
  3. A documented cross-language hazard: The discovery that C++ static initialization precedes Rust's main() serves as a cautionary tale for future developers working on mixed Rust/C++ codebases.
  4. A reusable pattern: The std::call_once + raw pointer approach can be applied to any C++ static that depends on runtime configuration set by Rust.

The Broader Significance

This message, while mechanically simple, illustrates a class of bugs that plague mixed-language systems. The static initialization order fiasco is well-known in pure C++ programs, where static objects in different translation units have undefined initialization order. But in Rust/C++ hybrid systems, an additional layer of complexity arises: the Rust runtime itself is initialized during main(), meaning any C++ static that depends on Rust-provided state (environment variables, configuration, etc.) will fail.

The solution—lazy initialization—is a classic pattern for defusing initialization order dependencies. By deferring construction to the point of first use, the dependency on Rust's environment setup is naturally satisfied because Rust's main() has already executed by the time any GPU proving operation is requested.

This episode also demonstrates the importance of testing assumptions about execution order in mixed-language systems. The assistant's initial approach was logically sound in isolation—set an environment variable, then read it—but failed because of an invisible ordering constraint imposed by the C++ runtime model. The debugging process, moving from crash observation to hypothesis testing (pre-setting the env var) to root cause identification to solution implementation, is a textbook example of systematic debugging.

Conclusion

Message <msg id=1937> is the mechanical culmination of a deeper debugging journey. The simple act of reading a file to find references for replacement belies the subtle initialization order problem that necessitated the change. The assistant's transition from a static-initialized thread pool to a lazy-initialized one, and the subsequent refactoring of all call sites, represents a correct and robust solution to a cross-language initialization hazard that could easily have remained hidden until runtime failure in production.

The key lesson is that in mixed Rust/C++ programs, C++ static initialization runs before Rust's main(), making std::env::set_var ineffective for configuring C++ statics. Lazy initialization via std::call_once provides a clean escape from this ordering constraint, at the cost of slightly more verbose code and the need to update all references from a variable to a function call.