Debugging the C++ Static Initialization Order: Thread Isolation for GPU Proving

In the high-stakes world of Filecoin proof generation, every millisecond of GPU idle time translates directly to lost revenue. The cuzk proving daemon, a sophisticated pipeline for generating Groth16 proofs over BLS12-381 curves, had reached a critical juncture: CPU contention between the synthesis phase (building circuits from vanilla proofs) and the GPU proving phase (running multi-scalar multiplications on the GPU) was starving both phases of compute resources. The assistant had just implemented a thread isolation strategy — limiting the rayon thread pool for synthesis and the C++ groth16_pool for GPU preprocessing — and was eager to benchmark the results. But the daemon refused to start.

The Failed Launch

Message [msg 1925] opens with a simple observation: "Daemon didn't start." The assistant had attempted to launch the daemon with a baseline configuration using nohup and redirecting output to a log file, but the log file was never created. This prompted a more direct approach — running the daemon in the foreground with output visible:

[bash] /home/theuser/curio/extern/cuzk/target/release/cuzk-daemon --config /tmp/cuzk-baseline.toml 2>&1 &
DAEMON_PID=$!
sleep 10
# Check if still running
if kill -0 $DAEMON_PID 2>/dev/null; then
    echo "DAEMON RUNNING (PID=$DAEMON_PID)"
else
    echo "DAEMON DIED"
    wait $DAEMON_PID 2>/dev/null
fi

The output revealed that the daemon did start successfully:

2026-02-18T19:49:06.365700Z INFO cuzk_daemon: cuzk-daemon starting
2026-02-18T19:49:06.365715Z INFO cuzk_daemon: configuration loaded listen=0.0.0.0:9820
2026-02-18T19:49:06.373709Z INFO cuzk_daemon: rayon global thread pool configured rayon_threads=192
2026-02-18T19:49:06.373736Z INFO cuzk_core::engine: starting cuzk engine p...

The daemon was alive, the SRS (Structured Reference String) was loading — the earlier failure was merely a timing issue with the nohup approach. But this seemingly mundane debugging step revealed a critical piece of information: rayon_threads=192. The machine had 96 physical cores with hyperthreading enabled, and the default rayon configuration was using all 192 logical processors. This was precisely the problem the thread isolation strategy was meant to address — having 192 rayon threads competing with the C++ GPU thread pool for CPU resources was creating the contention bottleneck identified in earlier benchmarks.

The Hidden Bug: Static Initialization Order

What the assistant didn't yet realize was that a more subtle bug lurked beneath the surface. The thread isolation implementation had two components:

  1. Rayon pool limiting: Configured via rayon::ThreadPoolBuilder::new().num_threads(N).build_global() called in Rust's main() — this worked correctly.
  2. C++ groth16_pool limiting: The assistant had modified groth16_cuda.cu to read a CUZK_GPU_THREADS environment variable, and set this variable in Rust's main() via std::env::set_var("CUZK_GPU_THREADS", "32"). The problem, which the assistant would discover in the following messages ([msg 1928] through [msg 1934]), is that C++ static objects are initialized at library load time — which, for a statically linked library, occurs before main() executes. The groth16_pool was declared as:
static thread_pool_t groth16_pool([]() -> unsigned int {
    const char* env = getenv("CUZK_GPU_THREADS");
    return env ? (unsigned int)atoi(env) : 0;
}());

This lambda runs during library initialization, calling getenv("CUZK_GPU_THREADS") before Rust's main() has had a chance to call setenv(). The result: the environment variable is never seen, the pool defaults to hardware_concurrency() (all 192 threads), and the isolation strategy is silently defeated.

The Reasoning Behind the Approach

The assistant's design decisions in this message reveal a deep understanding of the system's architecture. The thread isolation strategy was motivated by earlier benchmarking that showed severe CPU contention when parallel synthesis was enabled ([msg 1921]). With 10 PoRep C2 partitions synthesizing simultaneously via rayon, and the C++ groth16_pool spawning additional threads for b_g2_msm preprocessing, the CPU was oversubscribed by a factor of 2-3x, leading to context-switching overhead that negated any parallelism gains.

The assistant's plan was elegant in theory:

Input Knowledge Required

To fully understand this message, one needs familiarity with several domains:

Output Knowledge Created

This message produced several important outputs:

  1. Confirmation that the daemon builds and starts correctly with the new thread isolation code — the build in [msg 1916] succeeded, and the daemon initializes without crashes.
  2. A baseline rayon_threads=192 measurement — confirming that the default configuration uses all hyperthreads, which is the contention source.
  3. A working benchmark infrastructure — the daemon is running and ready to accept benchmark requests, with the SRS loading in the background.
  4. An implicit discovery of the initialization order bug — though the assistant doesn't articulate it in this message, the fact that the daemon started with rayon_threads=192 rather than the configured synthesis.threads=64 (from the baseline config) would trigger the investigation in subsequent messages.

Assumptions and Their Consequences

The assistant made several assumptions in this message, some of which would prove incorrect:

Assumption 1: The nohup launch failed due to a daemon crash. In reality, the daemon started fine — the nohup approach simply had a race condition where the log file wasn't flushed before the grep check ran. This assumption was reasonable and quickly corrected by running the daemon directly.

Assumption 2: Setting CUZK_GPU_THREADS via std::env::set_var in Rust's main() would be visible to C++ static constructors. This was the critical mistake. The assistant understood C++ static initialization order but underestimated the timing: in a Rust binary that links a C++ static library, the C++ statics are initialized during the dynamic linker's initialization phase, which occurs before Rust's main() entry point. The set_var call happens too late.

Assumption 3: The rayon_threads=192 log message indicates the default configuration is being used. This is partially correct — it does show the default. But the assistant hadn't yet verified whether the synthesis.threads config field was being properly applied. The baseline config (/tmp/cuzk-baseline.toml) likely had synthesis.threads = 0 (auto-detect), which would also produce 192 threads.

The Thinking Process

The assistant's reasoning in this message follows a clear debugging workflow:

  1. Observe failure: The daemon didn't start (or appeared not to).
  2. Eliminate variables: Run directly instead of through nohup to see actual output.
  3. Verify hypothesis: The daemon is actually running — the earlier failure was a false negative.
  4. Extract signal: The rayon_threads=192 log line is a valuable data point about the current configuration.
  5. Proceed to next step: With the daemon confirmed running, the assistant prepares to run benchmarks. This is a textbook debugging pattern: when a tool (nohup + log file) introduces uncertainty, bypass it with a simpler method (foreground execution). The assistant doesn't waste time debugging the nohup invocation — it immediately switches to a more reliable approach.

The Broader Context

This message sits at a pivotal moment in the optimization journey. The previous segments had established that: