Launching the Pipeline: The First E2E GPU Test of cuzk Phase 2

Introduction

In the development of high-performance SNARK proving infrastructure, few moments are as tense as the first end-to-end test of a new architecture. The message at [msg 540] captures exactly such a moment: the assistant launches the cuzk daemon with Phase 2's pipelined proving engine enabled for the first time, pointing it at real GPU hardware and a 32 GiB Filecoin PoRep (Proof-of-Replication) C2 proof. This single bash command — a nohup invocation of the daemon binary — represents the culmination of weeks of architectural work: a minimal bellperson fork to expose the synthesis/GPU split point, an SRS (Structured Reference String) manager for persistent parameter residency, a per-partition pipeline module, and the integration of all these pieces into the cuzk engine's dispatch logic. The message is brief, but it carries the weight of validating whether the entire Phase 2 design works in practice.

The Message in Full

The assistant executed the following commands:

# Start the daemon in the background with pipeline enabled
# Use nohup + redirect so we can check the logs
nohup /home/theuser/curio/extern/cuzk/target/release/cuzk-daemon --config /tmp/cuzk-pipeline-test.toml > /tmp/cuzk-daemon-pipeline.log 2>&1 &
echo "Daemon PID: $!"
sleep 3
# Check if the daemon started OK
tail -30 /tmp/cuzk-daemon-pipeline.log

The output showed:

Daemon PID: 1206920
2026-02-17T19:25:47.388456Z  INFO cuzk_daemon: cuzk-daemon starting
2026-02-17T19:25:47.388475Z  INFO cuzk_daemon: configuration loaded listen=0.0.0.0:9821
2026-02-17T19:25:47.388490Z  INFO cuzk_core::engine: starting cuzk engine pipeline_enabled=true
2026-02-17T19:25:47.388492Z  INFO cuzk_core::engine: preloading ...

The daemon started successfully, loaded its configuration, confirmed that pipeline mode was enabled, and began preloading the SRS parameters — a 45 GiB operation that would take between 15 and 90 seconds depending on disk cache warmth.

Why This Message Was Written

This message did not emerge from a vacuum. It was the direct result of a deliberate, multi-step decision process that began several messages earlier. At [msg 534], the assistant had surveyed the Phase 2 implementation and created a prioritized todo list. The top item was "E2E GPU test: build with --features cuda-supraseal, run PoRep C2 through pipeline, verify proof." When the user was asked at [msg 535] which step to tackle next, they chose "All of the above sequentially," giving the assistant a clear mandate to start with the GPU test.

The reasoning behind prioritizing the E2E test first was sound: before writing more pipeline code for PoSt and SnapDeals proof types, the assistant needed to confirm that the foundational plumbing worked. The bellperson fork, the SRS manager, the per-partition synthesis/GPU split — all of these were untested in a real GPU environment. A build-only validation had been performed at [msg 537] using cargo build --workspace --features cuda-supraseal --release, which succeeded, but compilation success is no guarantee of runtime correctness. The E2E test was the gatekeeper: if the daemon crashed during SRS loading, or the pipeline produced an invalid proof, or the GPU integration failed, the assistant would need to debug and fix those issues before layering on additional complexity.

The message also reflects a pragmatic engineering workflow. Rather than running the daemon interactively (which would block the shell), the assistant used nohup and output redirection to run it as a background process. This allowed the subsequent messages to poll the log file, wait for SRS loading, submit a test proof, and monitor progress — all without manual intervention. The sleep 3 and tail -30 sequence was a lightweight health check: confirm the daemon didn't crash immediately on startup, verify the configuration was parsed correctly, and check that pipeline mode was acknowledged.

Assumptions Embedded in the Launch

Several assumptions are baked into this seemingly simple command sequence. First, the assistant assumed that the compiled binary existed at the expected path — /home/theuser/curio/extern/cuzk/target/release/cuzk-daemon — and that it was built with the cuda-supraseal feature flag. This assumption was validated by the successful build at [msg 537], but it's worth noting that the build output showed only warnings, no errors, from the bellperson dependency's GPU locking code.

Second, the assistant assumed that the test configuration at /tmp/cuzk-pipeline-test.toml was correct. This config file had been written at [msg 539] and included settings for the daemon listen address (port 9821, different from the default 9820 to avoid conflicts), the SRS directory, and crucially, pipeline_enabled = true. The log output confirmed this assumption: "starting cuzk engine pipeline_enabled=true."

Third, the assistant assumed that the SRS parameters were already present on disk. The SRS manager's preload method would attempt to load the 45 GiB of Groth16 parameters from the configured directory. If the parameters were missing or corrupted, the daemon would either crash or hang. The subsequent message at [msg 541] revealed that SRS loading completed in 15.4 seconds — indicating the parameters were hot-cached from previous runs, a fortunate outcome that the assistant could not have guaranteed at the time of launching.

Fourth, the assistant assumed that the GPU was available and that the CUDA runtime could initialize successfully. The cuda-supraseal feature flag enables GPU proving via the supraseal library, which in turn depends on NVIDIA CUDA drivers and compatible hardware. If the GPU were absent or misconfigured, the daemon might start but fail during proof submission. This risk was implicitly accepted; the assistant would discover any GPU issues only when a proof was actually submitted.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several layers of the cuzk architecture. At the top level, cuzk is a proving engine for Filecoin storage proofs — it generates Groth16 SNARKs that storage providers submit to the Filecoin network to prove they are storing data correctly. The Phase 2 design, documented in cuzk-phase2-design.md, introduces a pipelined architecture that splits the monolithic proving process into two phases: CPU-bound circuit synthesis and GPU-bound NTT+MSM proving. The goal is to overlap these phases across multiple proofs for higher throughput.

The SRS manager is a critical component: it preloads the 45 GiB of Groth16 parameters into memory once, rather than loading them from disk on every proof call. This eliminates a major source of latency and I/O overhead. The pipeline module (pipeline.rs) implements per-partition synthesis and GPU proving for PoRep C2 proofs, which consist of 10 partitions that must each be synthesized and proven independently.

The bellperson fork is the most subtle dependency. The standard bellperson library (a Rust implementation of the Bellman zk-SNARK library) does not expose the intermediate state between circuit synthesis and GPU proving. The cuzk team created a minimal fork that adds this split point, allowing the pipeline to serialize the synthesized circuit data and pass it to the GPU prover without re-executing synthesis. Without this fork, the entire pipeline architecture would be impossible.

Output Knowledge Created

The log output from this message created immediate, actionable knowledge. The daemon started without errors, confirming that the binary was correctly compiled and linked. The configuration was parsed successfully, confirming that the TOML format and field names matched the daemon's expectations. Most importantly, the log line "starting cuzk engine pipeline_enabled=true" confirmed that the engine's dispatch logic correctly detected and enabled the pipeline mode — a non-trivial validation, since the pipeline mode affects how the engine routes proof requests to either the monolithic Phase 1 prover or the new Phase 2 pipeline.

The SRS preloading log line confirmed that the SRS manager's initialization path was reached. The subsequent message at [msg 541] would reveal that preloading completed in 15.4 seconds, but even at this early stage, the fact that preloading began without a crash was significant. The SRS manager involves memory-mapped file I/O, cache initialization, and parameter validation — any of which could have failed silently.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the command itself. The use of nohup and output redirection reveals an awareness that the daemon is a long-lived process that should not be tied to the shell's lifetime. The sleep 3 before the log check indicates an expectation that startup is near-instantaneous — the daemon should either crash or become ready within seconds. The tail -30 command, rather than a simple cat, shows a preference for recent output, acknowledging that the log file might contain prior runs or verbose initialization messages.

The choice of port 9821 (rather than the default 9820) is a deliberate safety measure: it prevents conflicts with any existing cuzk daemon that might be running for other purposes. The temporary config file at /tmp/cuzk-pipeline-test.toml is equally deliberate — it isolates the test configuration from any production configuration, reducing the risk of accidentally submitting test proofs to a production daemon.

The Broader Significance

In the arc of the cuzk Phase 2 development, this message is the inflection point where theory meets practice. The preceding messages were about design documents, code reviews, and compilation checks. This message is about running real software on real hardware. The fact that the daemon started successfully does not guarantee that the pipeline produces correct proofs — that validation would come in the following messages, where the assistant submitted a 32 GiB PoRep C2 proof and discovered that the per-partition sequential approach was 6.6× slower than the monolithic baseline. But without this first step — without the daemon starting, without pipeline mode being enabled, without SRS preloading — none of those discoveries could have been made.

This message also illustrates a key principle of the assistant's methodology: validate the foundation before building the superstructure. The E2E GPU test was prioritized over writing PoSt and SnapDeals synthesis code precisely because a failure at this level would invalidate the entire Phase 2 approach. By launching the daemon and confirming basic operability, the assistant established a stable platform for the more complex work that followed — the batch-all-partitions synthesis optimization, the PoSt/SnapDeals pipeline expansion, and ultimately the async overlap architecture for throughput scaling.