The Moment of Diagnosis: Reverting B1 and the Art of Performance Engineering

[assistant] 5 timing points remain (pin_abc removed). Now start daemon and run the test:
[bash] FIL_PROOFS_PARAMETER_CACHE=/data/zk/params RUST_LOG=info /home/theuser/curio/extern/cuzk/target/release/cuzk-daemon --config /tmp/cuzk-baseline-test.toml > /tmp/cuzk-phase4-test3.log 2>&1 &
echo "Daemon PID: $!"
sleep 2
tail -3 /tmp/cuzk-phase4-test3.log
Daemon PID: 579923
[2026-02-17T22:53:25.233175Z INFO cuzk_core::engine] starting cuzk engine pipeline_enabled=true
[2026-02-17T22:53:25.233178Z INFO cuzk_core::engine] preloading SRS via SrsManager (Phase 2) circuit_ids=[Porep32G]
[2026-02-17T22:53:25.233273Z INFO cuzk_core::srs_manager] loading SRS from disk circuit_id=porep-32g path=...

At first glance, message [msg 981] appears to be a routine operational command: start a daemon, run a test. But in the context of the broader investigation, this message represents a pivotal moment in a disciplined performance engineering campaign. It is the point at which a hypothesis has been confirmed, a harmful optimization has been surgically removed, and the system is being prepared for a critical validation run. The message is deceptively simple, but every element in it carries the weight of the preceding diagnostic work.

The Context: A Regression Investigation in Phase 4

To understand why this message was written, one must understand the arc of the cuzk project's Phase 4 optimization work. The team had successfully completed Phases 0 through 3, establishing a solid baseline of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 aimed to introduce a suite of compute-level optimizations drawn from a detailed proposal document (c2-optimization-proposal-4.md). Five optimizations were implemented in the first wave: A1 (SmallVec optimization for the LC Indexer), A2 (pre-sizing for ProvingAssignment), A4 (parallelizing B_G2 CPU MSMs), B1 (pinning a/b/c vectors with cudaHostRegister), and D4 (per-MSM window tuning).

When the combined changes were tested, the result was alarming: instead of improving, the proof time regressed to 106 seconds — a 17-second slowdown from the baseline. This triggered a systematic diagnostic effort. The assistant added detailed CUDA timing instrumentation (CUZK_TIMING printf statements) to the GPU code, enabling precise phase-level breakdowns. The first instrumented run (message [msg 962]) revealed the primary culprit: the B1 optimization — cudaHostRegister — was adding 5.7 seconds of overhead by pinning approximately 125 GiB of host memory. The proposal had estimated 150–300 ms for this operation, but the reality was nearly 20 times worse because mlock-ing 125 GiB forces the kernel to touch every page.

The Decision to Revert

Message [msg 981] is the direct consequence of that discovery. Between the first instrumented test and this message, the assistant made a series of deliberate decisions. First, the A2 pre-sizing optimization had already been partially reverted from the multi-sector synthesis path earlier (in chunk 1 of this segment). Second, the B1 code was removed from the CUDA source file in two edits: the pinning block at the start of the GPU prove function ([msg 970]) and the unpin block at the end ([msg 973]). The assistant then verified the rebuild by checking the library's symbol table — confirming that pin_abc no longer appeared in the binary ([msg 980]) while the remaining five CUZK_TIMING instrumentation points were preserved.

The phrase "5 timing points remain (pin_abc removed)" is significant. It tells us that the assistant is consciously tracking the instrumentation state. The CUZK_TIMING points are the diagnostic window into the GPU's behavior, and the assistant wants to ensure that after removing B1, the remaining instrumentation is intact to validate the next test. This is not a casual remark — it reflects a systematic approach to measurement where every change is verified at the binary level before proceeding.

Assumptions and Their Validation

This message embodies several assumptions, some explicit and some implicit. The primary assumption is that reverting B1 will recover most of the lost performance. The data strongly supports this: the first instrumented run showed 101.3 seconds total, with B1 contributing 5.7 seconds of overhead. Removing B1 should bring the time down to approximately 95.6 seconds, assuming no other interactions. However, the assistant also knows from the same data that synthesis had regressed from 54.7 seconds to 61.0 seconds — a 6.3-second increase that could not be explained by B1 alone, since A2 had already been reverted. This remaining regression pointed to A1 (SmallVec) as the next suspect.

Another assumption is that the build system correctly recompiled the CUDA code. The assistant had earlier struggled with this: cargo clean -p supraseal-c2 removed zero files because the CUDA compilation artifacts are managed by build.rs and live outside the standard cargo output directory ([msg 946]). The assistant had to manually remove the build directories and verify the library size changed (2791536 bytes before vs. 2790792 bytes after — a 744-byte difference confirming recompilation). This attention to build artifact verification shows an understanding that in a complex mixed-language project (Go, Rust, C++, CUDA), the build system can silently cache stale binaries.

Input Knowledge Required

To fully understand this message, one needs substantial context. The reader must know that FIL_PROOFS_PARAMETER_CACHE points to a directory containing Filecoin proof parameters (large structured reference strings and circuit parameters). They must understand that the daemon uses a pipeline architecture (Phase 2) with an SRS manager that preloads structured reference strings from disk. The --config /tmp/cuzk-baseline-test.toml references a temporary configuration file set up for benchmarking. The log output confirms the engine started successfully, pipeline mode is enabled, and SRS loading has begun for the Porep32G circuit — a 32 GiB PoRep circuit.

The message also assumes familiarity with the cuzk project's architecture: the daemon listens on a TCP port (9821), the bench tool connects to it remotely, and the pipeline enables asynchronous overlap between CPU synthesis and GPU proving. The SRS manager is a Phase 2 innovation that preloads SRS data to eliminate the loading overhead that plagued earlier monolithic implementations.

Output Knowledge Created

This message produces several forms of knowledge. First, it confirms that the daemon starts cleanly after the B1 revert — the log shows no errors, no crashes, and the expected initialization sequence. Second, it establishes the starting point for the next benchmark: the daemon PID (579923), the log file path (/tmp/cuzk-phase4-test3.log), and the timestamp (22:53:25). Third, it implicitly creates a baseline expectation: if the B1 revert is successful, the next proof should complete in approximately 95 seconds rather than 101 seconds.

The message also creates operational knowledge about the build system. The fact that the assistant can verify "5 timing points remain" by checking the compiled library's symbol table demonstrates a reproducible verification technique. Future developers working on this project can use the same strings | grep CUZK_TIMING approach to confirm that instrumentation is present in the binary.

The Thinking Process Visible in the Message

While the message itself is terse, the thinking process is visible in what is included and what is omitted. The assistant does not explain why the daemon needs to be restarted — it simply does it. The omission is meaningful: the assistant knows that the CUDA library has been relinked into the daemon binary, and the old daemon process still has the old binary mapped in memory. A restart is required to load the new code. This is obvious to someone familiar with the system, but the assistant's confidence in performing this step without explanation shows deep familiarity.

The choice to use tail -3 to verify the log output is also revealing. The assistant is not interested in the full startup sequence — it only needs to confirm three things: the engine started, pipeline mode is enabled, and SRS loading began. Any error would appear in these first few lines. This is efficient, targeted verification.

The Broader Significance

Message [msg 981] sits at the boundary between diagnosis and remediation. The diagnostic phase identified B1 as harmful; the remediation phase has removed it; and this message launches the validation that will confirm the fix works. It is a moment of transition, and the assistant's calm, methodical approach — verifying the binary, checking the instrumentation, starting the daemon, confirming the log — exemplifies disciplined performance engineering.

The message also illustrates a crucial principle: in optimization work, the ability to revert is as important as the ability to implement. The B1 optimization was well-intentioned — pinning host memory should reduce DMA transfer latency — but the actual cost of touching 125 GiB of pages overwhelmed any benefit. The assistant's willingness to revert, rather than trying to tune parameters or work around the issue, reflects a mature engineering judgment. Sometimes the best optimization is removing an optimization.

What follows this message (in the subsequent conversation) is the next instrumented test, which will show the proof time dropping to 94.4 seconds — a 7-second improvement from the regressed state, but still 5.5 seconds above the baseline. That remaining gap will lead the assistant to build a synth-only microbenchmark and discover that A1 (SmallVec) is causing a 5–6 second synthesis slowdown, setting up the next round of investigation. But in this moment, captured in message [msg 981], the focus is singular: start the daemon, run the test, and see if the B1 revert was the right call.