The Testing Manifesto: How a Todo List Became the Blueprint for Validating Cross-Sector SNARK Batching

Message Context

In the middle of a months-long engineering effort to build cuzk — a pipelined SNARK proving engine for Filecoin proof generation — the assistant receives a two-word instruction from the user: "Proceed to test" ([msg 695]). The response, [msg 697], is deceptively simple on its surface: a todowrite tool call that updates a structured todo list. But this message represents far more than a simple status update. It is the bridge between implementation and validation, the moment when a complex architectural change (Phase 3 cross-sector batching) transitions from theory to empirical reality.

The message reads:

[assistant] [todowrite] {"todos":[{"content":"Build cuzk with cuda-supraseal for GPU testing","priority":"high","status":"in_progress"},{"content":"Create test config with max_batch_size=2 for batch testing","priority":"high","status":"pending"},{"content":"Start daemon with batch config","priority":"high","status":"pending"},{"content":"Test single proof (batch_size=1 baseline — verify Phase 2 compatibility)","priority":"high","status":"pending"},...}

The ellipsis truncates the remaining items, but the pattern is clear: this is a structured testing plan encoded as a todo list, with each task assigned a priority and status. The first item is already marked in_progress, signaling that the assistant has already begun executing the plan before the todo list was even displayed.

Why This Message Was Written

The message exists at a critical inflection point in the project. The preceding messages ([msg 674] through [msg 693]) document the implementation and commit of Phase 3 cross-sector batching — a substantial architectural addition involving a new BatchCollector module, multi-sector synthesis functions, batched proof splitting, and a reworked GPU worker that handles batched results. The commit 1b3f1b39 on the feat/cuzk branch represents 1,134 lines of new code across 6 files.

But implementation alone is not enough. The entire cuzk project is built on a foundation of empirical validation. Every phase has been tested on real GPU hardware (an RTX 5070 Ti) with real 32 GiB Filecoin sector data. The Phase 2 async overlap pipeline, for instance, was validated by running three consecutive PoRep proofs and measuring the 1.27x throughput improvement. Without E2E GPU testing, the Phase 3 code is just untrusted logic — it could have correctness bugs, memory leaks, or performance regressions.

The user's instruction "Proceed to test" is the trigger, but the assistant's response reveals something deeper: a systematic testing methodology that has been refined over the course of the project. The todo list is not arbitrary — it encodes a specific testing strategy:

  1. Build first — ensure the CUDA-enabled binary compiles
  2. Create test configs — one for baseline (batch_size=1), one for batching (batch_size=2)
  3. Baseline first — run a single proof with batch_size=1 to verify Phase 2 compatibility hasn't regressed
  4. Then test batching — submit multiple concurrent proofs and verify correct behavior
  5. Test edge cases — timeout flush, non-batchable types, overflow behavior This ordering reveals a core engineering principle: never test a new feature without first establishing a baseline. The assistant knows that if the batch_size=2 test fails, it needs to distinguish between "Phase 3 batching is broken" and "something fundamental in the pipeline is broken." The baseline test isolates these variables.

How Decisions Were Made

The decision-making in this message is subtle because it's encoded in the structure of the todo list rather than explicit deliberation. Several design choices are visible:

The decision to use a todo list as a testing plan is itself noteworthy. Throughout the cuzk project, the assistant has used todowrite as a persistent progress tracker — a way to maintain state across conversation turns and keep the user informed. But in this message, the todo list serves a dual purpose: it's both a communication tool (showing the user what will happen) and an execution plan (the assistant will work through these items in order). The first item is already in_progress, indicating that the assistant didn't wait for the user's approval — it began building the CUDA binary immediately upon receiving the "Proceed to test" instruction.

The choice of test parameters reveals careful thinking. The assistant selects max_batch_size=2 for the batch test config — the smallest possible batch size above 1. This is deliberate: it tests the batching logic with minimal resource commitment. A batch of 2 doubles the synthesis memory (~272 GiB intermediate state for PoRep) but keeps the test feasible on the available hardware. The assistant also sets max_batch_wait_ms=30000 (visible in later messages), giving enough time for two concurrent proof requests to arrive without forcing an artificial timeout.

The decision to test baseline first is another deliberate choice. The assistant could have jumped straight to batch testing, but the todo list explicitly separates "Test single proof (batch_size=1 baseline)" from the batched tests. This creates a clear success criterion: if the baseline passes, Phase 2 compatibility is preserved. Only then does the assistant proceed to validate the new feature.

Assumptions Made

Several assumptions underpin this message:

The hardware assumption: The assistant assumes the RTX 5070 Ti with 16 GB VRAM and the machine's ~200 GiB system memory are sufficient for batch_size=2 testing. This is a reasonable assumption given that Phase 2 already demonstrated ~136 GiB intermediate state for a single batch of 10 partitions, and batch_size=2 would approximately double that. The assistant implicitly trusts that the system won't OOM.

The correctness assumption: The todo list assumes that the Phase 3 implementation is correct at the unit-test level (all 25 tests pass) and that any bugs will manifest as observable failures in E2E testing rather than silent data corruption. This is a standard assumption in software testing, but it's worth noting — the assistant is not running differential fuzzing or proof verification; it's checking that proofs complete and have the expected byte size (1920 bytes).

The serialization assumption: The assistant assumes that split_batched_proofs() correctly separates concatenated Groth16 proofs back into per-sector results. This is tested at the unit level (3 tests in pipeline.rs), but the E2E test will be the first time it runs on real GPU output.

The timeout assumption: The assistant assumes that max_batch_wait_ms=30000 is long enough for concurrent requests to arrive but short enough that the test doesn't take too long. This is a tuning parameter that may need adjustment.

Mistakes and Incorrect Assumptions

Looking at the subsequent messages ([msg 698] through [msg 715]), we can identify where assumptions proved incorrect or where the plan had to adapt:

The daemon was already running: When the assistant tried to start the baseline test ([msg 703]), it discovered a daemon process was already running on port 9821 (PID 2653379). The todo list didn't account for cleanup of previous sessions. The assistant had to kill the existing process before proceeding. This is a minor oversight — the testing plan assumed a clean state.

The kill failed: When the assistant tried to kill the existing daemon ([msg 704]), the process had already exited (zsh reported "no such process"). This was handled gracefully, but it reveals that the assistant's mental model of system state was slightly stale.

Memory monitoring was an afterthought: The user's follow-up message ([msg 702]) says "Note - for testing also record avg/peak ram memory use." The original todo list in [msg 697] did not include memory monitoring. The assistant had to adapt by creating a memory monitor script (/tmp/cuzk-memmon.sh) and running it alongside the daemon. This is a significant gap — the todo list focused on functional correctness and throughput but initially omitted resource measurement. Given that Phase 3's primary tradeoff is throughput vs. memory, this omission is notable.

The SRS loading time: The baseline test showed SRS loading taking ~15 seconds ([msg 712]), which the assistant noted but hadn't explicitly planned for in the todo list. The cold-start cost is a known factor from Phase 2 testing, but the testing plan didn't account for separating SRS load time from proof time in the measurements.

Input Knowledge Required

To understand this message, the reader needs knowledge of:

The cuzk project architecture: The todo list references cuda-supraseal (the CUDA backend for Groth16 proving), max_batch_size (the Phase 3 batching parameter), and the distinction between baseline (Phase 2) and batched (Phase 3) modes. Without understanding that Phase 2 established a working pipeline and Phase 3 adds cross-sector batching on top, the testing plan seems arbitrary.

The Filecoin proof generation pipeline: The message assumes knowledge that PoRep C2 proofs involve CPU-bound circuit synthesis followed by GPU-bound Groth16 proving, that synthesis uses ~136 GiB of intermediate state for a single 10-partition batch, and that the SRS parameters are ~45 GiB. The testing plan is shaped by these resource constraints.

The testing infrastructure: The message references cuzk-bench (the benchmarking tool), the daemon's gRPC API, and the config file format. The todo list assumes these tools exist and work correctly.

The git workflow: The message appears after a commit (1b3f1b39) and assumes the codebase is in a known good state. The testing plan is designed to validate that commit.

Output Knowledge Created

This message creates:

A structured testing methodology: The todo list encodes a reproducible testing protocol. Anyone following these steps would validate Phase 3 batching in the same systematic way — build, baseline, batch test, edge cases. This is tacit knowledge made explicit.

A progress tracking mechanism: The todo list serves as a shared state between assistant and user. The user can see what's been done, what's in progress, and what remains. This is especially valuable in a long-running engineering session where context can be lost between messages.

A decision record: The message documents the testing strategy at the moment it was formulated. Future readers (or the assistant itself in later context windows) can see that the baseline test was prioritized, that batch_size=2 was the first test case, and that the build step was initiated immediately.

The Thinking Process

The thinking visible in this message is primarily architectural — it's the reasoning of an engineer who has internalized a testing discipline and is now executing it systematically. The todo list format itself reveals the thinking: each item is a discrete, verifiable step. The assistant doesn't write "Test Phase 3" as a monolithic task; it decomposes the problem into build, configure, baseline, batch, and edge cases.

The prioritization ("priority":"high" for all items) signals that this testing is not optional — it's the gatekeeper for Phase 3 being considered complete. The status field (in_progress vs pending) creates a visual progress bar.

The most interesting thinking is what's not said. The assistant doesn't explain why it chose this testing order, because the reasoning is embedded in the structure. Baseline before batching. Single before concurrent. Simple before edge case. This is the hallmark of an engineer who has learned from experience that testing must be incremental and that each step should validate the foundation before building on it.

Conclusion

Message [msg 697] is a small message with outsized significance. It marks the transition from Phase 3 implementation to Phase 3 validation, encoding a systematic testing methodology in a structured todo list. The message reveals the assistant's engineering discipline: test incrementally, establish baselines, and decompose complex validation into discrete verifiable steps. While the initial plan had gaps (notably the absence of memory monitoring, which the user had to request separately), the overall approach reflects a mature understanding of how to validate performance-critical systems software. The subsequent messages show this plan being executed faithfully, with the assistant adapting to real-world complications (existing daemon processes, stale process state) while maintaining the testing structure. In the end, Phase 3 cross-sector batching would be validated at 1.46x throughput improvement with only ~2 GiB additional memory overhead — a result that this humble todo list helped make possible.