The Three Fixes: Diagnosing and Resolving a Param Cache Mismatch in the CuZK Benchmark Script

The Subject Message

The message under analysis is a concise summary from the assistant at the conclusion of a debugging and deployment cycle. It reads:

Pushed. Fixes: 1. -n → --count, -j → --concurrency — correct cuzk-bench batch arg names 2. Daemon param_cache — generates a minimal TOML config with param_cache pointing to $PARAM_DIR so the daemon looks in the right place instead of its hardcoded /data/zk/params default 3. export FIL_PROOFS_PARAMETER_CACHE — so cuzk-bench also resolves params from the correct directory

On its surface, this appears to be a simple status update—a "what was fixed" list appended to a Docker push. But beneath this brevity lies a rich debugging story: a tale of config precedence, CLI interface mismatches, and the subtle ways that environment variables and configuration struct defaults interact in a distributed proving system. Understanding this message requires reconstructing the chain of reasoning that led to each of these three fixes.

The Trigger: A Failed Benchmark Run

The story begins with the user running benchmark.sh on a remote vast.ai instance ([msg 675]). The script started a cuzk-daemon, attempted a warmup proof, and immediately failed with two distinct errors.

The first error was a param file lookup failure:

error: C1 parse failed: SRS param file not found for Porep32G:
/data/zk/params/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params

The daemon was searching for parameters in /data/zk/params, but the actual parameters had been downloaded to /var/tmp/filecoin-proof-parameters by the container's entrypoint script. The benchmark script's header even displayed "Param cache: /var/tmp/filecoin-proof-parameters," yet the daemon stubbornly looked elsewhere. This was the central mystery: why was the daemon ignoring the FIL_PROOFS_PARAMETER_CACHE environment variable that the benchmark script was dutifully setting?

The second error appeared after the warmup:

error: unexpected argument '-n' found

Usage: cuzk-bench batch [OPTIONS] --type <PROOF_TYPE>

The benchmark script was passing -n and -j flags to cuzk-bench batch, but the actual CLI used --count and --concurrency instead. This was a straightforward mismatch between the script's assumptions and the binary's interface.

The Investigation: Tracing Config Precedence

The assistant's response to the user's error report ([msg 676]) immediately identified both issues. The first fix was trivial: rename the CLI flags. The second required genuine detective work.

The assistant initially hypothesized that setting FIL_PROOFS_PARAMETER_CACHE would suffice. The benchmark script already exported this variable when launching the daemon. But the error proved otherwise. A grep for param_cache in the daemon source ([msg 680]) confirmed that the codebase used this config key extensively, but the question was how the daemon resolved its value.

The critical insight came in [msg 682]:

The daemon defaults param_cache to /data/zk/params (from config). The benchmark script starts the daemon with FIL_PROOFS_PARAMETER_CACHE but the daemon's own default config overrides that. The daemon reads its config, and the config default wins.

This is a subtle but important point about software architecture. The daemon's configuration is loaded from a config struct that has a hardcoded default value for param_cache: /data/zk/params. When the daemon starts without an explicit config file, it uses this default. The FIL_PROOFS_PARAMETER_CACHE environment variable, which the benchmark script was setting, is read by a different layer of the code—likely the Filecoin proof library (fil-proofs-tooling or similar). The daemon itself never consults this env var for its own config; it only reads its config struct. The env var only affects downstream proof operations that use the Filecoin proof APIs directly.

This is a classic configuration precedence bug. The daemon has three potential sources for the param cache path: (1) a config file, (2) a compiled-in default, and (3) an environment variable. The environment variable was being set, but the daemon's config struct default had higher priority because the daemon never checked the env var. The env var only affected cuzk-bench (which uses the proof library directly), not the daemon itself.

The Decision: Generating a Runtime Config File

The assistant considered several approaches. Could the daemon be started with a --param-cache flag? A grep of the daemon's CLI argument parsing ([msg 683]) revealed only --listen and --config flags. There was no --param-cache override. This meant the only way to change the daemon's param cache path was through a configuration file.

The solution was elegant: generate a minimal TOML config file at runtime ([msg 684]). The benchmark script would write a small config snippet setting param_cache to $PARAM_DIR and pass it to the daemon via --config. This sidestepped the entire precedence issue by providing the daemon with an explicit configuration that matched the actual parameter location.

The assistant also exported FIL_PROOFS_PARAMETER_CACHE globally ([msg 686]) so that cuzk-bench—which does respect the env var—would also find parameters in the correct directory. This dual approach ensured both the daemon and the benchmark client agreed on where parameters lived.

Assumptions and Their Validation

Several assumptions underpinned this debugging process. The assistant assumed that the daemon's config default would indeed override the environment variable—and this was confirmed by the error output. The assistant assumed that the daemon had no --param-cache CLI flag—confirmed by grepping the source. The assistant assumed that generating a TOML config file was the cleanest fix—this was a design judgment, not a factual claim, but it was reasonable given the architecture.

One implicit assumption worth examining: the assistant assumed that the FIL_PROOFS_PARAMETER_CACHE env var was the intended mechanism for setting the param cache path, and that the daemon's disregard for it was an oversight in the daemon's implementation. This is plausible—the env var is the standard mechanism in the Filecoin proof ecosystem, and the daemon's hardcoded default may simply be a leftover from early development. The fix of generating a config file bridges this gap without modifying the daemon's source code.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, an understanding of the CuZK proving system architecture: the daemon is a long-lived server that handles proof requests, while cuzk-bench is a benchmarking client that submits batch proof jobs. Second, familiarity with the Filecoin proof parameter ecosystem: the large SRS (Structured Reference String) and verification key files that must be downloaded and cached. Third, knowledge of configuration patterns in Rust applications: how struct defaults interact with CLI overrides and config file loading. Fourth, familiarity with TOML format and the Docker build/push workflow.

Output Knowledge Created

This message produces three concrete pieces of knowledge for anyone reading the conversation. First, the correct CLI flags for cuzk-bench batch are --count and --concurrency, not -n and -j. Second, the cuzk-daemon resolves its param_cache from its config struct default (or a config file), not from the FIL_PROOFS_PARAMETER_CACHE environment variable. Third, the workaround for this is to generate a minimal TOML config file at daemon startup time.

The Thinking Process

The reasoning visible in the preceding messages shows a methodical, hypothesis-driven approach. The assistant observed the error, formed two hypotheses (CLI mismatch + param path mismatch), tested the CLI hypothesis by reading the usage output, and investigated the param path mismatch by tracing through the source code. The grep for param_cache|FIL_PROOFS_PARAMETER_CACHE in [msg 680] was a targeted search to understand the relationship between the env var and the config key. The follow-up grep of the daemon's CLI in [msg 683] was a deliberate check for alternative override mechanisms. Each step narrowed the possibilities until the root cause was isolated.

The assistant's decision to generate a TOML config file rather than, say, patching the daemon source or symlinking directories, reflects a pragmatic engineering judgment: the config file approach is non-invasive, reversible, and works within the existing architecture without requiring a recompile or a source change.

Conclusion

Message [msg 689] is a deceptively simple summary of a nuanced debugging session. The three fixes it lists—CLI flag correction, daemon config generation, and env var export—each address a different layer of the system: the client CLI, the daemon configuration, and the proof library environment. Together, they ensure that the benchmark script can successfully start a daemon, locate the correct parameters, and submit proof jobs with the right arguments. The message is the capstone of a debugging chain that demonstrates systematic reasoning, source code investigation, and pragmatic workaround design.