The Birth of a Run Script: Orchestrating the CuZK Daemon in a Docker Container

Introduction

In the sprawling, multi-session effort to build a production-ready Docker image for Filecoin's CuZK proving engine, one message stands out as a quiet but pivotal moment: message 695, where the assistant writes the run.sh script. The message itself is deceptively simple — a single tool call, recorded as [write] /tmp/czk/docker/cuzk/run.sh followed by Wrote file successfully. — yet it represents the culmination of dozens of preceding iterations, the synthesis of hard-won lessons from debugging benchmark scripts, and the first concrete step toward a production deployment workflow. This article examines that message in depth: why it was written, the decisions embedded within it, the assumptions it carries, and the knowledge it both consumes and produces.

Context and Motivation

To understand why run.sh was created at this precise moment, one must trace the arc of the conversation leading up to it. The user and assistant had been building a Docker image (theuser/curio-cuzk:latest) containing the CuZK proving stack — a suite of binaries including curio, sptool, cuzk-daemon, and cuzk-bench. The image was intended for deployment on vast.ai GPU instances, where it would perform Filecoin proof generation at scale.

The immediate predecessor to run.sh was benchmark.sh, a script created earlier in the session to run performance benchmarks of PoRep (Proof of Replication) proofs. That script had gone through multiple iterations: it learned to wait for curio fetch-params to finish downloading parameters before proceeding (see [msg 655]), it acquired logic to auto-detect the parameter directory, it generated a minimal TOML configuration file for the daemon to ensure the param_cache pointed to the correct location (see [msg 684]), and it handled daemon lifecycle management (start, wait, cleanup). But benchmark.sh was designed for testing, not production. It ran a fixed number of proofs, printed timing statistics, and then shut everything down.

The user's request at [msg 693] was explicit: "Create a run script which starts cuzk; will later also start curio but for now just cuzk." This was a shift in purpose — from benchmarking to running. The new script needed to start the CuZK daemon and keep it running, potentially for days or weeks, as part of a live proving operation. The phrase "will later also start curio" foreshadowed that this script would eventually become the container's main entrypoint for production workloads, orchestrating both the proving daemon and the Filecoin storage mining node.

The Decisions Embedded in run.sh

Although the message itself only records the file write, the assistant's subsequent summary at [msg 698] reveals the key design decisions baked into run.sh:

Foreground execution with exec. The script uses exec to replace its own process with the cuzk-daemon binary. This is a critical detail for container environments: when a Docker container's main process runs with exec, signals (like SIGTERM from docker stop) are delivered directly to the daemon rather than to a shell wrapper. This ensures clean shutdown, proper signal handling, and correct PID 1 behavior — all essential for reliable container orchestration.

Auto-generated configuration. Rather than requiring a static config file to be baked into the image or mounted at runtime, run.sh generates a TOML configuration on startup, pointing param_cache at the correct parameter directory. This decision was born from painful experience: earlier, the daemon had crashed because its hardcoded default (/data/zk/params) didn't match where parameters were actually stored (/var/tmp/filecoin-proof-parameters), as seen in the user's crash report at [msg 675]. By generating the config dynamically, the script adapts to the runtime environment.

Waiting for parameter fetch. The script checks for any in-progress curio fetch-params process before starting the daemon. This was a direct response to the vast.ai deployment model, where the on-start script (which fetches parameters) runs in the background while the user connects via SSH. Without this wait, the daemon could start before the 44GB SRS parameter file and 25.7GB PCE file were fully downloaded, leading to immediate failure.

Default listen address. The daemon listens on 0.0.0.0:9820 by default, making it accessible from outside the container. This is appropriate for a service that other components (like curio or cuzk-bench) need to connect to, potentially across a network.

Assumptions Embedded in the Design

Every design decision carries assumptions, and run.sh is no exception. The script assumes that:

  1. The parameter directory is discoverable. It uses the FIL_PROOFS_PARAMETER_CACHE environment variable if set, falling back to /var/tmp/filecoin-proof-parameters. This assumes that either the environment variable is correctly configured by the container's entrypoint or that the default path matches where curio fetch-params stores files.
  2. The daemon binary is on PATH. The script presumably calls cuzk-daemon without an absolute path, relying on the container's PATH environment variable. In the Docker image, this is a safe assumption, but it means the script cannot be easily used outside the container without modification.
  3. The daemon accepts a config file generated inline. The script writes a TOML config to a temporary location and passes it via --config. This assumes the daemon's CLI interface supports this flag (which it does, as confirmed by earlier code inspection at [msg 683]).
  4. No other daemon is already running. The script does not check for an existing cuzk-daemon process on the target port before starting. In a well-orchestrated container environment this is fine, but it could cause confusion if the script is run manually in an already-provisioned instance.
  5. The daemon will eventually become ready. Unlike benchmark.sh, which had a 30-second startup timeout (later increased to 120 seconds at [msg 711]), run.sh starts the daemon in the foreground with exec, so there is no timeout logic — the script blocks indefinitely. This is correct for a production run script, but it means a daemon that hangs during startup (e.g., due to a corrupt parameter file) will stall the container forever.

Input Knowledge Required

To understand and appreciate message 695, a reader needs considerable context:

Output Knowledge Created

Message 695 produces a file — run.sh — that becomes part of the Docker image's operational toolkit. But the output knowledge extends beyond the file itself:

  1. A reusable pattern for daemon orchestration. The script establishes a template: generate config dynamically, wait for prerequisites, exec the daemon. This pattern can be reused for other services (like curio, which the user explicitly mentioned adding later).
  2. A bridge between development and production. Until this point, the only way to start the daemon was via benchmark.sh (which ran a fixed benchmark and then stopped) or manually. run.sh provides a production-oriented entrypoint that keeps the daemon running indefinitely.
  3. A foundation for the vast.ai management system. The later design document vast-cuzk-plan.md (described in the chunk summary) envisions a fleet of instances managed by a controller. Each instance would run run.sh (or a similar script) as its main process. The script's clean signal handling and foreground execution make it suitable for this role.
  4. A documented assumption about parameter paths. By defaulting to /var/tmp/filecoin-proof-parameters but respecting FIL_PROOFS_PARAMETER_CACHE, the script encodes the team's understanding of how parameters flow through the system — from curio fetch-params to the daemon's config.

The Thinking Process

While the message itself does not contain explicit reasoning (it is a simple tool call), the thinking behind it is visible in the surrounding conversation. The assistant had just read benchmark.sh at [msg 694] to understand the existing patterns. The user's request was straightforward — "Create a run script which starts cuzk" — but the assistant's implementation choices reveal a careful consideration of the deployment environment.

The assistant recognized that benchmark.sh contained useful infrastructure (config generation, param dir detection, fetch-params waiting) but was fundamentally a testing tool. A production run script needed to strip away the benchmark-specific logic (warmup proofs, timing output, daemon cleanup) while keeping the infrastructure that made the daemon start correctly. The use of exec was a deliberate choice for container correctness — a detail that might escape a less experienced developer but is crucial for reliable Docker deployments.

The assistant also implicitly understood that the script would be iterated upon. The user's comment "will later also start curio but for now just cuzk" signaled that this was an incremental step. The assistant kept the script simple and focused, adding only what was needed for the daemon, knowing that curio integration would come in a future iteration. This modularity — one script, one responsibility — is a hallmark of good operational design.

Mistakes and Refinements

No first draft is perfect, and run.sh was refined almost immediately. At <msg id=701-702>, the user requested a specific default configuration with explicit values for param_cache, preload, partition_workers, gpu_workers_per_device, and gpu_threads. The assistant applied this edit at [msg 703], expanding the generated config to include these parameters. This suggests that the initial version of run.sh either omitted these settings or used different defaults — a reasonable starting point, but one that needed tuning for the actual hardware (the user's instance had multiple GPUs and substantial RAM).

Additionally, the benchmark script's startup timeout was later increased from 30 to 120 seconds at [msg 711] after the user reported that the daemon took longer than 30 seconds to preload the SRS and PCE files. While run.sh itself didn't have a timeout (it uses exec), this discovery highlighted the scale of the startup cost — and reinforced the wisdom of the "wait for fetch-params" logic, since starting the daemon before parameters were fully downloaded would add even more delay.

Conclusion

Message 695 — the creation of run.sh — is a small but significant step in a much larger journey. It represents the transition from benchmarking to production, from ad-hoc testing to structured deployment. The script itself is unremarkable: a few dozen lines of bash that generate a config file, wait for a download to finish, and start a daemon. But the context that produced it — the debugging of parameter paths, the discovery of CLI flag mismatches, the understanding of vast.ai's asynchronous startup model, the lessons from SRS preloading times — gives it weight. Every line of run.sh encodes a lesson learned, a crash diagnosed, a design pattern validated. In the world of infrastructure engineering, that is what a "simple" script really is: crystallized experience.