The Docker Entrypoint Trap: Debugging CLI Discovery in a Containerized Proving Engine

Introduction

In distributed systems engineering, the simplest operations often conceal the most instructive failure modes. Message 1174 of this opencode session captures a seemingly trivial moment: an engineer runs a help command on a CLI tool inside a Docker container, discovers the entrypoint script is intercepting the call, and overrides it. But this two-line debugging interaction sits at the confluence of several converging crises — OOM-killed GPU instances, gRPC transport errors, benchmark timeouts, and the fundamental tension between containerized deployment and interactive debugging.

The message reads in full:

[assistant] The entrypoint is running instead of cuzk-bench. Let me override: [bash] docker run --rm --entrypoint cuzk-bench curio-cuzk:latest --help 2>&1 | head -40 cuzk proving engine test/benchmark utility

>

Usage: cuzk-bench [OPTIONS] <COMMAND>

>

Commands: single Run a single proof through the daemon batch Run N identical proofs and report throughput statistics status Query daemon status preload Pre-warm SRS parameters metrics Get Prometheus metrics from the daemon gen-vanilla Generate vanilla proof test data for PoSt/SnapDeals (requires gen-vanilla feature) pce-bench PCE (Pre-Compiled Constrai...

What makes this message worthy of deep analysis is not the command itself but the chain of reasoning that led to it, the assumptions it tests, and the knowledge it produces. It is a message about how to ask a question of a system that was not designed to answer it.

The Context: Two Instances, Two Failures

To understand why this message was written, we must trace the events of the preceding hour. The session had deployed two GPU instances on Vast.ai for benchmarking the CuZK proving engine: a Belgium instance with 2× NVIDIA A40 GPUs and 2 TB of RAM, and a Czechia instance with 2× RTX 3090 GPUs and 251 GB of RAM. Both had been configured with dynamic hardware-aware pipeline settings — partition_workers scaled to available memory, concurrency scaled to GPU count and RAM — a fix developed in the preceding segment to resolve OOM crashes that had plagued low-memory instances.

Both instances failed, but in diagnostically distinct ways.

The Belgium instance (2× A40, 2 TB RAM) was killed by the vast-manager's monitor after 20 minutes in the params_done state. The manager had a hardcoded timeout: if an instance remained in params_done for more than 20 minutes without transitioning to bench_done, it was assumed to be stuck and was destroyed. But the Belgium instance was not stuck — it was running a 12-proof benchmark with concurrency 6, and the warmup proof (with PCE extraction) alone took 5 minutes. The actual benchmark needed 15–20 more minutes. The total elapsed time from params_done to completion was ~25–30 minutes, well beyond the 20-minute window. The assistant diagnosed this in [msg 1159] and increased the timeout to 45 minutes.

The Czechia instance (2× RTX 3090, 251 GB RAM) failed differently. Its warmup completed successfully — the PCE was generated, the daemon restarted with partition_workers=10, and the batch benchmark began. But the very first proof in the batch immediately failed with a gRPC transport error: status: Unknown, message: &#34;transport error&#34;. The daemon logs showed it was alive and processing — both proofs had been accepted, all 10 partitions were synthesizing with the PCE fast path. The problem was not the daemon dying but the gRPC client timing out while waiting for the first proof to complete. After a daemon restart, the GPU kernels need to be compiled and cached, making the first proof significantly slower. The client's default timeout was too short.

The Search for a Timeout Flag

The assistant's reasoning, visible across messages 1170–1174, follows a classic debugging pattern: identify the symptom (gRPC transport error), hypothesize a cause (client timeout too short), and search for a configuration parameter to adjust it.

The first attempt, in [msg 1170], was to run cuzk-bench --help on the remote host directly:

[bash] ssh 10.1.16.218 'su curio -c "/usr/local/bin/cuzk-bench --help"' 2>/dev/null | head -30

This returned nothing — the binary was not installed on the host system. The assistant tried variations in [msg 1171] and [msg 1172], checking for cuzk-bench at different paths and with different flag formats, but each returned the same result: No such file or directory.

At this point, the assistant made a critical inference: cuzk-bench is only available inside the Docker container. This is a reasonable assumption given the deployment architecture — the Docker image theuser/curio-cuzk:latest bundles the CuZK proving engine, its dependencies, and the benchmark utility together. The host system only runs the container; all tooling lives inside it.

The third attempt, in [msg 1173], was:

[bash] docker run --rm curio-cuzk:latest cuzk-bench --help 2>&1 | head -40

But this produced:

[entrypoint] 03:13:30 No PAVAIL set, skipping tunnel (assuming local/direct connectivity)

The entrypoint script was running instead of cuzk-bench. This is the classic Docker ENTRYPOINT vs CMD interaction: when you specify a command after the image name in docker run, Docker passes it as arguments to the image's ENTRYPOINT, not as a replacement for it. The image's entrypoint.sh received cuzk-bench --help as arguments, printed its tunnel-skipping log line, and then presumably either ignored the arguments or passed them to the default CMD (which might be the Curio daemon, not the benchmark tool).

The Insight and the Override

Message 1174 captures the moment of realization. The assistant writes:

The entrypoint is running instead of cuzk-bench. Let me override:

This is the key reasoning step. The assistant recognizes the Docker ENTRYPOINT pattern and knows the fix: the --entrypoint flag overrides the image's default entrypoint, allowing direct execution of any binary inside the container.

The corrected command:

docker run --rm --entrypoint cuzk-bench curio-cuzk:latest --help 2>&1 | head -40

This runs cuzk-bench --help directly, bypassing entrypoint.sh entirely. The output reveals the top-level command structure of cuzk-bench:

Assumptions and Their Consequences

Several assumptions underpin this message, some correct and some problematic.

Correct assumption: The cuzk-bench binary exists inside the Docker image. This was confirmed by the successful --entrypoint invocation. The assistant correctly inferred from the deployment architecture that the benchmark tool would be bundled in the image rather than installed on the host.

Correct assumption: The Docker image uses an ENTRYPOINT script. The log line [entrypoint] 03:13:30 No PAVAIL set, skipping tunnel confirmed this. The assistant recognized this pattern immediately and knew the --entrypoint override mechanism.

Problematic assumption: That --help on the top-level binary would reveal the batch subcommand's timeout flag. In many CLI tools, --help on a subcommand (cuzk-bench batch --help) shows subcommand-specific options, while the top-level --help only shows available commands. The assistant's head -40 truncation compounded this — even if the top-level help had included some subcommand details, they were cut off.

Unstated assumption: That a --timeout flag exists at all. The gRPC timeout might be configured at the daemon level, in a configuration file, or in the client code itself — not necessarily exposed as a CLI flag. The assistant was searching for a configuration knob that might not exist in the form expected.

Input Knowledge Required

To understand this message, a reader needs:

  1. Docker fundamentals: The ENTRYPOINT vs CMD distinction, how docker run &lt;image&gt; &lt;command&gt; passes arguments to ENTRYPOINT, and the --entrypoint override flag. Without this knowledge, the assistant's realization ("The entrypoint is running instead of cuzk-bench") and the fix (--entrypoint cuzk-bench) are incomprehensible.
  2. The deployment architecture: That cuzk-bench is a CLI tool inside a Docker image, not installed on the host. The failed SSH attempts to run it directly (messages 1170–1172) establish this context.
  3. The gRPC timeout problem: The Czechia instance's failure with "transport error" and the assistant's hypothesis that the client timeout is too short. The search for --timeout is driven by this specific debugging need.
  4. The session's history: The Belgium timeout fix (increasing from 20 to 45 minutes), the OOM fixes from the previous segment, and the hardware-aware pipeline configuration. The message sits within a larger narrative of stabilizing a distributed proving system.

Output Knowledge Created

This message produces several pieces of knowledge:

  1. The cuzk-bench command structure: The top-level commands are now known: single, batch, status, preload, metrics, gen-vanilla, and pce-bench. This is the first time in the session that the tool's interface has been documented.
  2. The Docker entrypoint behavior: Confirmed that the image uses entrypoint.sh as its ENTRYPOINT, and that the tunnel-skipping logic runs before any command. This is useful for future debugging — any docker run invocation without --entrypoint will go through the entrypoint script.
  3. A debugging technique: The pattern of using docker run --rm --entrypoint &lt;binary&gt; &lt;image&gt; --help to inspect CLI tools inside containers is established. This is reusable knowledge for any future work with this image.
  4. Negative knowledge: The --help output was truncated, and the batch subcommand's options were not revealed. The assistant now knows it needs to run cuzk-bench batch --help (without the head -40 truncation) to find the timeout flag.

The Thinking Process

The assistant's reasoning, reconstructed from messages 1170–1174, follows this chain:

  1. Symptom observation: Czechia's first batch proof failed with "transport error" (gRPC). The daemon was alive and processing, so the failure is on the client side.
  2. Hypothesis formation: The gRPC client has a timeout that's too short for the first proof after daemon restart (which includes GPU kernel compilation overhead).
  3. Solution search: Look for a --timeout flag on cuzk-bench batch to increase it.
  4. Attempt 1: Run cuzk-bench --help on the host via SSH. Fails — binary not found.
  5. Inference: The binary is only inside the Docker container.
  6. Attempt 2: Run docker run curio-cuzk:latest cuzk-bench --help. Fails — entrypoint script runs instead.
  7. Realization: The ENTRYPOINT is intercepting the command. The assistant recognizes this Docker behavior and formulates the fix.
  8. Attempt 3: Use --entrypoint cuzk-bench to bypass the entrypoint. Success — help output is displayed.
  9. Partial result: The top-level commands are visible, but head -40 truncates the output before the batch subcommand details. The timeout flag is not yet found. The thinking is iterative and hypothesis-driven. Each failed attempt produces information that refines the next attempt. The assistant does not panic or restart — it adapts its approach based on what each failure reveals about the system's structure.

Mistakes and Incorrect Assumptions

The most significant mistake is the head -40 truncation. The assistant was running this command interactively and wanted to avoid overwhelming output, but the truncation cut off the very information being sought. The batch subcommand's options — including any --timeout flag — would appear under cuzk-bench batch --help, not under the top-level --help. Even without truncation, the top-level help would not have shown subcommand-specific flags.

A secondary issue: the assistant assumed the timeout was a client-side parameter. It could equally be a server-side configuration in the daemon, or a compiled-in constant in the gRPC client code. The search for a CLI flag might be fruitless if the timeout is not configurable at runtime.

The assistant also assumed that cuzk-bench was the right tool to configure. The gRPC timeout might be a property of the Curio daemon's RPC server, not the benchmark client. The cuzk-bench tool might only control how long it waits for the daemon to respond, which is a different parameter than the daemon's proof processing timeout.

Conclusion

Message 1174 is a small but revealing moment in a complex debugging session. It shows the assistant navigating the gap between a containerized deployment and the need for interactive tool inspection — a gap that every engineer working with containers eventually encounters. The Docker ENTRYPOINT trap is a classic gotcha, and the assistant's recognition of it demonstrates both experience with container tooling and a methodical approach to debugging.

The message also illustrates a broader principle: when a system fails, the first challenge is often not fixing the bug but figuring out how to ask the right questions of the system. The assistant had to discover how to invoke cuzk-bench before it could discover whether a --timeout flag existed. Each layer of indirection — SSH, Docker, ENTRYPOINT — added friction to the inquiry. The --entrypoint override was the key that unlocked access to the tool's interface.

What makes this message worth studying is not the command itself but the reasoning chain it represents: observe, hypothesize, attempt, fail, infer, adapt, retry. It is debugging in miniature, compressed into a single message but containing the full arc of diagnostic reasoning.