The Missing Port: How a Single Omission in a Tunnel Configuration Brought Down a Proving Cluster

The Message

There's the problem: port 1234 is not forwarded by portavailc. The tunnel only forwards -L 1235 -L 5433 -L 9042 but lotus is on port 1234. The portavaild config has --ports 1234,1235,5433,9042,4701 but the client side doesn't request 1234.

>

[edit] /tmp/czk/docker/cuzk/entrypoint.sh Edit applied successfully.

This message, from an opencode coding session building a distributed Filecoin proving infrastructure, is deceptively short. In just three sentences and an edit command, the assistant identifies the root cause of a complete service failure on a newly deployed worker instance, traces it to a configuration mismatch between two sides of a tunnel, and applies the fix. But behind this concise diagnosis lies a methodical debugging chain that reveals how complex distributed systems fail, how assumptions compound, and how the smallest configuration drift can render an entire deployment pipeline inoperable.

The Symptom: Curio Won't Start

The story begins with a user report ([msg 1408]): a worker instance running the Curio proving daemon cannot start. The error log is unambiguous:

[curio]WARN curio/chain  deps/apiinfo.go:65  Not able to establish connection to node with addr: ws://127.0.0.1:1234/rpc/v1
[curio]ERROR: failed to establish connection with all chain nodes

Curio, the proving orchestrator, requires a connection to a Lotus Filecoin node API to function. That API is supposed to be available at 127.0.0.1:1234 — but the connection is refused. The worker instance is isolated behind NAT on a Vast.ai GPU rental, so the Lotus node cannot be local; it must be tunneled from the controller host through the PortAvail secure forwarding system.

The assistant's first instinct is to check the server side ([msg 1409]). It SSHes into the controller (10.1.2.104) and confirms that portavaild — the PortAvail server daemon — is running and correctly configured to forward port 1234. A quick ss -tlnp check ([msg 1410]) confirms that Lotus is indeed listening on port 1234 on the controller. The server side is healthy.

This is the critical fork in the debugging road. The server has port 1234 open and ready. The client cannot reach it. The problem must be in the tunnel client — portavailc — running on the worker instance.

Tracing the Tunnel

The assistant pivots to the entrypoint script that governs the worker instance's startup sequence ([msg 1411]). A grep for portavailc|PAVAIL|tunnel reveals the relevant section, and the assistant reads the file ([msg 1412]). The entrypoint's tunnel setup block is straightforward: it runs portavailc with a set of -L (local forward) flags, each mapping a local port through the tunnel to the controller.

And there it is. The assistant scans the port list and sees -L 1235 -L 5433 -L 9042. Port 1234 is absent. The server (portavaild) is configured with --ports 1234,1235,5433,9042,4701 — a superset that includes 1234 — but the client (portavailc) only requests a subset. The tunnel simply never opens a path for port 1234 traffic.

The assistant's diagnosis in message 1413 crystallizes this mismatch in a single, precise sentence: "The tunnel only forwards -L 1235 -L 5433 -L 9042 but lotus is on port 1234."

Why This Happened

The omission of port 1234 from the client-side tunnel configuration is a classic configuration drift error. The PortAvail system has two configuration points: the server daemon (portavaild) declares which ports it is willing to forward, and the client (portavailc) declares which ports it actually wants forwarded. These two lists must intersect — the client cannot forward a port the server doesn't offer, but equally, the server offering a port means nothing if the client never asks for it.

In this case, the server's port list had been updated to include port 1234 (the Lotus API) at some point, likely when the Lotus dependency was added to the proving pipeline. But the client-side invocation in entrypoint.sh was never updated to match. The entrypoint had been iterated many times across the session — it was originally written for parameter fetching and benchmarking, and the port forwarding list was built incrementally. Port 1235 was needed for the vast-manager API, port 5433 for YugabyteDB, port 9042 for the Curio API. Port 1234, the Lotus API, was simply forgotten.

This is a mistake born of two systems evolving independently. The portavaild configuration and the entrypoint.sh script live in different files, are maintained by different reasoning contexts, and are updated at different times. Without a single source of truth or an automated check that the client requests all ports the server offers, drift is inevitable.

The Fix

The fix itself is trivial — a single edit to entrypoint.sh to add -L 1234 to the portavailc command line. The assistant applies it with an edit tool call and confirms "Edit applied successfully." In terms of lines changed, it is a one-character addition. In terms of operational impact, it is the difference between a worker that boots successfully and one that crashes immediately with a connection error.

The assistant does not stop to test the fix in this message — that will come in subsequent rounds. The edit is applied, and the next deployment will carry the corrected entrypoint. But the brevity of the message should not obscure the depth of the reasoning that preceded it.

The Broader Pattern

This debugging episode illustrates a recurring theme in distributed systems engineering: the asymmetry between server and client configuration. When a service fails, the natural instinct is to check the server — is it running? Is it listening? Is the port open? The assistant dutifully performs these checks and finds the server healthy. Only then does it pivot to the client, where the actual fault lies.

The pattern is also one of configuration surface area. The PortAvail tunnel has two configuration files, two sets of port lists, and two deployment contexts (controller and worker). Any mismatch between them produces a silent failure — the tunnel establishes, the worker thinks it has connectivity, but the specific port needed by Curio is simply not forwarded. The error message ("connection refused") is indistinguishable from the server being down, which is why the assistant had to rule that out first.

There is also an interesting assumption embedded in the assistant's debugging approach. When it first checks the controller ([msg 1409]), it sees portavaild running with --ports 1234,1235,5433,9042,4701 and implicitly assumes that the client side must be correctly configured because the server side is. This is a natural assumption — if the system was working before, why would the client be missing a port? But the assumption is wrong, and the assistant corrects it within two messages by actually reading the client configuration.

Conclusion

Message 1413 is a masterclass in concise root-cause diagnosis. In three sentences, the assistant identifies a configuration mismatch between a server and its client, traces it to a specific missing flag in a shell script, and applies the fix. The message is the payoff of a methodical debugging chain that checked the server, verified the service, read the client configuration, and compared the two port lists.

The missing port 1234 is a small thing — a single integer in a single command-line argument. But in distributed systems, small things compound. A port not forwarded means a service cannot start. A service that cannot start means a GPU sits idle. A GPU that sits idle means proofs are not generated. And proofs not generated mean lost revenue. The assistant's ability to trace from "curio failed to start" to "port 1234 is missing from portavailc" in under four messages is exactly the kind of systems thinking that separates effective debugging from aimless tinkering.

The fix is applied. The entrypoint is updated. The next worker instance will forward port 1234, Curio will find its Lotus API, and the proving pipeline will hum along. But the lesson lingers: in any system with multiple configuration surfaces, the client and server must sing from the same song sheet — and the only way to catch a missing verse is to read both sides.