When Curio Can't Connect: Diagnosing a Port Forwarding Failure in a Distributed Proving System

The Message

In the middle of an extensive overhaul of the vast-manager deployment platform, the user reported a critical operational failure with a single, concise error log:

[user] Curio failed to start - [curio]repopath /var/lib/curio
[curio]2026-03-12T10:23:26.948Z	INFO	repo	repo/fsrepo.go:230	Initializing repo at '/var/lib/curio'
[curio]2026-03-12T10:23:26.948Z	INFO	harmonyquery	harmonyquery@v0.0.0-20260127224413-4c39280f279e/harmonyquery.go:150	Yugabyte connection config: loadBalance=true, hosts=[127.0.0.1], port=5433
[curio]2026-03-12T10:23:28.432Z	WARN	curio/chain	deps/apiinfo.go:65	Not able to establish connection to node with addr: ws://127.0.0.1:1234/rpc/v1, Reason: cannot dial address ws://127.0.0.1:1234/rpc/v1 for dial tcp 127.0.0.1:1234: connect: connection refused: dial tcp 127.0.0.1:1234: connect: connection refused
[curio]ERROR: failed to establish connection with all chain nodes  - on ...742; try to fix the instance before restarting

This message appears at first glance to be a simple startup failure report. But within the broader context of the system being built—a distributed Filecoin proving network managed by a custom orchestration platform called vast-manager—this log snippet represents a critical symptom of a deeper infrastructure bug. Understanding why this message was written, what it reveals, and how it drove the next phase of development requires unpacking the architecture, the deployment pipeline, and the assumptions that led to this failure.

The Context: A Distributed Proving Network Under Construction

To understand this message, one must first understand the system it belongs to. The user and assistant were building a distributed proving network for Filecoin's Curio protocol, deployed across rented GPU instances from Vast.ai. The architecture involved several layers:

1. Curio: The proving software that performs storage proofs (WindowPoSt, WinningPoSt, SnapDeals) on Filecoin. Each Curio instance needs to connect to a Lotus node (the Filecoin blockchain node) via a WebSocket API to receive tasks and submit proofs.
2. Lotus API on port 1234: The Lotus node exposes a WebSocket RPC endpoint on port 1234 (ws://127.0.0.1:1234/rpc/v1). Curio connects to this endpoint to communicate with the blockchain.
3. portavailc tunnel: A port forwarding/tunneling mechanism used to expose the Lotus API from the controller host (where the Lotus node runs) to worker instances. The tunnel forwards ports from the controller to the worker, allowing Curio on the worker to connect to 127.0.0.1:1234 as if the Lotus node were local.
4. vast-manager: A custom management service that orchestrates instance lifecycle—registration, parameter fetching, benchmarking, and deployment—across Vast.ai GPU instances. The critical architectural detail is that Curio on each worker instance connects to the Lotus API via a tunnel. The tunnel is set up in the instance's entrypoint script (entrypoint.sh), which runs when the Vast.ai container starts. If the tunnel doesn't forward port 1234, Curio cannot connect to the Lotus node and fails to start.

What the Error Log Reveals

The log shows a clear failure sequence:

1. Initialization succeeds: Curio initializes its repo at /var/lib/curio and configures the Yugabyte connection (the metadata database). These steps complete successfully.
2. Connection attempt fails: Curio tries to connect to ws://127.0.0.1:1234/rpc/v1 and gets connection refused. This means the Lotus API is not listening on port 1234 on the local machine.
3. Startup aborts: Curio reports that it "failed to establish connection with all chain nodes" and exits with an error, advising the user to "fix the instance before restarting." The key diagnostic insight is that Curio is looking for the Lotus API at 127.0.0.1:1234—a localhost address. This confirms that the system design relies on a tunnel or proxy to make the remote Lotus node appear local. The connection refusal means the tunnel is either not set up, not forwarding the correct port, or the Lotus node itself is not running.

The Deeper Problem: A Missing Port in the Tunnel Configuration

The user's message was not merely a status update—it was a targeted bug report that triggered a specific debugging and remediation process. The assistant, upon seeing this log, would need to determine why port 1234 was not reachable. The possibilities include:

• The Lotus node on the controller host is down or not listening on port 1234.
• The tunnel (portavailc) is not configured to forward port 1234.
• The tunnel process failed to start or crashed.
• A firewall or network issue is blocking the connection. Given the context of the ongoing work (the vast-manager overhaul in segment 9), the most likely culprit was the tunnel configuration. The assistant had recently been iterating on the entrypoint script (entrypoint.sh) that sets up the portavailc tunnel. The tunnel was designed to forward specific ports needed by Curio, but port 1234—the Lotus WebSocket API—had apparently been omitted. This is a classic infrastructure bug: a configuration oversight where a critical dependency (the Lotus API port) was not included in the tunnel forwarding rules. The system worked for other functions (parameter fetching, Yugabyte connectivity) but failed at the final step of connecting to the blockchain node.

Assumptions and Their Consequences

Several assumptions contributed to this failure:

1. Assumption that the tunnel forwarded all necessary ports: The entrypoint script was written with a specific set of ports, but the full list of ports Curio needs was not verified against the tunnel configuration. The Lotus API port (1234) was assumed to be available or was simply overlooked.
2. Assumption that a successful partial startup implies full readiness: Curio initializes its repo and connects to Yugabyte before attempting the Lotus connection. A developer monitoring startup might see the INFO logs and assume everything is working, only to miss the WARN/ERROR that follows.
3. Assumption about the tunnel's default behavior: The portavailc tool might have been assumed to forward all ports by default, or to use a configuration file that included port 1234. In reality, the tunnel was explicitly configured with a subset of ports.
4. Assumption that the Lotus node runs on the worker: The architecture of running Lotus on a separate controller and tunneling to workers is a common pattern, but it introduces a single point of failure. If the tunnel is misconfigured, every worker instance fails identically.

Input Knowledge Required

To fully understand this message, a reader needs:

• Knowledge of Filecoin's Curio architecture: Understanding that Curio is a proving daemon that requires a connection to a Lotus node via WebSocket.
• Knowledge of the portavailc tunneling mechanism: Understanding that the tunnel makes remote ports appear local on the worker instance.
• Knowledge of the deployment topology: The controller host runs the Lotus node, and workers connect via tunnels. This is not obvious from the log alone.
• Knowledge of the ongoing development context: The vast-manager system was being actively developed, and the entrypoint script had been modified multiple times in recent sessions.

Output Knowledge Created

This message created critical operational knowledge:

1. A specific, reproducible failure mode: Curio fails to start if port 1234 is not forwarded. This becomes a checklist item for deployment verification.
2. A diagnostic pattern: The connection refused on 127.0.0.1:1234 is a reliable indicator of a missing tunnel port, not a Curio bug.
3. A remediation requirement: The entrypoint script must be updated to include port 1234 in the portavailc tunnel configuration.
4. A broader lesson in dependency mapping: Every port that a service needs must be explicitly enumerated and verified in the infrastructure configuration. Implicit assumptions about port availability are a common source of deployment failures.

The Thinking Process Behind the Report

The user's decision to report this failure as a concise log dump rather than a descriptive narrative reveals a sophisticated debugging approach. Instead of saying "Curio doesn't work," the user provided the exact error output, enabling the assistant to:

1. See the exact error message and stack trace.
2. Identify the specific port (1234) and protocol (WebSocket) involved.
3. Determine that the connection was refused (not timed out, not blocked by firewall—the port simply had nothing listening).
4. Rule out other failure modes (the repo initialization and Yugabyte connection succeeded, so the issue is specifically with the Lotus API connection). This is a pattern of effective bug reporting: provide the raw evidence, not the interpretation. The user trusted the assistant to interpret the logs correctly, and the assistant could immediately focus on the port forwarding configuration rather than wasting time on other potential causes.

Conclusion

Message 1408 is a deceptively simple error report that encapsulates a critical infrastructure lesson: in distributed systems, the difference between a working deployment and a broken one is often a single missing configuration detail. The Curio startup failure on port 1234 was not a software bug in Curio or Lotus—it was a gap in the tunnel configuration that assumed a critical port would be available without explicitly forwarding it.

This message drove the next phase of work: updating the entrypoint script to include port 1234 in the portavailc tunnel, rebuilding the Docker image, redeploying to worker instances, and verifying that Curio could establish its Lotus connection. It also reinforced the importance of comprehensive dependency enumeration in infrastructure-as-code systems. In a proving network spanning multiple continents, a single missing port number can bring the entire deployment pipeline to a halt—and the only way to catch it is through careful log analysis and a willingness to treat every startup failure as a clue to a deeper configuration issue.