The Final Configuration: Connecting a Distributed Storage Cluster to the Public Internet

Introduction

In the lifecycle of any distributed infrastructure project, there comes a moment when the system transitions from being an internally-accessible test cluster to a publicly-reachable service. That moment is captured in message 2127 of this coding session, where the assistant updates the EXTERNAL_LOCALWEB_URL environment variable on two storage nodes to point to a newly-mapped public IP address. Though the message itself consists of only two sed commands followed by service restarts, it represents the culmination of an extensive debugging and configuration effort spanning multiple sessions, and marks the point at which the Filecoin Gateway's distributed S3 storage cluster becomes accessible from the outside world.

The Message

The assistant writes:

Let me update the LocalWeb URLs on both kuri nodes to use the public IP: ```bash # Update kuri1 LocalWeb URL (port 7010) ssh 10.1.232.83 "sudo sed -i 's|EXTERNAL_LOCALWEB_URL=.*|EXTERNAL_LOCALWEB_URL=http://[REDACTED]:7010|' /data/fgw/config/settings.env && sudo systemctl restart kuri"

>

# Update kuri2 LocalWeb URL (port 7011) ssh 10.1.232.84 "sudo sed -i 's|EXTERNAL_LOCALWEB_URL=.*|EXTERNAL_LOCALWEB_URL=http://[REDACTED]:7011|' /data/fgw/config/settings.env && sudo systemctl restart kuri" ```

The public IP address and ports have been redacted here, but the structure is clear: two remote SSH commands, each performing an in-place sed substitution on a configuration file, followed by a systemctl restart of the kuri service.

Context: What Led to This Message

To understand why this message was written, one must trace back through the preceding conversation. The assistant and user had been building and debugging a horizontally-scalable S3 storage cluster based on the Filecoin Gateway architecture. The cluster consists of three physical nodes: a head node running an S3 frontend proxy, and two "kuri" storage nodes (kuri_01 and kuri_02) that actually store the data. The architecture separates the stateless S3 proxy layer from the stateful storage layer, with a shared YugabyteDB database providing metadata persistence.

Earlier in the session, the assistant had verified that the cluster was functioning correctly internally. Load tests showed successful reads and writes with zero data corruption. The cluster topology was visible from both nodes. The S3 proxy was routing requests to both backends with an acceptable load distribution (roughly 62/38 split due to hash-based routing). However, the system was only accessible on the internal network (10.1.232.x range). The user had been waiting on the network team to add internet port mappings that would expose the cluster's LocalWeb CAR (Content-Addressable Retrieval) service to external clients.

The critical trigger came in message 2126, where the user announced: "Port mappings added, from [REDACTED]:7010/7011." This single sentence was the green light the assistant needed. The network team (or the user acting in that capacity) had configured the external firewall to forward ports 7010 and 7011 from a public IP address to the two kuri nodes on the internal network. With this infrastructure change in place, the assistant could now update the nodes' configuration to advertise the correct external URLs.

Why This Message Was Written: The Reasoning

The EXTERNAL_LOCALWEB_URL environment variable controls the URL that the kuri node advertises to clients and other cluster members for its LocalWeb CAR service. LocalWeb is a feature that allows retrieval of content-addressed data (CAR files) directly from a storage node via HTTP. When a client or another node wants to fetch data from a specific kuri node, it needs to know the reachable URL for that node's LocalWeb endpoint.

Before this message, the EXTERNAL_LOCALWEB_URL was likely either unset or set to an internal IP address (e.g., http://10.1.232.83:7010). Internal IPs are not routable from the public internet. If a client outside the internal network tried to access the LocalWeb service using the internal URL, the connection would fail. The port mappings provided by the network team created a bridge: the public IP [REDACTED] on port 7010 forwards to kuri_01's LocalWeb port, and port 7011 forwards to kuri_02's LocalWeb port.

The assistant's reasoning was straightforward: now that the network infrastructure supports external access, the application configuration must be updated to reflect this new reality. Without this update, the LocalWeb URLs would remain internally-focused, defeating the purpose of the port mappings. The cluster would technically be reachable from the internet (thanks to the firewall rules), but the nodes would advertise incorrect URLs, causing clients to attempt connections to unreachable internal addresses.

Decisions Made in This Message

Several implicit and explicit decisions are visible in this brief exchange:

Decision 1: Use sed for configuration updates. The assistant chose to perform an in-place substitution using sed with the pipe delimiter (s|old|new|). This is a pragmatic choice for a single-line environment variable change. Alternatives would include copying a new configuration file via scp, using Ansible (which had been set up earlier in the project), or manually editing the file. The sed approach is fast, minimal, and appropriate for a targeted change.

Decision 2: Restart the service immediately. The && sudo systemctl restart kuri chained command ensures the service picks up the new configuration right away. This is a deliberate choice over a rolling restart or a deferred restart. The assistant judged that the change was safe enough to apply immediately without a staged rollout.

Decision 3: Apply the change to both nodes simultaneously. The two SSH commands are independent and could be run in parallel. The assistant chose to write them sequentially in the message, but they execute independently. This reflects an assumption that both nodes can be updated at the same time without coordination.

Decision 4: Use the exact same public IP for both nodes, differentiated only by port. The assistant maps kuri_01 to port 7010 and kuri_02 to port 7011, matching the user's announcement. This is a one-to-one mapping between internal nodes and external ports, which is the simplest possible routing strategy.

Assumptions Made

The message rests on several assumptions, some explicit and some implicit:

Assumption 1: The port mappings are working correctly. The assistant assumes that the network team has correctly configured the firewall to forward ports 7010 and 7011 to the respective internal IPs. There is no verification step in the message — no curl test, no connectivity check. The assistant trusts the user's statement that the mappings were added.

Assumption 2: The EXTERNAL_LOCALWEB_URL variable exists in the settings file. The sed pattern EXTERNAL_LOCALWEB_URL=.* assumes that this variable is already present in the file. If it were missing, the sed command would silently do nothing, and the service would restart without the URL being updated. This is a reasonable assumption given that the configuration was generated by Ansible templates earlier in the project.

Assumption 3: The kuri service reads this variable at startup. The assistant assumes that restarting the service will cause it to re-read the environment file and pick up the new URL. This is consistent with the systemd service configuration, which uses EnvironmentFile=/data/fgw/config/settings.env.

Assumption 4: The public IP is stable. The assistant does not add any DNS name or dynamic resolution. The URL is hardcoded to the IP address. This assumes the IP will not change, which is typical for static public IP allocations but could become a maintenance burden if the IP ever changes.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of:

  1. The cluster architecture: Two kuri storage nodes (kuri_01, kuri_02) running on internal IPs 10.1.232.83 and 10.1.232.84, each serving a LocalWeb CAR retrieval service on a specific port.
  2. The configuration system: Environment variables stored in /data/fgw/config/settings.env, loaded by systemd via EnvironmentFile.
  3. The EXTERNAL_LOCALWEB_URL variable: Its purpose in advertising the node's reachable URL for CAR retrieval.
  4. The network topology: The head node at 10.1.232.82, the internal network range, and the newly-added public IP with port forwarding.
  5. The service management model: Systemd units named kuri that can be restarted with systemctl restart.
  6. The sed command syntax: In-place substitution with the pipe delimiter, and the regex pattern s|EXTERNAL_LOCALWEB_URL=.*|EXTERNAL_LOCALWEB_URL=http://...|.

Output Knowledge Created

This message produces several concrete outcomes:

  1. Updated configuration files: Both /data/fgw/config/settings.env files now contain the public URL instead of the internal one.
  2. Restarted services: The kuri systemd unit on both nodes has been restarted, causing a brief service interruption (typically a few seconds).
  3. Externally-reachable LocalWeb URLs: Clients outside the internal network can now access CAR retrieval at http://[REDACTED]:7010 (kuri_01) and http://[REDACTED]:7011 (kuri_02).
  4. Correct cluster advertisements: When the cluster topology is queried, the nodes will advertise the correct public URLs, allowing external clients to discover and connect to the appropriate node.

Mistakes or Incorrect Assumptions

While the message is straightforward, there are potential issues worth examining:

No verification step. The most notable omission is the lack of a verification command after the restart. The assistant does not check that the service came back up successfully, that the environment variable was actually updated, or that the external URL is reachable. In the context of the broader session, this is understandable — the assistant had been performing extensive verification throughout the session, and the user could easily check the LocalWeb URLs themselves. However, in a production deployment, a verification step (e.g., curl http://[REDACTED]:7010/health) would be prudent.

Simultaneous restart risk. Restarting both kuri nodes at the same moment creates a brief window where both storage nodes are unavailable. For a cluster with only two nodes, this means zero redundancy during the restart. A more cautious approach would be to restart one node, verify it comes back, then restart the second. However, given that the LocalWeb URL change is a configuration update that doesn't affect data integrity, and the restart is typically fast (seconds), the risk is minimal.

No validation of the sed pattern. The sed command uses EXTERNAL_LOCALWEB_URL=.* as the search pattern. If the variable appeared multiple times in the file (unlikely but possible), only the first occurrence would be replaced. If the variable had a different format (e.g., with quotes around the value), the replacement might produce a malformed line. The assistant assumes the file format is consistent with what Ansible generated.

The Broader Significance

This message, for all its brevity, marks a significant milestone in the infrastructure project. The cluster had been built, debugged, load-tested, and validated internally. The topology was correct, the data was distributed, the proxy was routing, and the monitoring was functional. But none of that mattered if external clients couldn't reach the service. The port mappings and the subsequent URL update were the final bridge between the internal cluster and the outside world.

In many ways, this message encapsulates the nature of infrastructure engineering: the most critical changes are often the simplest in terms of code, but they carry the weight of all the work that came before. A two-line sed command is trivial to write, but it only makes sense because dozens of previous messages had debugged database migrations, fixed configuration syntax, corrected architectural errors, and validated deployment pipelines. The message is the capstone on a long chain of effort.

Conclusion

Message 2127 is a study in minimalism and timing. The assistant writes two SSH commands, each performing a string substitution and a service restart. The reasoning is clear: the network team has provided public port mappings, so the application configuration must be updated to use them. The decisions are pragmatic: use sed for speed, restart immediately for effect, and trust the user's announcement. The assumptions are reasonable given the context, though a verification step would strengthen the change.

In the broader narrative of the coding session, this message represents the transition from internal validation to external availability. The Filecoin Gateway's distributed S3 cluster is no longer just a well-functioning internal system — it is now a publicly-accessible service, ready for real-world use.