The Art of Anticipating Confusion: A Documentation-First Approach to Distributed Systems Debugging

Introduction

In the middle of building a horizontally scalable S3-compatible storage cluster, a seemingly simple question from the user triggered a cascade of careful reasoning, architectural clarification, and ultimately, a documentation-first response. The message at index 208 of this coding session is deceptively brief—a single line of agent reasoning followed by a file read operation. But to understand why this message was written, we must unpack the entire chain of thought that led to it, the port conflict concern that preceded it, and the broader philosophy of test cluster design that it reveals.

The subject message reads in full:

## Agent Reasoning Let me also add a troubleshooting section about accessing individual node web UIs if needed. [read] /home/theuser/gw/test-cluster/README.md

This is not a message that implements a feature, fixes a bug, or introduces new architecture. It is a message about documentation—specifically, about proactively preventing future confusion by adding explanatory material to a README file. Understanding why this seemingly minor action was taken requires tracing the reasoning that led to it.

The Port Conflict Spark

The story begins at message 202, where the user asked a sharp operational question: "Is the :9010 cluster mon not going to confiict with :9010 on individual kuri nodes?" This question revealed a deep understanding of the system's networking topology. The user had noticed that the cluster monitoring web UI was exposed on port 9010, and that individual Kuri storage nodes also bind to port 9010 for their own web interfaces. In a Docker Compose setup where multiple containers are running simultaneously, this looked like a potential port collision.

The assistant's initial response (message 203) is a masterclass in careful reasoning. It walks through the Docker networking model step by step:

  1. It checks the docker-compose.yml to confirm the actual port mappings
  2. It realizes that Kuri nodes don't expose 9010 externally—they only bind to it inside their containers
  3. It recognizes that Docker's internal network isolation means there is no technical port conflict
  4. But it also acknowledges the user's deeper concern: if someone wants to debug individual Kuri nodes by accessing their web UIs, they currently can't, because those ports aren't mapped to the host This distinction between technical correctness and operational clarity is crucial. The assistant could have simply said "there's no conflict because Docker isolates container ports" and moved on. Instead, it recognized that the user was raising a legitimate usability concern, not just a technical one.

The Decision Cascade

Messages 204 through 207 show the assistant acting on this insight. It makes three concrete changes to the docker-compose.yml:

  1. It disables the web UI on kuri-1 by adding RIBS_WEBUI_BINDADDR= (empty string disables it)
  2. It does the same for kuri-2
  3. It adds a comment to the docker-compose.yml explaining the port setup rationale Then in message 207, it updates the README to reflect these changes. But the assistant isn't done yet. After making these changes, it pauses and thinks about what else might be needed. This brings us to message 208—the subject of this article.

Message 208: The Documentation Instinct

The reasoning in message 208 is: "Let me also add a troubleshooting section about accessing individual node web UIs if needed."

This is a remarkable moment because it represents a shift from reactive problem-solving to proactive documentation. The assistant has already fixed the port configuration. The technical issue is resolved. But the assistant recognizes that someone reading the README in the future might have the same question the user just asked. They might wonder: "What if I need to access an individual Kuri node's web UI for debugging? How do I do that?"

The assistant's decision to add a troubleshooting section is driven by several assumptions:

Assumption 1: The README is the primary entry point for new users. The assistant assumes that anyone setting up the test cluster will start by reading the README. If the README doesn't address common questions, users will either ask them (slowing down development) or make incorrect assumptions (causing errors).

Assumption 2: Individual node debugging is a legitimate use case. Even though the cluster monitoring UI provides an aggregated view, there will be situations where an operator needs to inspect a single Kuri node's internal state. The assistant assumes this need will arise and plans for it.

Assumption 3: The port configuration change needs explanation. By disabling the web UI on individual Kuri nodes, the assistant made a design decision that might not be obvious to someone reading the configuration. A troubleshooting section can explain why the web UI is disabled and how to re-enable it if needed.

Assumption 4: Documentation prevents support burden. The assistant is implicitly recognizing that every unanswered question in the documentation will eventually become a question asked to the developers. By proactively adding a troubleshooting section, the assistant reduces future cognitive load on both users and developers.

The Input Knowledge Required

To understand message 208, one needs several pieces of context:

  1. The Docker networking model: Understanding that containers have isolated network namespaces, that port mappings are explicit, and that internal container ports don't conflict with each other or with host ports unless explicitly mapped.
  2. The Kuri node architecture: Knowing that each Kuri storage node runs its own web UI on port 9010 by default, and that this web UI provides node-specific debugging information.
  3. The cluster monitoring architecture: Understanding that the cluster monitoring UI (also on port 9010) is a separate service that aggregates data from all nodes, and that it serves a different purpose from individual node web UIs.
  4. The conversation history: Knowing that the user previously asked about running a test cluster, that the assistant built the docker-compose setup, and that the user requested parameterized data directories and removal of hardcoded paths.
  5. The RIBSWeb application structure: Understanding that the web UI is a React application served by a Go backend, and that it communicates with backend services via WebSocket RPC.

The Output Knowledge Created

Message 208 itself doesn't create any output—it's a planning step. But it leads to the creation of:

  1. A troubleshooting section in the README that explains how to access individual node web UIs by temporarily modifying the docker-compose.yml or adding port mappings.
  2. A documented rationale for why individual node web UIs are disabled by default (to avoid port confusion and because the cluster monitoring UI provides the primary operational view).
  3. A reusable pattern for future documentation: when a configuration decision might be confusing, explain the reasoning and provide an escape hatch.

The Thinking Process Visible in Reasoning

The assistant's reasoning in message 208 reveals several cognitive patterns:

Pattern 1: Completeness checking. After making the port configuration changes, the assistant doesn't declare victory and move on. It asks itself: "What else is needed?" This is a form of mental checklist—the assistant is verifying that the full set of changes (code changes + documentation changes) is complete.

Pattern 2: User empathy. The assistant puts itself in the position of a future user who encounters the test cluster for the first time. It anticipates the question: "I see the cluster monitoring UI on 9010, but how do I access an individual node's web UI?" This is a sophisticated form of perspective-taking.

Pattern 3: Risk mitigation. The assistant recognizes that undocumented design decisions are a source of future errors. By documenting the port configuration rationale and providing instructions for accessing individual node web UIs, the assistant reduces the risk of someone making incorrect assumptions about the system.

Pattern 4: Iterative refinement. The assistant doesn't try to solve everything in one pass. It makes the code changes, then updates the README, then thinks about what else might be needed, and plans another README update. This iterative approach allows for incremental improvement without analysis paralysis.

The Broader Significance

Message 208, for all its brevity, illustrates a fundamental principle of robust system design: the boundary between the system and its documentation is porous. A configuration that is confusing to operate is a bug, even if the code is technically correct. By treating documentation as a first-class artifact—something that deserves the same careful reasoning as code—the assistant demonstrates a mature approach to software engineering.

This is especially important in distributed systems, where the operational surface area is large. A test cluster with 2 frontend proxies, 2 storage nodes, and a shared database has many moving parts. Each part has its own configuration, its own ports, its own logs, and its own web UI. Without careful documentation, operators are left to guess at the intended interaction patterns. Message 208 is a small but significant step toward making those patterns explicit.

Conclusion

The subject message is a moment of reflection in the midst of active development. The assistant has just resolved a port configuration issue, but instead of moving on to the next feature, it pauses to consider how the resolution will be perceived by future users. This is the mark of a developer who understands that code is read far more often than it is written, and that the quality of a system is measured not just by its correctness but by its comprehensibility.

In adding a troubleshooting section to the README, the assistant is doing something that is easy to overlook but hard to overvalue: it is making the system's design decisions visible and its operation accessible. For anyone who has ever stared at a configuration file wondering "why is it set up this way?" or "how do I override this default?", this kind of proactive documentation is invaluable. Message 208 may be short, but it speaks volumes about the philosophy of building systems that are not just functional, but understandable.