The Container That Forgot Its Config: A Lesson in Docker State Management
Introduction
In distributed systems debugging, the most elusive bugs are often the simplest in retrospect. Message 746 of this coding session captures one such moment—a single line of realization: "The nginx container needed a restart to pick up the config changes." This brief observation, buried within a verification script, represents the culmination of an extended debugging session spanning multiple layers of a horizontally scalable S3 architecture. The message itself is deceptively short, but it sits at the intersection of JSON serialization mismatches, container lifecycle management, and the subtle ways that distributed systems can appear broken when they are actually working correctly.
The Message
The assistant wrote:
Now it shows kuri-1. The nginx container needed a restart to pick up the config changes.
>
Let me verify everything is working: ``bash echo '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' | websocat ws://localhost:9010/rpc/v0 2>&1 | jq '.' { "id": 1, "jsonrpc": "2.0", "result": { "proxies": [ { "id": "kuri-1", "address": "localhost", "status": "healthy", "requestsPerSecond": 0, "activeConnections": 0, "backendPool": null, "latencyMs": 0, "errorRate": 0 } ], "storageNodes": [ { "id": "kuri-1", "address": "http://kuri-1:8078", "status": "healthy", "storageUsed": 0, "storageTotal": 0, "ob... ``
The Context: A Frontend That Showed Nothing
To understand why this message matters, we must trace the debugging chain that preceded it. The user had reported a frustrating problem: the cluster monitoring frontend was empty. The topology display showed nothing beyond a "load balancer" label at the top. Charts displayed "No data available." Events showed "Invalid Date." Yet the backend RPC calls were returning perfectly valid data—throughput metrics, latency distributions, error rates, and cluster topology information were all flowing correctly from the Kuri storage nodes.
This is a classic "it works on my machine" problem, but inverted: the backend worked, the frontend didn't, and the disconnect was invisible to both sides. The assistant's investigation revealed the root cause: a mismatch in JSON field naming conventions. Go's standard encoding/json library serializes struct fields using their exact Go names—PascalCase by convention (e.g., Timestamps, Total, Reads). The React frontend, however, expected camelCase field names (timestamps, total, reads). This mismatch caused every data structure flowing from backend to frontend to be silently ignored. JavaScript's destructuring and property access simply returned undefined for the expected fields, and the React components, checking for truthiness, concluded that no data existed.
The Fix That Almost Worked
The assistant corrected this by adding json struct tags to every relevant type in iface/iface_ribs.go, forcing camelCase serialization. This required updating approximately a dozen struct types including ClusterTopology, ProxyInfo, StorageNodeInfo, ThroughputHistory, LatencyDistribution, ClusterEvent, and several others. The change cascaded into rbstor/cluster_metrics.go, where anonymous struct types used in map literals needed to match the new tagged types exactly—a type compatibility issue that caused a build failure and required additional fixes.
After rebuilding the Go binary, rebuilding the Docker image, and redeploying the containers, the JSON output changed from PascalCase to camelCase. The RPC calls now returned timestamps instead of Timestamps, total instead of Total, reads instead of Reads. The frontend could finally parse the data. But a new problem emerged: the cluster topology was showing the wrong proxy node.
The Wrong Node: A Deeper Investigation
When the assistant queried RIBS.ClusterTopology on port 9010 (the web UI for kuri-1), the response showed kuri-2 as the proxy node. This was deeply confusing. The environment variables on kuri-1 were correct: FGW_NODE_ID=kuri-1, FGW_BACKEND_NODES listed both nodes properly. The code in rbstor/diag.go appeared to correctly read FGW_NODE_ID and return the appropriate node identity. So why was kuri-1 reporting itself as kuri-2?
The assistant pursued several hypotheses. Perhaps the request was being routed through the nginx reverse proxy to the wrong backend. Perhaps the code had a bug in how it parsed the backend nodes configuration. Perhaps there was a race condition in the container startup. Each of these was investigated in turn. The environment was checked via docker exec. The code was re-read. The nginx configuration was examined. Everything looked correct.
The Real Culprit: Container Configuration Immutability
The answer, when it came, was almost anticlimactic: the nginx container had not been restarted after the configuration file was updated. Docker containers are immutable by design—they run with the filesystem and configuration they had at startup. Changes to configuration files on the host's bind-mounted volumes are visible inside the container at the filesystem level, but the running process (nginx) does not automatically reload its configuration unless explicitly told to do so. In this case, the nginx configuration had been modified during the debugging session, but the container continued running with the old configuration loaded into memory.
This is a fundamental principle of containerized deployments that is easy to forget in the heat of debugging. When you docker compose up -d, the container starts with whatever configuration exists at that moment. If you later edit the configuration file, the container is blissfully unaware. You must either restart the container (docker compose restart or docker restart) or send a reload signal to the process inside (e.g., nginx -s reload). The assistant had done neither.
After restarting the web UI container with docker restart test-cluster-webui-1, the topology query immediately returned the correct node: kuri-1. The problem had never been a code bug. It had been a deployment workflow issue—a failure to account for the fact that configuration changes require explicit action to take effect in a running container.
The Assumptions and Their Consequences
This debugging episode reveals several assumptions that were made, some explicitly and some implicitly:
Assumption 1: Configuration changes are automatically picked up. This is perhaps the most common Docker pitfall. Developers accustomed to hot-reloading development servers or live-editing configuration files on bare-metal deployments often forget that containers freeze their state at startup. The bind mount provides file visibility, not process awareness.
Assumption 2: The code was the problem. The assistant spent considerable effort investigating the Go backend code, the JSON serialization, and the topology logic. While the JSON field naming fix was legitimate and necessary, the "wrong node" symptom was entirely unrelated to code. This is a classic debugging trap: when a system behaves unexpectedly, the natural instinct is to look for bugs in the code, when the real issue may be in the deployment or configuration management.
Assumption 3: Container state is transparent. The assistant used docker exec to inspect environment variables inside the container, confirming that FGW_NODE_ID=kuri-1. This inspection showed the current state of the container's filesystem and environment, but it did not reveal what configuration the nginx process had loaded at startup. There is no easy way to ask a running nginx process "what configuration file did you load?"—you must infer it from behavior.
Assumption 4: The nginx configuration was correct. The configuration file was indeed correct. But "correct" at the file level does not mean "active" at the process level. The assistant read the configuration file from the host filesystem (/data/fgw2/config/nginx.conf), which reflected the latest edits, but the container's nginx process was running an older version loaded before those edits were made.
Input Knowledge Required
To fully understand this message and the debugging session it concludes, several domains of knowledge are necessary:
Go serialization semantics: Understanding that Go's encoding/json uses exact struct field names unless json tags are specified is critical. The default behavior is PascalCase because Go convention dictates exported struct fields start with an uppercase letter. This is a common source of friction between Go backends and JavaScript frontends.
Docker container lifecycle: Containers are immutable processes. Changes to bind-mounted files are visible inside the container but do not automatically trigger process reloads. This distinction between "file visible" and "configuration active" is subtle but essential.
React component rendering patterns: The frontend components used a defensive pattern: if (!data || !data.timestamps) returns early with a "No data available" message. This pattern silently hides data when field names don't match, making debugging more difficult because there is no error—just absence.
WebSocket RPC architecture: The monitoring system uses a JSON-RPC over WebSocket protocol. The websocat tool was used to manually query RPC endpoints for debugging. Understanding this architecture is necessary to interpret the verification commands.
Nginx reverse proxy configuration: The web UI container runs nginx as a reverse proxy, forwarding requests to the appropriate Kuri node. The configuration determines which backend serves which port, and stale configurations cause requests to be routed incorrectly.
Output Knowledge Created
This message and the surrounding debugging session produced several valuable outputs:
A corrected deployment workflow: The lesson that containers must be restarted after configuration changes is now explicitly documented in the debugging history. Future debugging sessions can reference this experience.
A validated monitoring pipeline: After the nginx restart, the entire data flow from backend metrics collection through JSON-RPC serialization to React rendering was confirmed working. The topology now correctly identifies nodes, and the charts display real data.
A debugging methodology: The session demonstrates a systematic approach: verify backend data independently (via websocat), check frontend expectations (by reading React component code), fix serialization mismatches, rebuild and redeploy, and finally verify end-to-end. When unexpected behavior persists, check deployment state before assuming code bugs.
A cautionary tale about container assumptions: The most valuable output may be the meta-lesson: when debugging distributed systems, question every assumption about what state each component is actually in. A container's environment variables may be correct, its configuration file may be correct, and its code may be correct—but if the process hasn't reloaded, none of that matters.