From Bare Metal to Balanced Traffic: The Complete QA Cluster Deployment of a Distributed Storage System

Introduction

In the life of any distributed systems engineer, there are sessions that feel like a single, continuous debugging loop—a stretch of hours where every success is immediately followed by a new failure, and every fix reveals a deeper architectural assumption that had gone unexamined. This article examines one such session: the end-to-end deployment and validation of a QA test cluster for the Filecoin Gateway (FGW) distributed storage system across three physical nodes. Over the course of dozens of messages spanning multiple hours, the assistant and user navigated a cascade of interconnected problems—database installation failures, broken CLI tools, security corrections, dirty migration states, empty cluster topology APIs, missing environment variables, cross-node read failures, a missing proxy layer, a question about infrastructure automation, and even a silent codebase regression.

What makes this effort particularly instructive is not just the technical challenges overcome, but the methodological discipline that guided every step. The assistant researched before architecting, verified before assuming, and corrected course immediately when security gaps were identified. The user contributed a series of concise, high-impact observations that repeatedly realigned the trajectory of the work. Together, they transformed three bare-metal servers into a functioning distributed storage cluster with cross-node S3 access, secured credentials, and validated load distribution.

This article synthesizes the complete arc of that deployment—from the initial research phase through the final load test verification—drawing on the detailed message-level analyses captured in the companion articles. It is a story about the nature of distributed systems engineering: where the difference between a working cluster and a broken one is often a single boolean flag, a missing environment variable, or a question asked at the right moment.

Phase 1: Research Before Architecture

The deployment began not with commands but with questions. When the user asked for a QA environment specification, the assistant responded by delegating four parallel research tasks covering FGW deployment requirements, YugabyteDB minimum specifications, Filecoin mainnet connectivity, and MinIO S3 requirements. This research-before-architecture principle was a deliberate choice, explicitly instructed by the user: "Delegate agents to research anything you aren't 100% sure about."

The research yielded critical insights that shaped the entire deployment. The most important discovery was that FGW does not require a full Filecoin Lotus node—it connects to an external RPC endpoint (api.chain.love), eliminating the need for blockchain syncing and its associated hardware demands. YugabyteDB could run as a single-node yugabyted instance for QA purposes, requiring only 2 CPU cores and 2 GB RAM minimum. The architecture plan that emerged specified a three-node topology: two storage nodes with 1.25 TiB each running the Kuri daemon and S3 proxy, and one infrastructure node with 200 GiB hosting YugabyteDB and optional MinIO.

This phase established a pattern that would recur throughout the deployment: gather facts before making commitments, verify assumptions before executing commands, and document the reasoning so that future operators can understand why decisions were made. The architecture plan was not a static document—it evolved as the deployment uncovered new constraints, but its foundation was built on verified facts rather than assumptions.

Phase 2: From Blueprint to Bare Metal

The transition from architecture plan to physical deployment began with a simple connectivity check. The assistant SSHed into each of the three nodes, verifying hostnames, kernel versions, and OS releases. The machines revealed themselves as fgw-qa-head, fgw-ribs1, and fgw-ribs2—all running Ubuntu 24.04.3 LTS with identical kernel builds. This uniformity was a stroke of luck; identical operating systems meant that Ansible roles and package versions would work across all nodes without OS-specific adjustments.

The assistant then methodically assessed hardware specifications, checked for existing Docker installations, verified sudo access, and examined the production Ansible inventory for reusable patterns. This assessment phase was not merely preparatory—it was a form of due diligence that prevented countless downstream failures. By understanding exactly what each node offered (CPU cores, RAM, disk layout) and what it lacked (Docker, YugabyteDB, the fgw user), the assistant could make informed decisions about where to install each component and what prerequisites needed to be fulfilled.

The Ansible inventory structure was a critical artifact. The assistant created a dedicated qa inventory directory under ansible/inventories/, with group variables for all, kuri_node, and s3_frontend. Each node was assigned to its appropriate groups, and the inventory was designed to be reusable for future QA deployments. This infrastructure-as-code approach ensured that the deployment could be reproduced, audited, and modified without manual intervention.## Phase 3: The Database Foundation

Installing YugabyteDB on the head node appeared straightforward—a curl download, a tar extraction, a directory creation—but it immediately revealed a subtle permissions issue. The tarball was extracted with sudo, leaving files owned by root, but the yugabyted process would run as the theuser user. The resulting startup failure required a chown correction, a small debugging detour that nonetheless illustrated a recurring theme: the gap between installation ownership and runtime ownership is a common source of infrastructure failures.

The real challenge, however, lay in creating the CQL keyspaces. YugabyteDB provides both a PostgreSQL-compatible SQL interface (YSQL) and a Cassandra-compatible CQL interface (YCQL). The SQL databases were created without incident using ysqlsh. But the CQL keyspaces—filecoingw_kuri_01, filecoingw_kuri_02, and filecoingw_s3—proved stubbornly elusive. The bundled ycqlsh tool crashed with a cryptic ModuleNotFoundError: No module named 'six.moves'.

This error triggered an extensive debugging spiral that consumed nearly thirty messages. The assistant checked Python paths, installed python3-six, created a python symlink, tried setting CQLSH_PYTHON and PYTHONPATH, and even examined the ycqlsh.py source code directly. Nothing worked. The root cause was a fragile dependency chain in the ycqlsh wrapper script: it discovered the Python interpreter through a shebang mechanism that failed to find a version with the six.moves module available.

The breakthrough came when the assistant abandoned the CLI tool entirely and used the Python Cassandra driver directly via an inline script executed over SSH. The first attempt failed due to quote escaping—the CQL replication parameters required single quotes, but the shell quoting layers were converting them to double quotes. After correcting the escaping strategy, the keyspaces were created successfully. This workaround, bypassing a broken tool in favor of a direct API call, is a textbook example of pragmatic infrastructure debugging: when a tool is broken, do not fix the tool—work around it.

Phase 4: Building and Deploying Binaries

With the database foundation in place, the assistant turned to building the FGW software stack. The project's Makefile revealed three build targets: kuri (the storage node daemon, ~175 MB), gwcfg (a configuration utility, ~67 MB), and s3-proxy (the S3 frontend proxy, ~39 MB). A clean build succeeded without errors, producing binaries that were then copied to both storage nodes via scp.

The binary deployment step, however, contained a subtle architectural misunderstanding. The assistant copied both kuri and s3-proxy to each storage node, implicitly designing a topology where every storage node was also an S3 endpoint. This contradicted the project roadmap's requirement for separate stateless frontend proxy nodes. The mistake was understandable—the earlier Docker-based test harness had co-located all components on a single machine—but it would later manifest as cross-node S3 read failures. Each node could only serve data from its local blockstore, not from the peer node. The eventual solution was to deploy the s3-proxy as a separate layer on the head node, routing requests to the correct backend based on object metadata in the shared S3 keyspace.

Phase 5: The Security Correction

Perhaps the most consequential moment in the deployment was not a technical breakthrough but a security correction. When the user provided the CIDgravity API token and wallet files, the assistant initially embedded the token directly into the settings.env configuration files in plaintext. The user's response was immediate and unambiguous: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS."

The assistant's response demonstrated a crucial capability: the ability to recognize a mistake, apologize, and redesign the approach from first principles. The corrected architecture separated configuration from secrets entirely. The settings.env files contained all non-sensitive parameters (node IDs, database hosts, API endpoints) but explicitly omitted the token, with a comment noting where it would be loaded from. The token was stored in a separate file at /home/fgw/.ribswallet/cidg.token with permissions 600, owned by the fgw user.

The systemd service files completed this architecture with a sophisticated runtime secret injection mechanism. An ExecStartPre hook read the token from the restricted file and wrote it to a runtime environment file in /run/fgw/token.env—a memory-backed tmpfs that never touched persistent storage. The service then loaded this file via EnvironmentFile=-/run/fgw/token.env, with the leading dash ensuring graceful degradation if the file did not exist. Security hardening directives (NoNewPrivileges=true, ProtectSystem=strict, ProtectHome=read-only) transformed the service from a simple daemon launcher into a sandboxed execution environment.

This correction elevated the deployment from hobbyist-grade to enterprise-grade. It demonstrated that security is not a feature to be added after the fact but a constraint that must be baked into the architecture from the start. The separation of configuration from secrets, the use of restricted-permission vault files, and the runtime-only token loading pattern became the standard for the entire QA deployment.## Phase 6: The Dirty Migration That Blocked Service Startup

With the binaries deployed, systemd services configured, and database keyspaces created, the moment arrived to start the kuri daemons. The assistant ran sudo systemctl start kuri on both storage nodes, expecting clean startup logs and active services. Instead, both nodes reported failure with exit code 1.

The failure was traced to a condition known as a "dirty migration" in the YugabyteDB CQL keyspaces. The schema_migrations table—a standard pattern used by database migration frameworks to track which schema versions have been applied—contained a row with dirty = true for migration version 1769890615. This flag signals that a migration was interrupted or failed partway through, and most migration-aware applications refuse to proceed until the flag is cleared, to prevent inconsistent schema states.

The root cause was that the test suite had left these dirty flags behind. This is a classic artifact of development and testing workflows: test harnesses often perform schema migrations as part of setup and teardown, and if a test is interrupted or crashes, it can leave the migration state in an ambiguous condition. The dirty flag is a safety mechanism, but in this case it became a deployment blocker.

The assistant diagnosed this by checking the logs on kuri1 (journalctl -u kuri), which showed the daemon initializing but failing to proceed past schema validation. Then, using a Python script with the Cassandra driver, the assistant queried the schema_migrations table directly and confirmed the dirty flag. The fix was straightforward: update the flag to false for all affected keyspaces—filecoingw_kuri_01, filecoingw_kuri_02, and the shared S3 keyspace filecoingw_s3.

A notable mistake during this process was the initial failure to check all three keyspaces. The assistant first fixed only filecoingw_kuri_01 and filecoingw_kuri_02, restarted the service, and it still failed. Only then did the assistant check filecoingw_s3 and find the dirty flag there too. This is a classic debugging oversight—fixing the most obvious problem without considering that the same condition might exist in related components.

Once all three keyspaces were corrected, both kuri nodes came online successfully. Kuri1 ran with 70.3 MB of memory consumed, 35 tasks, and a clean Active: active (running) status. Kuri2 followed suit with the same clean startup. The cluster had its storage layer operational.

Phase 7: The Topology That Wouldn't Render

With both storage nodes running, the next task was to verify the cluster topology. The kuri daemon exposes a Web UI on port 9010 that includes a cluster topology visualization—a React-based dashboard showing the distributed storage cluster's nodes, their health status, and data flow relationships. But when the user checked this interface, the topology was blank.

The user's bug report was a model of brevity: ":9010 cluster topology doesn't render." Seven words, a port number, and a symptom. No greeting, no stack trace, no screenshot. Yet this single sentence set the agenda for the next dozen assistant actions and exposed a critical configuration gap.

The assistant traced the empty topology to the source code. Reading rbstor/diag.go, the assistant found the ClusterTopology() function and its dependency on FGW_BACKEND_NODES—an environment variable that lists the backend storage nodes with their IDs and URLs. This was the root cause: the environment variable was not set during the initial deployment.

The assistant configured the variable on both nodes, adding FGW_BACKEND_NODES="kuri_01:http://10.1.232.83:8079,kuri_02:http://10.1.232.84:8079" to the settings files and restarting the kuri services. The subsequent API call returned a full topology with both nodes showing as healthy. The topology rendered, but this fix was cosmetic—it made the Web UI display correctly without actually enabling cross-node data access. The real architectural problem was still lurking.## Phase 8: The Cross-Node Read Problem

The topology fix made the cluster look correct, but the user's next observation revealed the deeper issue. After running a load test against kuri1's direct S3 endpoint, the user noted: "Only kuri1 getting traffic seems wrong." This observation was a masterclass in concise, high-impact feedback. It was not a complaint about the load test tool or a request for a configuration tweak. It was an architectural observation that revealed a fundamental gap between the deployed system and the intended design.

The assistant investigated and discovered the root cause: when kuri2 received a GET request for an object written to kuri1, it found the object metadata in the shared filecoingw_s3 keyspace (which recorded node_id='kuri_01'), but it could not retrieve the actual block data. The error message was telling: "block was not found locally (offline): ipld: could not find bafkreicq..." The S3 server on kuri2 was looking for the block in its local blockstore, not asking kuri1 for it.

The assistant attempted a workaround—connecting the two kuri nodes via IPFS swarm to enable bitswap-based block exchange. This succeeded at the network level but did not fix the S3 reads because the S3 layer does not use bitswap for retrieval; it uses a direct local blockstore lookup. The architecture simply did not support cross-node reads at the S3 level.

This was the moment of architectural clarity. The system was designed with a three-layer hierarchy: stateless S3 frontend proxies → kuri storage nodes → YugabyteDB. The kuri nodes were never meant to serve S3 requests directly in a multi-node configuration. The proxy layer was the missing piece.

Phase 9: The Silent Regression

Amidst all the deployment and debugging work, a different kind of crisis emerged. The user noticed something alarming in the git working tree: code related to garbage collection and caching was being removed. The user asked: "Why is some gc/cache code removed in git diff??"

The assistant investigated and discovered that three files had been modified in the working directory, with 397 lines deleted. The assistant had inadvertently removed the GC configuration struct and cache integration code from configuration/config.go, stripped the multi-tier cache import and usage from rbdeal/retr_provider.go, and gutted the GC integration from rbdeal/claim_extender.go. These were not cosmetic changes—they removed the core integration points that made the caching and garbage collection systems functional.

How did this happen? The most likely scenario is that the assistant, while working on the QA deployment configuration, made changes to these files as part of debugging or testing and inadvertently saved modifications. The configuration/config.go file, in particular, contains the RibsConfig struct that controls all system settings—it would be a natural file to touch when adjusting environment variables or deployment parameters. A stray keystroke, an accidental save, or a misapplied search-and-replace could have introduced these deletions without the assistant realizing it.

This is a classic version control hazard. When working under pressure—deploying to physical nodes, debugging service failures, managing SSH sessions across three machines—it is easy to make unintended modifications to source files. The assistant was operating in a mode of rapid iteration, running commands, checking logs, and fixing issues as they appeared. The cognitive load of managing the deployment likely overshadowed awareness of the working tree state.

The assistant restored the files with git checkout -- configuration/config.go rbdeal/claim_extender.go rbdeal/retr_provider.go and verified the working tree was clean. The code was back to the properly committed state, but the incident served as a powerful reminder that working tree integrity needs active monitoring during long sessions.## Phase 10: Deploying the S3 Proxy—From Manual SSH to Ansible Automation

The correct solution to the cross-node read problem was to deploy the s3-proxy binary on the head node (10.1.232.82), configured with the list of backend kuri nodes and access to the shared S3 CQL keyspace. The proxy would receive all S3 requests, look up the object's node_id in the shared metadata, and route the request to the correct backend node.

What followed was a textbook example of ad-hoc infrastructure deployment. Over the course of several messages, the assistant copied the binary using scp, created configuration files by echoing heredocs through SSH, wrote a systemd service file using sudo tee, and started the service—which immediately failed because the environment variable names in the config file didn't match what the binary expected.

The assistant then discovered, through grep, that the s3-proxy binary expected FGW_YCQL_HOSTS rather than RIBS_S3_CQL_HOSTS. The manual approach had introduced a naming mismatch that would have been caught by the existing Ansible templating system.

At this point, the user interjected with a question that changed the trajectory of the entire deployment: "Why are you not doing this with ansible?" These eight words were a masterclass in effective engineering communication. The question was not a criticism or a technical correction—it was a pure, almost Socratic inquiry that forced a reckoning with a specific kind of engineering failure: the failure to use existing abstractions.

The assistant's response was immediate and unambiguous: "You're right—I should be using ansible properly instead of manually SSHing and creating ad-hoc configs." The assistant stopped the manual work, read the existing s3_frontend role, examined the Jinja2 templates, and updated the QA inventory to add the head node to the s3_frontend group. The playbook was run, and the s3-proxy was deployed correctly, with the right environment variable names, proper systemd integration, and the full audit trail that infrastructure-as-code provides.

This pivot was critical. The Ansible roles for the s3_frontend already existed—complete with Jinja2 templates for configuration files, systemd service definitions, and proper variable management. The QA inventory file already had the host groups defined. Everything needed to deploy the s3-proxy through automation was already in place. The assistant had fallen into a common trap: the belief that a "quick" manual deployment would be faster than updating the inventory and running a playbook. But the manual approach had already consumed multiple message turns and produced a broken service.

Phase 11: The Moment of Proof

With the s3-proxy deployed via Ansible, the assistant ran a series of curl commands to validate cross-node access. The test script wrote a new object through the proxy, read it back, and then read an older object that had been written directly to kuri1 before the proxy existed. All three operations succeeded.

The most important test was the third: reading test.txt (written directly to kuri1) through the proxy at http://10.1.232.82:8078. The proxy returned test-data-123—the correct content. This confirmed that the proxy could discover and retrieve objects regardless of their origin, using the shared metadata in the CQL keyspace.

The assistant then escalated to a load test, running ritool loadtest with 10 concurrent workers, 30-second duration, and object sizes between 1KB and 100KB against the proxy endpoint. The test completed successfully with approximately 345 writes per second and 147 reads per second, with zero data corruption—all Read-After-Write verifications passed.

The final verification was to confirm that both backend nodes actually received traffic. The assistant SSHed into each node and ran journalctl -u kuri --since '2 min ago' | grep -c 'PUT\|DELETE'. Both nodes returned exactly 3 operations each. This was the empirical confirmation that the proxy was distributing traffic correctly across the multi-node cluster. The architecture was working as designed.

Phase 12: The Verification That Failed—And What It Taught

After the load test confirmed both nodes were receiving traffic, the assistant attempted to verify object distribution by querying the shared CQL keyspace directly. The query was straightforward: SELECT node_id, count(*) as cnt FROM s3objects GROUP BY node_id ALLOW FILTERING. But it failed with a SyntaxException because YugabyteDB's YCQL does not support GROUP BY.

This failure was far more revealing than any success could have been. It highlighted a fundamental tension in YugabyteDB's design: the YCQL compatibility layer implements the Cassandra wire protocol and query language, but some Cassandra features—like GROUP BY—are not supported by default. The error message itself was informative: "Feature Not Supported. This is not recommended but this error can be suppressed by setting ycql_suppress_group_by_error flag to false."

The assistant's assumption that CQL would support standard aggregation syntax was reasonable—that is the entire point of a compatibility layer. But compatibility layers are never perfect, and GROUP BY is a complex analytical operation that different databases implement differently. The failure was not in the storage system itself—the s3-proxy was routing correctly, both nodes were storing data, and the load test passed. The failure was in the verification of the system, which is a fundamentally different problem from building it.

This episode underscores a critical lesson for distributed systems engineering: the tools used to verify a system have their own failure modes, and an engineer who can build a working distributed system may still struggle to prove that it is working correctly. Verification is its own engineering challenge, requiring its own set of skills and knowledge about the database layer's quirks and limitations.## Themes and Lessons

Several themes emerge from this deployment effort that are broadly applicable to distributed systems engineering.

The hidden cost of test residue. The dirty migration flags were left behind by a test suite that had run earlier. This is a classic "test residue" problem—the test environment's database state does not get fully cleaned up, and when you try to deploy for real, the leftover state blocks you. The lesson is that test harnesses must be designed to clean up after themselves, or deployment pipelines must account for the possibility of stale state.

The gap between cosmetic and functional fixes. The FGW_BACKEND_NODES environment variable made the cluster topology render correctly, but it did not enable cross-node data access. The topology fix was cosmetic—it made the system look correct without making it work correctly. The real fix required the s3-proxy, which was a fundamentally different architectural component. This is a common pattern in distributed systems: making the monitoring work is not the same as making the system work.

The architecture that was already designed. The three-layer hierarchy (proxy → kuri → database) was specified in the project roadmap, but the assistant initially deployed kuri nodes as direct S3 endpoints. The cross-node read failures were not bugs in the code—they were consequences of deploying the system in a configuration that bypassed its own architecture. The lesson is that architecture documents are not optional reading; they encode design decisions that have real operational consequences.

The power of a well-timed question. The user's question "Why are you not doing this with ansible?" was more valuable than any technical fix. It realigned the deployment methodology, ensured consistency with the rest of the infrastructure, and reinforced the discipline of infrastructure-as-code. The question worked because the foundation was already laid: the Ansible roles existed, the inventory was structured, and the team had already committed to automation. All that was needed was someone to ask the obvious question at the right moment.

The fragility of long coding sessions. The accidental code deletion incident demonstrated that even experienced engineers can make silent, destructive mistakes when operating under cognitive load. The working tree is a shared resource, and its integrity depends on active monitoring. The user's attentive observation prevented a regression that would have removed critical garbage collection and caching functionality from the codebase.

Security as a first-class constraint. The vault principle—secrets must never be stored in configuration files—was enforced by the user and internalized by the assistant. The resulting architecture of runtime secret injection via systemd ExecStartPre is a pattern worth replicating in any deployment. The separation of configuration from secrets, the use of restricted-permission vault files, and the runtime-only token loading pattern transformed the deployment from hobbyist-grade to enterprise-grade.

Pragmatic workarounds over tool fixation. When the ycqlsh tool proved unfixable, the assistant bypassed it entirely and used the Cassandra driver directly. This willingness to abandon broken tools in favor of direct approaches is a hallmark of effective infrastructure engineering.

Verification is its own engineering challenge. The failed CQL GROUP BY query showed that verifying a distributed system requires its own set of skills and knowledge. The database layer has its own quirks and limitations, and the tools used for verification (CQL queries, log greps, API calls) each have their own failure modes.

Conclusion

The QA cluster deployment examined in this article is a microcosm of the challenges inherent in distributed systems engineering. It began with research and architecture planning, proceeded through database installation and binary deployment, encountered a security correction that reshaped the configuration architecture, hit a dirty migration flag that blocked service startup, navigated a missing environment variable that broke the topology visualization, survived an accidental code deletion that threatened milestone integrity, encountered a fundamental architectural gap that required a proxy layer, pivoted from manual SSH to Ansible automation, and culminated in a successful load test that confirmed both backend nodes were receiving traffic.

Each problem was solvable in isolation, but together they formed a chain of dependencies that demanded sustained attention, systematic debugging, and a willingness to question assumptions. The dirty migration fix required understanding the CQL schema migration system. The topology fix required reading source code to find the environment variable dependency. The cross-node read fix required recognizing that the architecture demanded a proxy layer. The Ansible pivot required acknowledging that manual deployment was the wrong approach. The code restoration required noticing that the working tree had been silently corrupted.

In the end, the cluster worked. Both kuri nodes ran, the s3-proxy routed requests correctly, objects were readable across nodes, and the load test showed balanced traffic. But the journey to that working state was not a straight line—it was a spiral of discovery, where each fix revealed a deeper layer of the system's architecture and each failure taught something about how the system was designed to operate. That is the nature of distributed systems engineering: the work is never just about making things work, but about understanding why they work, and why they sometimes do not.

The three-node QA cluster that emerged from this process was not just a test environment—it was a testament to the power of systematic, security-conscious, research-driven infrastructure engineering. And the lessons learned along the way—about test residue, cosmetic fixes, architectural assumptions, the power of good questions, the fragility of long sessions, and the challenge of verification—are lessons that apply to any distributed systems deployment, regardless of the specific technology stack.## References

[1] "From Research to Reality: Deploying a Distributed Storage QA Cluster Across Three Physical Nodes" — Companion article analyzing the initial deployment phases including research, architecture, database installation, binary deployment, security correction, and initialization.

[2] "From Dirty Migrations to Distributed Traffic: The Complete Arc of a QA Cluster Deployment" — Companion article analyzing the debugging and validation phases including dirty migration resolution, topology fixes, cross-node read architecture, the Ansible pivot, code restoration, and load test verification.

[3] "The Research-Before-Architecture Principle" — Analysis of the assistant's decision to delegate parallel research tasks before committing to an architecture design.

[4] "The Vault Principle: How a Single Security Correction Reshaped Configuration Architecture" — Examination of the runtime secret injection pattern using systemd ExecStartPre and restricted-permission vault files.

[5] "The Dirty Migration: When the Database Refuses to Let Go" — Analysis of the schema_migrations dirty flag issue that blocked kuri service startup.

[6] "The Seven-Word Bug Report That Uncovered a Missing Configuration" — Analysis of the user's concise bug report about the empty cluster topology.

[7] "The Cross-Node Read Problem: Diagnosing a Missing Routing Layer" — Examination of why kuri2 could not serve objects written to kuri1 without a proxy layer.

[8] "The Six-Word Question That Saved a Codebase" — Analysis of the user's discovery of uncommitted code deletions in the working tree.

[9] "Why Are You Not Doing This With Ansible?" — Examination of the pivotal user question that triggered the pivot to infrastructure automation.

[10] "The Moment of Proof: Validating Cross-Node S3 Access Through a Proxy Layer" — Analysis of the curl-based validation and load test that confirmed correct proxy routing.