Chunk 3.0

## Summary: Horizontally Scalable S3 Architecture for Filecoin Gateway ### Project Overview We're building a horizontally scalable S3-compatible storage system where multiple Kuri storage nodes operate independently while sharing object routing metadata via YugabyteDB. ### Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ S3 Frontend Proxy (stateless, scalable) │ │ :8078 │ └─────────────────────────┬───────────────────────────────────┘ │ ┌───────────────┼───────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Kuri Node 1 │ │ Kuri Node 2 │ (independent storage) │ RIBS keyspace: │ │ RIBS keyspace: │ │ filecoingw_kuri1│ │ filecoingw_kuri2│ │ :7001 │ │ :7002 │ └────────┬────────┘ └────────┬────────┘ │ │ └─────────┬─────────┘ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Shared S3 Keyspace │ │ filecoingw_s3 │ │ (S3Objects table with node_id for routing) │ └─────────────────────────────────────────────────────────────┘ ``` ### Test Cluster Ports - `:8078` - S3 API (Frontend Proxy) - `:9010` - Web UI for kuri-1 (via nginx) - `:9011` - Web UI for kuri-2 (via nginx) - `:7001` - kuri-1 LocalWeb (CAR files) - `:7002` - kuri-2 LocalWeb (CAR files) ### What Was Completed #### 1. Core S3 Frontend Proxy (`server/s3frontend/`) - Stateless proxy that routes requests to Kuri backends - Round-robin writes, YCQL lookup for reads - Health checks, multipart upload coordination #### 2. Fixed Critical Bugs - **HTTP route conflict**: Go 1.22 ServeMux pattern conflicts between `HEAD /` and `GET /healthz` - fixed with custom handler in `server/s3/fx.go` - **JSON case mismatch**: Frontend expected camelCase, backend sent PascalCase - added JSON tags to all structs in `iface/iface_ribs.go` - **Missing S3 tables**: Added `node_id` column to S3Objects table in db-init #### 3. Cluster Monitoring Implementation **Backend files modified:** - `iface/iface_ribs.go` - Added JSON tags to all cluster monitoring structs (ClusterTopology, ProxyInfo, StorageNodeInfo, ThroughputHistory, IOThroughputHistory, LatencyDistribution, ErrorRates, ActiveRequests, ClusterEvent) - `iface/iface_rbs.go` - Added `IOThroughput()` method to RBSDiag interface - `rbstor/cluster_metrics.go` - Real-time metrics collection (request counts, latency, I/O bytes, events) - `rbstor/diag.go` - ClusterTopology returns live stats (storageUsed, groupsCount, requestsPerSecond) - `integrations/web/rpc.go` - Added IOThroughput RPC method - `server/s3/server.go` - Added `responseRecorder` to track bytes, metrics recording in handlers **Frontend files modified:** - `integrations/web/ribswebapp/src/components/ClusterTopology.js` - Visual distinction between S3 Frontend (blue) and Kuri Storage (green) nodes - `integrations/web/ribswebapp/src/components/IOThroughputChart.js` - NEW: I/O bytes/sec chart - `integrations/web/ribswebapp/src/components/LatencyDistributionChart.js` - Changed SLA→SLO at 350ms - `integrations/web/ribswebapp/src/routes/Cluster.js` - Added IOThroughput chart, improved layout - `integrations/web/ribswebapp/src/routes/Cluster.css` - New 2-column layout with charts grid #### 4. Test Cluster (`test-cluster/`) - `docker-compose.yml` - YugabyteDB + 2 Kuri nodes + S3 proxy + nginx webui - `gen-config.sh` - Generates per-node configs and nginx.conf - Nginx proxies both kuri UIs on :9010 and :9011 ### Current State / What's Happening Now The React frontend was rebuilt with `npm run build` but the Docker image hasn't been rebuilt yet. Need to: 1. **Rebuild Docker image**: `docker build -t fgw:local .` 2. **Restart cluster**: `cd test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose up -d --force-recreate` ### What Needs To Be Done Next 1. Deploy the updated frontend (rebuild docker, restart containers) 2. Verify the new layout appears (topology + active requests side-by-side, 2x2 chart grid) 3. Verify I/O Throughput chart appears and shows data 4. Verify SLO label at 350ms on latency chart 5. Test that storage nodes table shows live stats (storage used, groups, req/sec) ### Key Commands ```bash # Rebuild frontend cd /home/theuser/gw/integrations/web/ribswebapp && npm run build # Rebuild docker image cd /home/theuser/gw && docker build -t fgw:local . # Restart cluster cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose up -d --force-recreate # Test RPC echo '{"jsonrpc":"2.0","method":"RIBS.IOThroughput","params":["5m"],"id":1}' | websocat ws://localhost:9010/rpc/v0 # Generate test traffic for i in {1..10}; do dd if=/dev/urandom bs=10K count=1 2>/dev/null | curl -s -X PUT --data-binary @- http://localhost:8078/test/file$i.bin; done ``` ### Known Issues - `rbstor/rbs.go:41` has LSP error `ribsCfg.NodeID undefined` - this is a pre-existing issue not related to our changes - nginx webui container sometimes needs restart after kuri nodes restart to pick up correct routing

The Optimization Pipeline: From 700 MB/s to 85 GB/s in a Distributed S3 Load Test 2296 words

Message Articles