The Quiet Foundation: Creating the MultipartUploads Table in a Distributed S3 Architecture

The Message

[assistant] [bash] docker exec test-cluster-yugabyte-1 bin/ycqlsh yugabyte -e "CREATE TABLE IF NOT EXISTS filecoingw_s3.MultipartUploads (
  bucket text,
  key text,
  upload_id text,
  node_id text,
  created_at timestamp,
  PRIMARY KEY ((bucket, key), upload_id)
);" 2>&1

At first glance, this is a single database command—a CQL CREATE TABLE statement executed against a YugabyteDB container. But in the context of the broader debugging session, this line represents a critical piece of infrastructure being deliberately assembled. It is not a random act of schema creation; it is the culmination of a chain of diagnostic reasoning, a response to a specific failure mode, and a forward-looking architectural decision. To understand why this message matters, one must trace the path that led to it.

The Context: A Cluster in Recovery

The assistant had been building and debugging a horizontally scalable S3-compatible storage system based on a three-layer architecture: stateless S3 frontend proxies routing requests to Kuri storage nodes, which in turn store data and metadata in a shared YugabyteDB cluster. The session immediately preceding this message was a firefight. Multiple things were broken simultaneously: both Kuri nodes were crashing with a Go 1.22 HTTP route conflict between HEAD / and GET /healthz; the web UI container was a placeholder that didn't actually proxy to any backend; and the S3 proxy was returning Internal Server Error because a critical column—node_id—was missing from the S3Objects table in YugabyteDB.

The assistant had methodically worked through each issue. The route conflict was resolved by replacing Go's standard ServeMux with a custom handler that manually dispatched requests based on path and method. The web UI was fixed by replacing the placeholder with an Nginx reverse proxy configuration. The missing node_id column was addressed by dropping the old S3Objects table (which had been created without the column by an earlier migration) and recreating it with the correct schema. Message 609 shows that recreation. Message 610—the subject of this article—creates the companion MultipartUploads table.

Why This Table Exists

The MultipartUploads table is not an afterthought. In the S3 protocol, multipart uploads are a first-class feature that allows clients to upload large objects in parts, with the parts being assembled into a final object upon completion. This is essential for large file transfers, parallel uploads, and resumable uploads. Any S3-compatible storage system that claims to support the S3 API must handle multipart uploads.

But why was this table being created now, in the middle of debugging a broken cluster? The answer lies in the dependency chain. The S3 frontend proxy, when it receives a request, needs to look up object metadata to determine which Kuri storage node holds the data. It does this by querying the S3Objects table, which maps (bucket, key) to a node_id. However, multipart uploads introduce a complication: before an object is fully assembled, there is no entry in S3Objects. Instead, the in-progress upload is tracked in MultipartUploads, which maps (bucket, key, upload_id) to a node_id. When the client completes the upload, the proxy assembles the parts and creates the final entry in S3Objects.

Without the MultipartUploads table, the proxy would have no way to route part upload requests to the correct storage node. A client initiating a multipart upload would receive an UploadId, and subsequent UploadPart requests would need to be routed to the same node that holds the other parts. The node_id column in MultipartUploads enables this routing. So the table is not optional—it is a structural requirement for the architecture to function.

The Schema Design Decisions

The schema chosen for MultipartUploads reveals several design decisions worth examining.

Composite primary key with compound partition key: PRIMARY KEY ((bucket, key), upload_id). The double parentheses around (bucket, key) indicate that these two columns together form the partition key. This is a deliberate choice for data distribution in YugabyteDB (which is compatible with Apache Cassandra's CQL). By using both bucket and key as the partition key, all uploads for the same object (across different upload attempts) are colocated on the same node. This is efficient because when the upload is completed, the proxy needs to assemble parts and create the final S3Objects entry, which is also partitioned by (bucket, key). Colocation minimizes cross-node coordination.

Upload ID as a clustering column: Within a partition, multiple upload attempts for the same object are ordered by upload_id. This allows efficient queries like "list all in-progress uploads for this object" or "find the specific upload by its ID." The upload_id is typically a UUID generated by the server, ensuring uniqueness across the cluster.

The node_id column: This is the linchpin of the routing architecture. It records which Kuri storage node "owns" this multipart upload. When the proxy receives a part upload request, it reads node_id from this table and forwards the request to the correct backend. Without this column, the proxy would have no way to know where to send the data.

The created_at timestamp: This enables lifecycle management. In-progress uploads that are abandoned by the client need to be cleaned up after a timeout. The created_at timestamp allows the system to identify and abort stale uploads.

Assumptions Embedded in the Command

Every database schema carries assumptions about the workload and the data. This one is no exception.

Assumption 1: The keyspace already exists. The command references filecoingw_s3.MultipartUploads, assuming that the keyspace filecoingw_s3 was already created by the db-init container. This is a safe assumption given that the S3Objects table (in message 609) was created in the same keyspace moments earlier.

Assumption 2: The table does not already exist. The IF NOT EXISTS clause handles the case where the table might have been created by a previous run. This is a defensive measure, appropriate for a test cluster that may be restarted multiple times.

Assumption 3: The schema is correct and complete. The assistant did not verify the schema against a migration file or a specification before executing it. The schema was written from memory or from prior knowledge of the codebase. This is a reasonable risk in a debugging session where speed matters, but it carries the possibility of a mismatch between the schema expected by the application code and the schema actually created.

Assumption 4: YugabyteDB's CQL implementation supports this syntax. YugabyteDB is compatible with Cassandra's CQL, but there are edge cases. The compound primary key syntax ((bucket, key), upload_id) is standard CQL and should work, but the assistant is trusting that the database will accept it without error.

What This Message Does Not Do

It is worth noting what this message does not accomplish. It does not run any migration framework—the table is created directly via ycqlsh, bypassing any versioned migration system. It does not create indexes (though none may be needed for this table). It does not set any table properties like default_time_to_live, compaction, or caching. The S3Objects table created in message 609 had WITH CLUSTERING ORDER BY (key ASC) and other properties inherited from the old table definition; the MultipartUploads table here has none of those, relying on defaults.

This is a pragmatic choice for a test cluster. The goal is to get a working system, not to productionize the schema. But it also means that if this table were promoted to production, additional tuning would be needed.

The Thinking Process Visible in the Sequence

The sequence of messages 607 through 610 reveals a clear diagnostic pattern:

Observe failure: The S3 proxy returns Internal Server Error with the error message Undefined Column. Column doesn't exist — SELECT node_id FROM S3Objects.
Inspect the database: The assistant runs DESCRIBE TABLE to see the actual schema in YugabyteDB, confirming the column is missing.
Repair the schema: The assistant drops the old tables and recreates S3Objects with the correct columns.
Complete the schema: Recognizing that S3Objects is not the only table the proxy needs, the assistant proactively creates MultipartUploads before the proxy encounters a related error. This is a pattern of anticipatory debugging. Rather than waiting for the next error to surface, the assistant reasons about what other tables the proxy will need and creates them preemptively. The MultipartUploads table is the natural next dependency: if the proxy handles multipart uploads (which it must, to be S3-compatible), it will need this table. Creating it now saves a future debugging cycle.

Input Knowledge Required

To understand this message fully, one needs:

Knowledge of the S3 protocol: Specifically, the multipart upload API and how UploadId is used to track in-progress uploads.
Knowledge of CQL (Cassandra Query Language): The compound primary key syntax, the distinction between partition keys and clustering columns, and how data distribution works in a distributed database.
Knowledge of the architecture: The three-layer design with stateless S3 proxies, Kuri storage nodes, and YugabyteDB as the metadata store. Understanding that node_id is the routing key that connects the proxy layer to the storage layer.
Knowledge of the debugging context: The preceding failures (missing node_id in S3Objects, HTTP route conflicts, web UI placeholder) and how they were resolved.

Output Knowledge Created

This message produces:

A database table: filecoingw_s3.MultipartUploads in YugabyteDB, ready to accept multipart upload metadata.
A precondition for S3 multipart upload support: The proxy can now create, read, and delete entries in this table as part of the multipart upload workflow.
A reduction in technical debt: By creating this table now, the assistant prevents a future failure when a client attempts a multipart upload.
Documentation of the schema: The command itself serves as a record of the table structure, which can be referenced later for migration scripts or production deployment.

Conclusion

Message 610 is a quiet moment in a noisy debugging session. It is not a dramatic fix—no panic is resolved, no crash is averted. But it is a foundational act. The MultipartUploads table is one of the invisible dependencies that a distributed storage system relies on to function correctly. By creating it proactively, the assistant demonstrates an understanding of the system's full dependency graph and a commitment to building it out completely, not just reactively patching the immediate failure. In the architecture of the horizontally scalable S3 gateway, this table is a small but essential gear, and its creation marks a step toward a complete, working system.