The Diagnostic Pivot: When a Single Curl Command Exposes a Broken Assumption

Message in Focus

[assistant] Kuri is starting but complaining about configuration. Let me check a simpler test first: [bash] echo "test" | curl -s -X PUT -d @- http://localhost:8078/loadtest/test-object 2>&1 Service Unavailable - No healthy backends

At first glance, this message from a distributed systems debugging session appears unremarkable — a quick curl command, a seven-word error response. But in the arc of the conversation, it represents a critical diagnostic pivot: the moment when the engineer stops trusting the load test tool's output and reaches for the most primitive possible probe to determine whether the system is fundamentally alive. The message is brief, but it carries the weight of an entire debugging methodology.

Context: The Hunt for a Phantom Corruption

To understand why this message was written, we must trace the thread that led to it. The assistant had been deep in a multi-session effort to build and optimize a horizontally scalable S3-compatible storage system, comprising stateless frontend proxies, Kuri storage nodes, and a YugabyteDB coordination layer. The immediate preceding work involved a suspected data corruption problem during load testing — a serious concern for any storage system. After careful investigation, the assistant had determined that the "corruption" was actually a misclassification of timeout errors in the load test tool. The system was clean.

But the user wanted more. They asked for a restart with the newly implemented CQL batcher (a performance optimization for database writes) and a test at three escalating concurrency levels: 10, 100, and 1000 parallel workers. This was a reasonable request — validate that the optimization doesn't break correctness, and measure the throughput improvement.

What followed was a cascade of infrastructure failures that had nothing to do with the batcher code.

The Infrastructure Spiral

The assistant attempted to restart the Docker containers running the Kuri storage nodes. This simple operation turned into a multi-step ordeal:

  1. Permission denied: A pkill command failed because the processes were owned by root.
  2. Missing config: The assistant looked for configuration files in the wrong directory, revealing confusion about the deployment model.
  3. User correction: "It's running in docker-compose, no?" the user prompted, redirecting the assistant to the test-cluster directory.
  4. Old image: The initial restart used an old container image that didn't include the batcher changes. The assistant had to force-recreate with --force-recreate and a freshly built fgw:local image.
  5. Configuration error: The kuri-1 logs showed a fatal-sounding error: Configuration load failed: %w RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1. At this point, the assistant was staring at a broken cluster. The load test tool had been returning write errors. The kuri logs showed a configuration validation failure. The natural instinct was to dive into the configuration file, fix the threshold parameter, and restart again.

The Diagnostic Pivot

But the assistant did something smarter. Instead of immediately chasing the configuration error, they paused and wrote the subject message: "Kuri is starting but complaining about configuration. Let me check a simpler test first."

This sentence reveals a crucial assumption: the assistant believed the configuration error might be a non-fatal warning — a "complaint" rather than a showstopper. The phrasing "is starting but complaining" suggests the assistant thought the Kuri node might have launched despite the configuration issue, perhaps with degraded functionality. This was a reasonable hypothesis given that the logs showed the node proceeding past the error to generate an ED25519 keypair and initialize an IPFS node.

The decision to "check a simpler test first" is a textbook debugging maneuver. The load test tool is a complex piece of software — it generates random object names, manages concurrent workers, tracks statistics, performs read-after-write verification, and handles cleanup. Any of these subsystems could introduce confounding behavior. By reaching for curl, the assistant eliminates all that complexity. A single PUT request with a literal string "test" as the body is the simplest possible write operation. If it works, the problem is in the load test tool or its configuration. If it fails, the problem is in the infrastructure.

The Response: "Service Unavailable - No Healthy Backends"

The response was immediate and devastating. The S3 frontend proxy — the stateless entry point that routes requests to Kuri storage nodes — returned "Service Unavailable - No healthy backends." This is not a vague error. It is a specific, actionable signal from the proxy's backend health-checking system.

This response tells us several things simultaneously:

  1. The S3 proxy is running: It accepted the TCP connection and processed the HTTP request. The proxy itself is operational.
  2. The backend pool is empty or all backends are unhealthy: The proxy maintains a registry of Kuri nodes and periodically checks their health. Either no Kuri nodes are registered, or all registered nodes have failed their health checks.
  3. The configuration error is fatal: The "complaint" about RetrievableRepairThreshold > MinimumReplicaCount was not a warning — it prevented the Kuri node from fully initializing its storage layer, which in turn prevented it from passing the proxy's health check.

Assumptions Under the Microscope

The assistant made several assumptions in this message, some explicit and some implicit:

"Kuri is starting but complaining about configuration" — This assumes the configuration error is non-fatal. In many systems, configuration validation errors are indeed warnings; the system starts with degraded functionality or falls back to defaults. But in this case, the Kuri node's storage layer (the RIBS blockstore) apparently refused to initialize with an invalid threshold configuration, leaving the node in a state where it could generate a peer identity and initialize IPFS but could not serve storage requests.

"Let me check a simpler test first" — This assumes that the load test failures could be artifacts of the test tool rather than the infrastructure. It's a sound methodological assumption — always eliminate the test harness before debugging the system — but it implicitly assumes the infrastructure might be basically functional.

That a single PUT request would succeed if the system was operational — This is correct in principle but assumes the proxy's routing and health-checking logic is working correctly. As the next message (index 1119) would reveal, the proxy had indeed registered both Kuri backends — it just couldn't reach them because they weren't healthy.

Input Knowledge Required

To fully understand this message, a reader needs:

Output Knowledge Created

This message produced one of the most valuable outputs in debugging: a clear, unambiguous failure signal. Before this curl command, the assistant had ambiguous data — load test errors that could be infrastructure failures, test tool bugs, or genuine corruption. After this command, the problem was localized to the backend health-check system. The assistant now knew:

  1. The S3 proxy is alive and accepting connections.
  2. No Kuri backend is considered healthy.
  3. The configuration error in the Kuri logs is the likely root cause.
  4. The next debugging step is to examine the proxy's backend pool and health-check mechanism.

The Thinking Process Visible in the Message

The reasoning in this message is compressed but discernible. The assistant is performing a classic binary search of the problem space:

The Deeper Lesson

This message exemplifies a pattern that appears repeatedly in expert debugging: when a complex tool fails, reach for a simpler one. The load test tool is a sophisticated piece of instrumentation with concurrency management, statistics tracking, and verification logic. Each of these features is a potential source of confounding behavior. Curl, by contrast, has no features. It sends exactly the bytes you give it and reports exactly the response it receives. When curl fails, you know the problem is not in your test harness.

The "Service Unavailable - No healthy backends" response also demonstrates the value of descriptive error messages. A generic "500 Internal Server Error" would have left the assistant guessing. A "Connection refused" would have suggested the proxy itself was down. But "No healthy backends" precisely describes the proxy's internal state: it's running, it has backends registered, but none of them pass health checks. This error message itself becomes a debugging roadmap.

Conclusion

Message 1118 is a masterclass in diagnostic minimalism. In two lines of thought and one line of bash, the assistant transforms a confusing constellation of load test failures, configuration errors, and container restart issues into a single, actionable finding: the backend pool is empty. The message is short, but it represents the culmination of a debugging methodology that prioritizes simplicity, isolates variables, and lets the system speak for itself. The curl command is not a fix — it's a diagnosis. And in debugging, a precise diagnosis is often more valuable than a premature cure.