Breaking a Distributed Deadlock: The First Surgical Strike

The Message

### Fix 1: `CreateWorkAsk` — return immediately on 429
[edit] /tmp/czk/lib/proofsvc/provictl.go
Edit applied successfully.

This terse status line from the assistant at <msg id=1920> is the first concrete implementation step in a multi-fix campaign targeting a critical deadlock in the ProofShare system. On its surface, the message reports nothing more than a successful file edit. But to understand why this two-line message matters—why it represents the culmination of a deep systems analysis and the first incision into a production bug—requires reconstructing the chain of reasoning that led here.

The Deadlock: A Perfect Distributed Systems Trap

The story begins at <msg id=1907>, where the user reported a baffling production issue in TaskRequestProofs, a singleton harmony task that manages the lifecycle of "work asks" with a remote proof service. The system worked like this: TaskRequestProofs ran a polling loop that would contact the remote service, discover available work requests, insert them into a local proofshare_queue table, and then create new "asks" (offers to compute proofs) to keep the pipeline filled. The target was a high-water mark of eight concurrent asks.

The deadlock emerged from an interaction between two independent constraints. First, the remote service rate-limited ask creation: if a provider already had two active asks, it would return HTTP 429 (Too Many Requests). Second, the CreateWorkAsk function in provictl.go treated 429 as a transient error and retried indefinitely with exponential backoff, eventually capping at five-minute intervals. Third—and this was the crux—the polling loop could not advance past CreateWorkAsk. It was a sequential loop: poll, insert, create asks, sleep, repeat. If CreateWorkAsk hung forever on a 429, the loop never reached the next poll iteration.

The trap snapped shut as follows. The provider would have, say, two active asks. The loop calculated it needed six more, entered the ask-creation inner loop, and immediately hit a 429. CreateWorkAsk began its infinite retry dance. Meanwhile, the two existing asks got matched to work by the remote service—but the provider never polled again to discover this. The matched work never got inserted into proofshare_queue. Without queue entries, the PSProve tasks (which actually compute the proofs) never ran. Without completed proofs, the service never freed the ask slots. The 429 never cleared. The system was permanently deadlocked, spinning on a retry loop that could never succeed.

The user confirmed this diagnosis at <msg id=1910>: restarting a locked-up node broke the deadlock because the fresh poll discovered the already-matched work and inserted it into the queue, allowing PSProve tasks to finally run and free slots.

The Design Refinement

The assistant's initial proposal at <msg id=1915> was straightforward: make CreateWorkAsk return a sentinel ErrTooManyRequests immediately on 429 instead of retrying, and let the natural three-second poll interval provide backoff. The user pushed back at <msg id=1916> with a sharp observation: "We still want exponential backoff if there were no asks created, correct?"

This was a critical refinement. Without backoff, a system hammering a rate-limited service at three-second intervals would generate needless load and potentially trigger more aggressive rate limiting. The assistant revised the design at <msg id=1917> to track whether any progress had been made in a given iteration (either new work inserted or at least one ask created). If progress was made, the poll interval stayed at the normal three seconds. If no progress was made—meaning every ask attempt returned 429 and no new work appeared—the poll interval would back off exponentially from three seconds up to a two-minute cap. This preserved the deadlock-breaking property (the loop always returns to poll) while adding graceful degradation under sustained rate limiting.

The user responded with a single word at `<msg id=1918'>: "implement."

The First Edit

At <msg id=1919>, the assistant began executing the plan, creating a todo list with Fix 1 marked "in progress." Then came <msg id=1920>—the subject of this article.

The edit to /tmp/czk/lib/proofsvc/provictl.go accomplished two things in one stroke. First, it added a package-level sentinel error:

var ErrTooManyRequests = xerrors.New("too many requests")

This is a standard Go idiom for signaling a specific, recoverable error condition. The sentinel is important because it allows callers to distinguish "the service is busy, try later" from genuine failures like network errors or malformed responses. The comment in the code (visible in the follow-up read at <msg id=1928>) explicitly documents the design intent: "Callers should back off and retry later instead of blocking inside CreateWorkAsk, so that the poll loop can continue discovering work that was matched to existing asks."

Second, the edit removed the 429 retry loop from CreateWorkAsk. Previously, the function had a retry loop that caught HTTP 429 and re-attempted the request with exponential backoff up to a five-minute cap, using context.Background() (no overall timeout). After the edit, a 429 response causes CreateWorkAsk to return (0, ErrTooManyRequests) immediately—no retries, no blocking.

Why This Message Matters

This message is the fulcrum of the entire deadlock fix. Everything before it was analysis and design; everything after it is implementation and deployment. The edit itself is small—a sentinel error and a changed return path—but it represents a fundamental shift in the system's failure semantics.

The old design treated 429 as a transient condition to be retried until it succeeded. This is a reasonable default for many distributed systems: rate limits are usually temporary, and retrying with backoff is polite. But in this specific context, the retry created a circular dependency. The condition that would clear the 429 (completing existing proofs) could only happen if the polling loop advanced, but the polling loop could not advance because it was stuck in the retry.

The new design treats 429 as a signal to yield control back to the scheduler. The polling loop can continue, discover matched work, insert it into the queue, and let the proving pipeline drain. The backoff responsibility moves from the leaf function (CreateWorkAsk) to the orchestrator (Do()), which has the global context to decide whether to back off aggressively (no progress across multiple dimensions) or lightly (progress is being made somewhere).

Assumptions and Risks

The fix makes several assumptions. It assumes that the caller (Do() in task_request.go) will handle ErrTooManyRequests correctly—which the assistant was about to implement in the next edit at <msg id=1923>. It assumes that returning to the poll loop quickly is always better than waiting for the rate limit to clear, which is true as long as there is any possibility of discovering matched work. It assumes that the three-second default poll interval is short enough to not introduce unacceptable latency in ask creation, which the existing system already relied on.

One subtle risk: if the system is under sustained load and all ask attempts return 429, the progress-based backoff will kick in, potentially delaying ask creation by up to two minutes. This is a deliberate trade-off—better to create asks slowly than to deadlock permanently—but it could be surprising in a system that previously would have eventually retried through the 429.

The Broader Campaign

This edit was the first of five fixes the assistant implemented in this session. The subsequent messages show the companion changes: handling ErrTooManyRequests in task_request.go with progress-based backoff (<msg id=1923>), scoping the dedup SELECT to non-submitted rows (<msg id=1923>), hoisting the address resolver out of the inner loop (<msg id=1923>), changing orphan cleanup from DELETE to UPDATE to preserve fetched work (<msg id=1926>), and adding a routine to purge completed rows older than two days (<msg id=1926>). Each fix addressed a different facet of the same systemic issue: the polling loop must never block on a condition that can only be resolved by the loop itself.

The assistant's approach demonstrates a pattern for debugging distributed deadlocks: map the exact cycle of dependencies, identify the blocking point, and redesign the failure semantics so that the system can make progress on at least one axis. In this case, the insight was that CreateWorkAsk should never be the bottleneck—it should get out of the way and let the rest of the pipeline drain.