The Art of Architectural Deferral: Why GPU Affinity Wasn't Built in Phase 1

Introduction

In the life of any complex software project, there comes a moment when the team must decide not what to build, but what not to build right now. These decisions are often invisible in the final product—no code is written, no feature is shipped—yet they shape the architecture more profoundly than any implementation. Message 347 of the cuzk proving engine development session captures exactly such a moment: a careful, reasoned deferral of GPU affinity-based scheduling from Phase 1 to Phase 2, grounded in a deep understanding of how the underlying SRS (Structured Reference String) caching actually works.

This article examines that single message in detail, unpacking the reasoning process, the assumptions that were tested, and the architectural insight that led to a cleaner, more honest Phase 1 design.

The Message in Full

Let us begin by quoting the subject message exactly as it was written:

The current architecture has all workers calling scheduler.next() — they race on the same queue and whichever worker is idle first grabs the next job. This is actually a reasonable design for Phase 1 because:

1. With a shared queue + N workers, all N workers are always busy when there are N+ jobs 2. The priority ordering is already correct (WinningPoSt CRITICAL > WindowPoSt HIGH > PoRep NORMAL) 3. SRS affinity only matters when switching proof types (e.g., a PoRep worker picks up a WindowPoSt job) — and in the current GROTH_PARAM_MEMORY_CACHE model, the SRS is process-global (all workers share the same in-memory cache), so there's no per-GPU SRS to track

The GROTH_PARAM_MEMORY_CACHE is a global lazy_static HashMap — it's shared across all threads. So actually, affinity-based routing to specific GPUs for SRS reasons doesn't apply in Phase 1 because the SRS is in host memory, not per-GPU VRAM. All workers benefit from the same cache.

True GPU affinity (routing based on which GPU has the SRS in VRAM) requires the custom SRS manager from Phase 2+. For Phase 1, the shared-queue-with-priority-ordering is the correct design.

However, what I should do is implement notify_waiters() to wake ALL workers (not just one) when a high-priority job arrives, so that if a lower-priority job is about to be picked up, the higher-priority one wins. Currently notify_one() is used. But since workers call next() which always pops the highest-priority entry, the ordering is already correct as long as any worker wakes up. The notify_one() + notify_waiters() distinction only matters for fairness, not correctness.

Let me instead focus on the gen-vanilla command which will allow us to actually test the new proof types end-to-end. This is a meaningful deliverable.

Context: What Led to This Message

To understand why message 347 was written, we must trace the events that preceded it. The cuzk project is a pipelined SNARK proving daemon for Filecoin—a system designed to continuously generate Groth16 proofs for storage proofs across multiple sectors and proof types. By Phase 1, the assistant had already implemented the core PoRep C2 proving pipeline, added observability, and was now extending the engine to support all four Filecoin proof types: PoRep C2, WinningPoSt, WindowPoSt, and SnapDeals.

The immediate trigger for message 347 was a review of the project plan document (cuzk-project.md). After committing the Phase 1 core implementation at commit d8aa4f1d—which included real proving functions for all four proof types, a multi-GPU worker pool, priority scheduling, and an updated bench tool—the assistant turned to assess what remained. The project plan explicitly listed "Scheduler GPU affinity routing (dispatch to worker with matching SRS)" as a Phase 1 deliverable. The assistant had just read the plan ([msg 343] through [msg 345]) and was evaluating whether to implement this feature next.

The todo list at that moment showed:

Phase 1 remaining: Scheduler GPU affinity routing (dispatch to worker with matching SRS) — status: in_progress
Phase 1 remaining: gen-vanilla command — status: pending

The assistant was poised to implement GPU affinity routing. But instead of diving into code, it paused to think.

The Reasoning Process: A Step-by-Step Analysis

What makes message 347 remarkable is the clarity and honesty of its reasoning. Let us walk through each step.

Step 1: Recognizing the Current Architecture

The assistant begins by stating the current design: all workers call scheduler.next() on a shared priority queue. This is a classic work-stealing pattern—N workers compete for jobs from a single BinaryHeap, and whichever worker is idle first grabs the next highest-priority job. This design is simple, correct, and well-understood.

Step 2: Evaluating Against Requirements

The assistant then evaluates this architecture against three criteria:

Utilization: With a shared queue and N workers, all workers stay busy as long as there are N+ jobs pending. This is optimal for throughput.
Priority ordering: The BinaryHeap naturally orders by priority (CRITICAL > HIGH > NORMAL), so the highest-priority job is always served next regardless of which worker picks it up.
SRS affinity: This is the crux of the analysis. The project plan calls for routing jobs to workers whose GPUs already have the matching SRS loaded, to avoid costly SRS swaps. But does this actually matter in Phase 1?

Step 3: The Critical Insight — Process-Global SRS

Here the assistant makes a crucial observation: the GROTH_PARAM_MEMORY_CACHE is a global lazy_static HashMap shared across all threads. This means the SRS lives in host memory, not in per-GPU VRAM. Every worker, regardless of which GPU it controls, benefits from the same in-memory cache. There is no per-GPU SRS to track because the SRS is never loaded into GPU VRAM in the current architecture—it's loaded into host memory by the FFI layer and accessed by GPU kernels through CUDA's unified memory or explicit H2D transfers.

This insight fundamentally changes the analysis. If the SRS is process-global, then there is no benefit to routing a job to a specific GPU based on SRS affinity. All GPUs have equal access to the same cached SRS. The feature the project plan calls for is, in the current architecture, a solution to a problem that doesn't exist.

Step 4: Distinguishing Phase 1 from Phase 2

The assistant then draws a clear boundary: "True GPU affinity (routing based on which GPU has the SRS in VRAM) requires the custom SRS manager from Phase 2+." This is a forward-looking architectural statement. The project plan envisions a future where the SRS manager loads parameters directly into GPU VRAM, bypassing host memory for lower latency. In that future architecture, each GPU would have its own VRAM-resident SRS, and routing a job to the wrong GPU would incur a costly reload. But that future doesn't exist yet. The custom SRS manager is a Phase 2+ deliverable.

By deferring GPU affinity to Phase 2, the assistant aligns the implementation with the actual architecture rather than with an aspirational plan. This is a mark of engineering maturity: building what the system is, not what the plan says it will be.

Step 5: A Minor Optimization Consideration

The assistant briefly considers a small optimization: changing notify_one() to notify_waiters() to ensure all workers wake up when a high-priority job arrives. But it quickly dismisses this as unnecessary for correctness, since the priority queue always pops the highest-priority entry regardless of which worker is woken. The distinction between notify_one and notify_waiters matters for fairness (preventing starvation), not for priority ordering. This shows the assistant's attention to detail—it considered the optimization but correctly judged it as out of scope.

Step 6: Redirecting Effort

Finally, the assistant redirects its effort toward the gen-vanilla command—a tool for generating vanilla proofs for PoSt and SnapDeals, which would enable end-to-end testing of the newly implemented proof types. This is a meaningful deliverable that directly supports the Phase 1 goal of supporting all four proof types. The assistant explicitly states: "This is a meaningful deliverable."

Assumptions Made and Tested

Message 347 reveals several assumptions, both explicit and implicit:

Assumption 1: The SRS is process-global

This is the foundational assumption of the entire analysis. The assistant asserts that GROTH_PARAM_MEMORY_CACHE is a global lazy_static HashMap shared across all threads. This assumption is based on prior knowledge of the filecoin-proofs-api and bellperson internals, gained through extensive research earlier in the session (see [msg 318] through [msg 339]). The assistant had read the FFI layer, examined the SRS loading code, and understood the caching mechanism.

Was this assumption correct? Based on the evidence available in the conversation, yes. The GROTH_PARAM_MEMORY_CACHE is indeed a process-global cache. The assistant's deep investigation of the C2 pipeline in Segment 0 had mapped the entire call chain from Curio to CUDA kernels, including memory accounting. This assumption is well-grounded.

Assumption 2: GPU affinity only matters for SRS reasons

The assistant assumes that the only reason to route jobs to specific GPUs is SRS affinity. This is a reasonable assumption given the project plan's framing, but it's worth questioning. Could there be other reasons for GPU affinity? For example:

Thermal or power management: Some GPUs might be throttled while others are cool.
Memory fragmentation: Different GPUs might have different amounts of free VRAM.
Job size matching: Smaller proofs could be routed to less powerful GPUs. None of these are mentioned in the project plan, and the assistant's focus on SRS affinity is consistent with the documented requirements. The assumption is reasonable for Phase 1.

Assumption 3: The shared queue design is correct for Phase 1

The assistant asserts that the shared-queue-with-priority-ordering design is "actually a reasonable design for Phase 1." This is a judgment call that balances simplicity against theoretical optimality. A per-worker channel design with explicit routing could theoretically achieve better SRS affinity, but since SRS affinity doesn't matter in Phase 1, the shared queue is simpler and equally effective.

Assumption 4: The plan can be deviated from

The assistant implicitly assumes that the project plan is a guide, not a specification. This is a healthy engineering attitude. The plan says "implement GPU affinity routing," but the assistant evaluates whether that actually makes sense given the current architecture and decides to defer it. This requires confidence and authority—the assistant is acting as an architect, not a code monkey.

Input Knowledge Required

To fully understand message 347, one needs:

Understanding of the cuzk architecture: The message references the scheduler, worker pool, priority queue, and the distinction between notify_one() and notify_waiters(). Without knowing that workers call scheduler.next() on a shared BinaryHeap, the reasoning is opaque.
Knowledge of Groth16 and SRS: The message discusses SRS (Structured Reference String) caching, the difference between host-memory and VRAM-resident SRS, and the implications for GPU affinity. This requires domain knowledge of zero-knowledge proof systems.
Familiarity with the GROTH_PARAM_MEMORY_CACHE: The message references this specific global cache. Understanding that it's a lazy_static HashMap in the filecoin-proofs-api layer is essential to following the reasoning.
Context of the project plan: The message is a response to reading cuzk-project.md and evaluating remaining deliverables. The reader needs to know that the plan explicitly lists GPU affinity routing as a Phase 1 deliverable.
Knowledge of CUDA and GPU programming: The distinction between host-memory SRS and VRAM-resident SRS, and the implications for multi-GPU setups, requires understanding of CUDA memory models.
Awareness of the four Filecoin proof types: The message references PoRep C2, WinningPoSt, WindowPoSt, and SnapDeals, and their priority levels (CRITICAL, HIGH, NORMAL).

Output Knowledge Created

Message 347 produces several valuable pieces of knowledge:

A clear architectural decision: GPU affinity scheduling is deferred to Phase 2. This is recorded in the todo list as "DEFERRED to Phase 2 (SRS is process-global in Phase 1, per-GPU affinity needs custom SRS manager)."
A documented rationale: The reasoning is captured in the conversation, providing future developers (or the same assistant in later phases) with the context for why this decision was made. When Phase 2 arrives and someone asks "why wasn't GPU affinity built in Phase 1?", the answer is here.
A boundary between phases: The message explicitly defines what belongs to Phase 1 (shared queue, priority ordering, multi-GPU workers) and what belongs to Phase 2+ (custom SRS manager, VRAM-resident SRS, GPU affinity routing). This boundary prevents scope creep and keeps Phase 1 focused.
A redirection of effort: The assistant commits to building the gen-vanilla command instead, which is a concrete, testable deliverable that directly supports the Phase 1 goal.
A subtle optimization insight: The analysis of notify_one() vs notify_waiters() and why it doesn't matter for correctness is a piece of systems knowledge that might be useful later.

Mistakes and Incorrect Assumptions

Are there any mistakes in this message? Let us examine critically.

Potential Mistake: Overlooking SRS Load Time

The assistant assumes that because the SRS is process-global, there is no benefit to GPU affinity. But consider: even if the SRS is cached in host memory, loading it into GPU VRAM (via cudaMemcpy) is not instantaneous. If a worker switches from PoRep (32 GiB SRS) to WindowPoSt (different SRS), the GPU must transfer the new SRS from host to device. This H2D transfer time is non-trivial—earlier analysis in the session identified it as a significant overhead (see the micro-optimization analysis in Segment 1).

However, this is a subtle point. The GROTH_PARAM_MEMORY_CACHE caches the parameters in host memory, but the GPU kernels still need to access them. In the current architecture, the FFI layer handles H2D transfers internally. The question is whether the GPU VRAM cache persists across proof types. If the FFI re-uploads parameters for each proof, then there is a benefit to keeping a worker on the same proof type—it avoids the H2D transfer. But this benefit is independent of which GPU the worker runs on; it's about keeping the same worker on the same proof type, not routing to a specific GPU.

The assistant's conclusion that "affinity-based routing to specific GPUs for SRS reasons doesn't apply in Phase 1" is still correct. The optimization that would apply is proof-type affinity (keep a worker on PoRep rather than switching it to WindowPoSt), not GPU affinity. But that's a different feature, and the assistant correctly identifies that the shared queue already handles this naturally—if there are always PoRep jobs in the queue, a worker will tend to stay on PoRep.

Potential Mistake: Underestimating notify_waiters()

The assistant says the notify_one() vs notify_waiters() distinction "only matters for fairness, not correctness." This is true for the current design where all workers are identical. But if Phase 2 introduces GPU affinity, then notify_waiters() becomes important: when a high-priority job arrives that must go to a specific GPU, you need to wake all workers to ensure the right one picks it up. The assistant acknowledges this implicitly by saying it's "out of scope for now."

No Actual Mistakes

Overall, the message is remarkably free of errors. The reasoning is sound, the assumptions are well-grounded, and the conclusion is correct. The only potential oversight—the H2D transfer cost—doesn't change the conclusion because it's about proof-type affinity, not GPU affinity.

The Thinking Process: A Window into Engineering Judgment

What makes message 347 worth studying is not the code it produces (it produces none) but the thinking it reveals. Let us examine the cognitive process visible in the reasoning.

Pattern Recognition

The assistant recognizes a pattern: the project plan calls for a feature (GPU affinity routing) that is typically implemented in systems where SRS is per-GPU. But the assistant has internalized the actual architecture and recognizes that this pattern doesn't apply. This is pattern recognition at the architectural level—seeing beyond the abstract requirement to the concrete implementation.

Cost-Benefit Analysis

The assistant implicitly performs a cost-benefit analysis:

Cost of implementing GPU affinity now: Significant refactoring of the scheduler, adding per-worker channels, implementing routing logic, testing, and maintenance.
Benefit of GPU affinity now: Zero, because the SRS is process-global and there's no per-GPU state to optimize for.
Cost of deferring to Phase 2: None, because the feature will be implemented when it actually provides value (when the custom SRS manager makes SRS per-GPU). This is a textbook example of the YAGNI (You Ain't Gonna Need It) principle applied correctly.

Honesty with the Plan

The assistant demonstrates intellectual honesty by questioning the plan rather than blindly implementing it. This is a rare and valuable quality. The plan says "implement GPU affinity routing," but the assistant evaluates whether that makes sense now and concludes it doesn't. This requires confidence, domain knowledge, and the willingness to push back against documented requirements.

Forward Thinking

The assistant doesn't just defer the feature; it provides the condition for its eventual implementation: "True GPU affinity requires the custom SRS manager from Phase 2+." This forward reference creates a clear dependency chain. When Phase 2 arrives, the developer (or the assistant) will know exactly what prerequisite must be met before GPU affinity becomes meaningful.

Conclusion

Message 347 is a masterclass in architectural decision-making. It demonstrates that the most important decisions in software engineering are often not about what to build, but what to defer, and why. The assistant's reasoning—grounded in a deep understanding of the SRS caching model, the distinction between host-memory and VRAM-resident parameters, and the honest evaluation of the current architecture against the project plan—produces a cleaner, more honest Phase 1 design.

The message also reveals something about the nature of expertise. An inexperienced engineer might have dutifully implemented GPU affinity routing as the plan specified, adding complexity without benefit. An experienced engineer recognizes when a feature is premature and has the confidence to defer it. This is the art of architectural deferral: knowing not just how to build, but when to wait.

The cuzk project will eventually need GPU affinity routing—when the custom SRS manager loads parameters into VRAM, when different GPUs have different SRS states, when the cost of switching a GPU between proof types becomes measurable. But that day is not Phase 1. By deferring the feature to Phase 2, the assistant ensures that when it is built, it will be built for the right reasons, in the right architecture, at the right time.

In the meantime, the assistant redirects effort toward the gen-vanilla command—a concrete, testable deliverable that enables end-to-end verification of the Phase 1 proof types. This is the hallmark of a disciplined engineer: always asking not just "can I build this?" but "should I build this now?"