The Architecture of a Diagnosis: Designing Phase 11 for Memory-Bandwidth-Aware Pipeline Scheduling

Introduction

In the high-stakes world of Filecoin proof generation, every second counts. The cuzk SNARK proving engine had already been through ten optimization phases, squeezing performance from a baseline of 360 seconds per proof down to just 32 seconds in ideal conditions. But a troubling pattern had emerged: under realistic high-concurrency workloads, throughput degraded back to 38 seconds per proof, and the team couldn't figure out why. The GPU was only 90.8% utilized. The kernels were fast. The architecture was pipelined. Something else was stealing performance.

This article examines a single pivotal message in that optimization journey — message index 2703 — where the assistant synthesized weeks of diagnostics, four parallel research subagent tasks, and deep waterfall timing analysis into a concrete three-intervention plan called Phase 11. The message represents a critical inflection point: the moment when a team stops optimizing the obvious bottlenecks and starts addressing the subtle, systemic ones. It is a masterclass in diagnostic reasoning, cross-language systems design, and the discipline of evidence-based optimization.

The Context: A Failed Phase and a Mysterious Bottleneck

To understand why message 2703 matters, we must first understand what led to it. The cuzk project had been on a remarkable trajectory. Phase 0 established SRS residency, jumping from 360s to 89s per proof. Phase 2 introduced synthesis/GPU pipelining. Phase 3 added cross-sector batching. Phase 7 redesigned the engine for per-partition dispatch. Phase 8 implemented a dual-worker GPU interlock that pushed throughput to 37.4 seconds — a 9.6x improvement over baseline.

Phase 9 was the PCIe transfer optimization. By switching from unpinned stream.HtoD() (which goes through a bounce buffer) to pinned DMA with cudaHostRegister, and overlapping upload with compute via CUDA events, the team cut GPU kernel time per partition from 3.75 seconds to 1.82 seconds. The NTT phase alone dropped from 2.4 seconds to 0.69 seconds — a 71% improvement. In isolation, Phase 9 achieved 32.1 seconds per proof.

Then came Phase 10. The design was ambitious: split the single GPU mutex into two locks — a compute_mtx for kernel execution and a mem_mtx for VRAM allocation — allowing three workers to overlap CPU work with GPU compute. It was a natural evolution of the dual-worker interlock from Phase 8. But it failed catastrophically. The 16 GB VRAM on the RTX 5070 Ti simply couldn't accommodate pre-staged buffers from multiple workers simultaneously. Worse, CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global — they serialize with any running kernels, defeating the entire purpose of splitting the lock.

The team reverted to Phase 9's proven single-lock approach and ran comprehensive benchmarks across concurrency levels from c=5 j=5 through c=20 j=15. The waterfall timing analysis, extracted from TIMELINE events in the daemon logs, revealed the truth: at high concurrency, GPU per-partition time inflated from 4.9s to 7.5s, prep_msm inflated from 1.7s to 2.7s+, and synthesis inflated from 35s to 54s. The GPU was waiting — not for compute, but for data. The bottleneck was DDR5 memory bandwidth contention.

The Research Phase: Four Subagent Tasks in Parallel

Before message 2703 could be written, the assistant needed to understand the bottleneck at a fundamental level. The user had asked two pointed questions (message 2696/2698): could hugepages help, or could the pipeline be interleaved to spread out memory-bandwidth-heavy phases? These questions triggered four parallel research subagent tasks, each exploring a different dimension of the problem.

The first task researched hugepages and memory bandwidth. The finding was nuanced: the bottleneck wasn't raw DDR5 bandwidth (the system had ~333 GB/s theoretical, and the pipeline only demanded ~34 GB/s), but rather TLB pressure. With 4 KB pages, the L2 TLB covers only 12 MB. The working set was 200+ GiB across 10 concurrent syntheses plus prep_msm plus deallocation. Every access to the 25.7 GiB PCE CSR matrices was an L1 TLB miss. Hugepages could help, but the mechanism wasn't about bandwidth — it was about reducing TLB miss penalties.

The second task explored the memory-bandwidth-heavy code paths in detail. It mapped every significant memory operation: the PCE MatVec (sparse matrix-vector multiply reading ~49 GiB of shared CSR data), prep_msm (reading ~23 GiB for bitmap classification and tail MSM population), b_g2_msm (the 0.4s GPU-side MSM that runs 192 threads in parallel), and async_dealloc (background threads freeing ~37 GiB of std::vector contents via munmap()).

The third task ran diagnostic commands to understand the system's NUMA topology and hugepage state. The system was a single NUMA node (NPS1) with 12 CCDs, each with 32 MB L3 cache — 384 MB total. Transparent Hugepages was in always mode, but defrag was madvise-only. Only 325 × 2 MB hugepages were reserved (trivial), and no 1 GB hugepages were configured. Free RAM was ~433 GiB — ample room for reconfiguration.

The fourth task performed a temporal overlap analysis of the pipeline at steady state. With c=15 j=15, pw=10, and gw=2, the system had 10 partition workers running synthesis concurrently, while prep_msm and b_g2_msm ran on the C++ groth16_pool (192 threads) and async_dealloc ran unbounded in the background. The analysis revealed that all these memory-heavy phases overlapped in time, competing for the same DDR5 memory channels and L3 cache domains.

The Message: Phase 11 — Memory-Bandwidth-Aware Pipeline Scheduling

Message 2703 is the synthesis of all this research. It is not a code change, not a benchmark run, not a debugging session — it is a design proposal. The assistant lays out the problem statement, identifies the three root causes, and proposes three independent interventions with effort estimates, implementation details, and a recommended order of execution.

The message begins with a stark problem statement:

At high concurrency (c=15-20 j=15), throughput degrades from 32.1s/proof (isolation) to 38.0s/proof. The waterfall analysis shows: - GPU per-partition time inflates from 4.9s to 7.5s under load - prep_msm inflates from 1.7s to 2.7s+ - Synthesis inflates from 35s to 54s

Then the crucial reframing: the root cause is not raw DDR5 bandwidth. The system has ~333 GB/s theoretical bandwidth, and the pipeline only demands ~34 GB/s. The problem is structural — three distinct mechanisms that each amplify the others:

  1. TLB shootdowns from async_dealloc: Background threads freeing ~37 GiB each via munmap() trigger thousands of TLB invalidation IPIs (Inter-Processor Interrupts) that stall all 192 hardware threads.
  2. Thread pool thrashing: Two separate thread pools — rayon for synthesis and groth16_pool for prep_msm/b_g2_msm — both default to 192 threads, all competing for 12 L3 cache domains.
  3. Phase overlap: prep_msm, b_g2_msm, async_dealloc, and 10 concurrent syntheses all hammer memory simultaneously, creating contention that inflates each phase's duration. This reframing is the intellectual core of the message. The assistant has moved from "the GPU is slow" to "the CPU memory subsystem is congested" to "the congestion has three specific, measurable causes." Each cause points to a different intervention.

Intervention 1: Bounding async_dealloc

The first intervention is the simplest and most surgical. The C++ code in groth16_cuda.cu at line 1044 spawns deallocation threads with std::thread(...).detach() — unbounded background threads that call munmap() on ~37 GiB of memory. Under load, 2-3 concurrent dealloc threads can fire simultaneously, each triggering thousands of TLB invalidation IPIs.

The fix is a static std::counting_semaphore<1> that serializes deallocation to at most one concurrent thread. The dealloc thread acquires the semaphore before starting and releases it when done. This is approximately 30 lines of C++ code — low effort, high impact.

But the assistant goes further, identifying a second dealloc point on the Rust side (supraseal.rs:393-404) that frees ~130 GiB. Both need serialization. The Rust side would use a tokio::sync::Semaphore or std::sync::Mutex.

The bonus suggestion — replacing munmap()-heavy deallocation with madvise(MADV_DONTNEED) + deferred munmap() — shows deep understanding of the Linux memory subsystem. MADV_DONTNEED marks pages as reclaimable without modifying page tables immediately, avoiding the TLB shootdown entirely. This is a more sophisticated approach that could be explored if the semaphore alone isn't sufficient.

Intervention 2: Memory-Phase Interlock

The second intervention is the most architecturally interesting. The key insight is that prep_msm (1.9 seconds, reading ~23 GiB) and the PCE MatVec in synthesis (9 seconds per partition, reading ~49 GiB of shared CSR data) both do streaming memory reads. When they overlap, each gets roughly 50% of available memory bandwidth due to L3 thrashing across CCDs.

The proposed solution is a shared "memory-heavy phase" semaphore with configurable capacity N, passed through the FFI layer (similar to how gpu_mtx is already passed). Synthesis would acquire one permit before entering spmv_parallel() and release after. prep_msm would acquire one permit before its CPU-heavy phases and release after. With capacity N=8 (out of 10 partition workers), at most 8 syntheses and 0 prep_msm can run simultaneously, OR 7 syntheses and 1 prep_msm. This ensures prep_msm never competes with the full pack of synthesis workers.

The assistant also presents a simpler alternative: since prep_msm and synthesis use different thread pools, simply reducing groth16_pool thread count (via CUZK_GPU_THREADS=32 instead of default 192) would reduce contention without adding a semaphore. But the user chose the full semaphore approach, and the message ends with a question confirming that choice.

The assistant's note that b_g2_msm doesn't need the semaphore — it's short (0.4s) and runs after prep_msm releases — shows careful attention to the actual timing profile rather than applying a blanket solution.

Intervention 3: CCD-Aware Thread Placement

The third intervention addresses the spatial competition between thread pools. Currently, both rayon (192 threads) and groth16_pool (192 threads) spread across all 12 CCDs. When prep_msm runs on groth16_pool, its threads compete with synthesis rayon threads for L3 cache on every CCD.

The proposal is to set CUZK_GPU_THREADS=16 (2 CCDs worth) and optionally pin groth16_pool threads to specific CCDs (e.g., Dies 10-11, cores 80-95 / 176-191) using pthread_setaffinity_np(). The rayon threads would use the remaining 10 CCDs (cores 0-79 / 96-175).

This creates spatial isolation: synthesis uses 10 CCDs (80 cores), prep_msm uses 2 CCDs (16 cores). They share memory channels but not L3 caches, reducing eviction pressure. The assistant correctly notes that this doesn't eliminate memory channel contention (both groups still use DDR5), but it eliminates L3 thrashing, which is a significant source of latency.

The Implementation Order and the Question

The message concludes with a recommended implementation order: Intervention 1 first (smallest change, eliminates the most unpredictable interference source), then Intervention 3 (config change + small code change, easy to benchmark), then Intervention 2 (most complex, deferred to evaluate if 1+3 are sufficient).

The final element is a structured question to the user with two options: implement all three interventions including the full FFI semaphore, or implement all three but use pool size reduction instead of the semaphore. The user chose "All 3, semaphore interlock," which set the direction for the next phase of work.

Assumptions and Knowledge Boundaries

The message makes several implicit assumptions worth examining. First, it assumes that TLB shootdowns from munmap() are a significant contributor to performance degradation. This is well-supported by Linux kernel literature — TLB invalidation IPIs are known to cause measurable stalls on large-manycore systems — but the assistant hasn't directly measured the stall time. The assumption is based on the mechanism, not on direct instrumentation.

Second, it assumes that L3 cache contention between thread pools is a significant factor. The Threadripper PRO 7995WX has 12 CCDs each with 32 MB L3, totaling 384 MB. The working set of a single synthesis is ~49 GiB for the CSR matrices alone — far larger than L3. However, temporal locality within the SpMV kernel means that some fraction of accesses benefit from L3, and contention reduces that benefit. The magnitude of this effect is estimated but not measured.

Third, the message assumes that the three interventions are independent and composable — that implementing all three will produce additive benefits without negative interactions. This is a reasonable engineering assumption, but it's untested. The semaphore interlock (Intervention 2) and the thread pool size reduction (Intervention 3) could interact in unexpected ways, for example if the reduced thread pool causes prep_msm to take longer, which changes the overlap pattern that the semaphore is designed to manage.

The assistant also assumes that the FFI layer can pass a semaphore with the same mechanism used for gpu_mtx. This is a safe assumption given the existing architecture, but it does require cross-language coordination between C++ (where the semaphore lives) and Rust (where synthesis runs).

Input Knowledge and Output Knowledge

To understand this message, a reader needs significant context. They need to know the cuzk architecture: the partition pipeline, the GPU worker model, the synthesis/GPU overlap, the role of prep_msm and b_g2_msm. They need to understand the CUDA programming model: streams, events, device-global synchronization, pinned memory. They need to understand the Linux memory subsystem: TLB, hugepages, munmap(), madvise(), IPI-based TLB shootdowns. They need to understand the Threadripper architecture: CCDs, L3 cache domains, NUMA topology, DDR5 channels.

The message creates new knowledge in several forms. It establishes a causal model for the observed throughput degradation at high concurrency — moving from "GPU utilization is 90.8%" to "three specific memory subsystem mechanisms are causing contention." It produces a ranked, implementable plan with effort estimates and risk assessments. It identifies specific code locations (groth16_cuda.cu:1044, supraseal.rs:393-404, cuzk-pce/src/eval.rs) where changes are needed. It creates a decision point for the user (the semaphore vs. pool-size-reduction question) that shapes the next phase of work.

The Thinking Process: From Symptoms to Mechanisms

What makes message 2703 remarkable is the quality of reasoning visible in its structure. The assistant doesn't jump to conclusions. It follows a disciplined diagnostic process:

  1. Observe the symptom: Throughput degrades from 32.1s to 38.0s under concurrency.
  2. Measure precisely: Waterfall timing analysis shows which phases inflate and by how much.
  3. Rule out obvious causes: Raw DDR5 bandwidth isn't saturated (34 GB/s demand vs 333 GB/s capacity).
  4. Identify mechanisms: TLB shootdowns, thread pool thrashing, phase overlap — three distinct mechanisms that each degrade performance.
  5. Map mechanisms to interventions: Each mechanism has a natural fix.
  6. Estimate effort and impact: Rank interventions by complexity and expected benefit.
  7. Propose an order: Start with the simplest, most impactful change.
  8. Ask for confirmation: Present the user with a clear choice before proceeding. This is textbook systems optimization. The assistant resists the temptation to throw solutions at the problem. Instead, it builds a causal chain from symptom → measurement → mechanism → intervention. Each intervention is justified not by intuition but by a specific, measurable mechanism.

Conclusion: The Significance of a Design Message

Message 2703 is a design document, not an execution one. It contains no code changes, no benchmark results, no debugging output. Yet it is arguably the most important message in the Phase 11 cycle. It represents the moment when a complex, multi-dimensional performance problem was decomposed into three independent, tractable sub-problems, each with a clear fix.

The message also demonstrates the value of the subagent architecture. The assistant dispatched four parallel research tasks — hugepages, memory-heavy code paths, system diagnostics, temporal overlap analysis — and synthesized their results into a coherent plan. Each subagent explored a different dimension, and the synthesis produced insights that none could have produced alone.

The user's response — choosing the full semaphore interlock over the simpler pool-size reduction — set the direction for the implementation that followed. In the next chunk (Chunk 1 of Segment 28), the assistant documented the plan in c2-optimization-proposal-11.md, updated cuzk-project.md with the Phase 10 post-mortem, and committed all documentation. The implementation would follow, with careful benchmarking of each intervention independently.

Phase 11 would go on to achieve meaningful throughput improvements, but the real achievement of message 2703 is the diagnostic framework it established. The team now understood why performance degraded under concurrency, not just that it did. That understanding would inform every subsequent optimization, from thread placement to memory allocation strategies. In the end, the most valuable output of this message was not the three interventions — it was the causal model that made them visible.