From Bandwidth Wall to Hugepages: A Systems-Level Investigation of Memory Contention in Filecoin Proof Generation

Introduction

In the course of optimizing the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), a critical bottleneck emerged that resisted purely architectural solutions: DDR5 memory bandwidth contention. The pipeline, which consumes approximately 200 GiB of peak memory and spans a call chain from Curio's Go task layer through Rust FFI into C++ and CUDA kernels, had been subjected to extensive waterfall timing analysis. Phase 10's two-lock GPU interlock design had been abandoned after discovering fundamental CUDA device-global synchronization conflicts. The code was reverted to Phase 9's proven single-lock approach, and comprehensive benchmarks across concurrency levels (c=5 through c=20) had revealed 90.8% GPU utilization at high concurrency, with throughput plateauing at ~38 seconds per proof. The culprit: synthesis and prep_msm phases both inflating under load as they competed for the same DDR5 memory channels.

Phase 11 was designed with three targeted interventions to reduce this contention—bounding async_deallocation to a single thread, reducing the groth16_pool thread count, and adding a lightweight semaphore interlock. But one question remained unanswered, a question that could unlock significant additional headroom without altering a single line of proof-generation code: Could hugepages reduce memory bandwidth contention on a Threadripper PRO 7995WX system?

This article examines the deep-dive investigation that answered that question—a subagent session that researched five interconnected technical domains, synthesized findings into actionable recommendations, and produced knowledge that directly informed the Phase 11 optimization strategy.

The Memory Bandwidth Wall

The Threadripper PRO 7995WX is a formidable piece of hardware: 96 Zen 4 cores across 12 CCDs, 8-channel DDR5-5200 memory, and a theoretical peak bandwidth of approximately 333 GB/s. Yet the C2 proof generation pipeline was experiencing bandwidth bottlenecks that limited throughput. The waterfall timing analysis had been precise: by extracting TIMELINE events from daemon logs, the team could see exactly when each phase of proof generation consumed memory bandwidth and how those phases overlapped.

The synthesis phase—responsible for constructing the rank-1 constraint system (R1CS) witness—and the prep_msm phase—which prepares the multi-scalar multiplication inputs—were both memory-bandwidth-heavy. When they overlapped, they competed for the same DDR5 channels, causing both phases to inflate in duration. The result was a throughput plateau that no amount of concurrency tuning could突破.

This was not a GPU bottleneck. At 90.8% GPU utilization, the GPUs were waiting on data from the CPU. The bottleneck was in the host memory subsystem—specifically, in how quickly the CPU could feed data to the GPU and how quickly the CPU could perform its own memory-intensive computations.

The Hugepages Hypothesis

The hypothesis was subtle but potentially high-impact. Every memory access by a running process requires a virtual-to-physical address translation. The CPU caches these translations in the Translation Lookaside Buffer (TLB), but TLBs are small. With standard 4 KiB pages, a workload with a 200 GiB working set will rapidly exhaust the TLB's capacity. When a TLB miss occurs, the CPU must perform a page table walk—up to four memory accesses on x86-64 with 4-level page tables. These page walks consume memory bandwidth and stall execution.

The Zen 4 microarchitecture has an L1 DTLB with 72 entries and an L2 DTLB with 3072 entries. With 4 KiB pages, the L2 DTLB covers only 12 MiB—a tiny fraction of the 200 GiB working set. The miss rate approaches 100% for random or streaming access patterns. With 2 MiB pages, the L1 DTLB covers 144 MiB and the L2 DTLB covers 6 TiB—effectively the entire address space. With 1 GiB pages, even the L1 DTLB covers 72 GiB.

The hypothesis was that the ~200 GiB working set was causing TLB thrashing, which in turn was consuming DDR5 bandwidth that should be going to actual computation. If this hypothesis were correct, switching to hugepages could reclaim a significant fraction of memory bandwidth without changing any application code.

The Five Research Questions

The subagent was tasked with researching five specific questions, each targeting a different technical dimension of the hugepages problem. The questions reveal a sophisticated understanding of the optimization landscape—they span the entire stack from hardware to application.

1. TLB Coverage and Memory Bandwidth

The first and most fundamental question asked whether 2 MiB or 1 GiB hugepages improve memory bandwidth for sequential scan workloads, what the mechanism is, and what realistic speedup numbers look like on Zen 4 / DDR5 systems.

The research confirmed the TLB coverage mechanism and provided concrete speedup numbers from multiple sources. Intel's Scalable Vector Search (SVS) showed 20% to 90% throughput improvement for graph-based similarity search with hugepages. The Rust compiler benchmark by Kobzol showed ~5% wall-time reduction and ~60% fewer page faults. PostgreSQL showed 5-15% improvement for query throughput with large shared_buffers. Erik Rigtorp's hash table benchmark showed dTLB-load-misses dropping from 93% to near-zero.

Critically, the research distinguished between random-access workloads (where the benefit is largest) and sequential scan workloads (where the benefit is more modest at 5-15%) because hardware prefetchers can partially overlap page walk latency with useful data fetches. For the C2 pipeline, which involves both sequential scans (large array traversals for polynomial evaluation) and irregular access patterns (sparse matrix-vector multiply for MSM), the expected benefit was estimated at 5-30% depending on the specific kernel.

2. Transparent Huge Pages vs. Explicit Hugepages

The second question addressed the operational tradeoff between Transparent Huge Pages (THP) and explicit HugeTLB allocation for long-lived daemon processes with 25-430 GiB allocations.

The research was unequivocal: Explicit HugeTLB is strongly preferred for this use case. The evidence came from documented production incidents. YugabyteDB's TA-26440 (2025) documented THP causing TServer RSS to grow from 9.6 GiB to 21 GiB actual RSS, triggering the OOM killer. A DigitalOcean/Redis incident showed THP with jemalloc causing memory that should have been freed to remain allocated because the kernel wouldn't break up huge pages. The khugepaged kernel thread introduces unpredictable latency spikes of 1-10 ms as it scans for 4 KiB pages to coalesce into 2 MiB pages.

The recommendation was a hybrid approach: reserve 1 GiB hugepages at boot via kernel parameters, use mmap with MAP_HUGETLB | MAP_HUGE_1GB for the large allocations, set THP to madvise mode (not always), and use madvise(MADV_HUGEPAGE) selectively for medium allocations. This approach maximizes predictability while still getting hugepage benefits for the bulk data structures.

3. NUMA Topology of Threadripper PRO 7995WX

The third question addressed the NUMA topology of the Storm Peak platform. The research revealed three NPS (NUMA Per Socket) configurations: NPS1 (1 NUMA node, 96 cores, 8 memory channels interleaved), NPS2 (2 nodes, 48 cores each, 4 channels each), and NPS4 (4 nodes, 24 cores each, 2 channels each).

The bandwidth implications were significant. In NPS4, accessing memory on a remote quadrant traverses the Infinity Fabric, adding ~30-50 ns of latency and reducing per-thread bandwidth by 30-50%. Phoronix benchmarks showed that for NUMA-aware workloads, NPS4 could provide 5-15% improvement, but for NUMA-unaware workloads, it could cause 5-10% regression.

The recommendation for a single-process large-allocation workload was NPS1 (default), because the OS interleaves memory across all 8 channels, giving maximum bandwidth to any thread regardless of which CCD it runs on. NPS2 was identified as a safe middle ground if thread pinning is feasible.

4. cudaHostRegister and Hugepages

The fourth question addressed whether CUDA's cudaHostRegister function works with hugepages—a critical concern because the C2 pipeline uses CUDA for GPU acceleration and relies on pinned host memory for DMA transfers.

The research confirmed that cudaHostRegister does work with hugepages, with significant benefits. Pinning 100 GiB of 4 KiB pages requires pinning 26.2 million pages; with 2 MiB pages, that drops to 51,200 pages; with 1 GiB pages, just 100 pages. This means faster registration, faster DMA setup (fewer scatter-gather list entries), and reduced kernel overhead.

The research also identified known issues, including NVIDIA open-gpu-kernel-modules issue #732 where cudaHostRegister fails on page-backed memory mapped with remap_pfn_range since driver version 550.54.14, but noted this is specific to custom kernel drivers and not relevant to standard hugetlbfs usage.

5. Rust/jemalloc/glibc and Hugepages

The fifth question addressed the practical mechanics of enabling hugepages in a Rust application. The research provided four strategies with different tradeoffs: jemalloc with THP (easiest, ~5% speedup with ~15% RSS increase), the jemalloc crate for Rust applications, glibc malloc with madvise(MADV_HUGEPAGE), and explicit hugepages via mmap with MAP_HUGETLB.

The recommendation for the C2 pipeline was explicit hugepages via mmap for the large allocations (bypassing the allocator entirely) combined with jemalloc THP for smaller heap allocations. This hybrid approach maximizes benefit while minimizing operational complexity.

Synthesis and Recommendations

The research culminated in seven actionable recommendations that synthesized findings across all five domains:

  1. Reserve 1 GiB hugepages at boot for bulk data structures
  2. Use mmap(MAP_HUGETLB | MAP_HUGE_1GB) for large allocations directly
  3. Set NPS1 (default) for simplicity, or NPS2 if thread pinning is possible
  4. Use numactl --interleave=all to spread allocation across all 8 memory channels
  5. For CUDA DMA: cudaHostRegister the hugepage-backed buffers
  6. Set system THP to madvise to avoid compaction jitter
  7. If using jemalloc, set MALLOC_CONF="thp:always,metadata_thp:always" for heap allocations These recommendations are notable for their specificity and practicality. They do not just say "use hugepages"—they specify which hugepage size, which allocation method, which NUMA setting, and which CUDA registration pattern. They also provide fallback options that allow the team to choose based on their specific constraints.

Impact on Phase 11

The hugepages research directly informed the Phase 11 optimization strategy. The three interventions already planned—bounding async_deallocation to a single thread, reducing the groth16_pool thread count, and adding a lightweight semaphore interlock—were designed to reduce DDR5 memory bandwidth contention. The hugepages research provided a complementary approach: reduce the bandwidth consumed by TLB misses, thereby freeing up bandwidth for the actual computation.

Crucially, the research also identified risks and mitigations. The THP pitfalls (RSS bloat, latency spikes) meant that the team should not simply enable THP system-wide. Instead, they should use explicit hugepages for the large allocations and THP in madvise mode for the rest. This nuanced recommendation prevented a potentially costly mistake.

Conclusion

The hugepages investigation represents a pivotal moment in the SUPRASEAL_C2 optimization journey. Faced with a memory bandwidth bottleneck that resisted purely architectural solutions, the team pivoted to a systems-level investigation that spanned the entire stack—from CPU microarchitecture (TLB coverage, NUMA topology) through the operating system (THP vs. explicit hugepages) to the application layer (Rust allocator, CUDA DMA). The result was a set of concrete, actionable recommendations that could be implemented without changing a single line of proof-generation code.

The investigation also exemplifies a productive pattern of human-AI collaboration in systems optimization: the human identifies a bottleneck and poses a research question; the AI conducts a thorough investigation across multiple technical domains; the human evaluates the findings and decides whether to implement. This pattern, repeated across the conversation, creates a cumulative understanding that neither human nor AI could achieve alone.

Whether the hugepages intervention yields the expected 5-30% improvement depends on the specifics of the workload and the system configuration. But the research has given the team the knowledge and confidence to try—and the nuanced understanding to avoid the pitfalls that have tripped up other large-scale deployments.## References

[1] "The Hugepages Research Mandate: A Deep Dive into Memory Bandwidth Optimization for Filecoin PoRep" — Analyzes the opening query of the subagent session, examining the reasoning behind each of the five research questions and the assumptions embedded in the investigation.

[2] "Drilling Down on Zen 4's TLB: The Critical Research Step in Hugepage Optimization for Filecoin Proof Generation" — Examines the subagent's deep-dive into Zen 4 TLB specifications from WikiChip and Chips and Cheese, providing the microarchitectural foundation for the hugepages hypothesis.

[3] "Drilling Down into the NUMA Topology of Threadripper PRO 7995WX: A Research Assistant's Targeted Deep-Dive" — Analyzes the subagent's research into the Storm Peak platform's NUMA topology, including the Phoronix NPS/SNC benchmarks.

[4] "Launching the Research Salvo: A Deep Dive into a Parallel Web Search Strategy for Hugepage Optimization" — Examines the subagent's initial parallel search strategy across all five research questions.

[5] "The Targeted Deep-Dive: How a Subagent Refined Its Research by Fetching Primary Sources" — Analyzes the subagent's iterative deepening strategy, fetching primary sources like AMD's Software Optimization Guide and NVIDIA developer forums.

[6] "Drilling Down: The Third Round of Targeted Searches in a Hugepage Performance Investigation" — Examines the subagent's third round of searches targeting specific benchmarks and production incident reports.

[7] "The Art of the Refined Query: How a Single Message Unlocked Deep Technical Research" — Analyzes the subagent's refined query strategy for obtaining specific TLB and NUMA topology information.

[8] "The Hugepages Investigation: Unlocking DDR5 Bandwidth for a 200-GiB Proof Generation Pipeline" — The comprehensive research report produced by the subagent, synthesizing findings across all five domains into actionable recommendations.