The Opaque Pointer: Tracing Memory Across the FFI Boundary in the CuZK SRS Investigation

The Message

At message index 3993 in this opencode session, the assistant executed a single, seemingly mundane action: it read a file. The file was /tmp/czk/extern/supraseal-c2/src/lib.rs, and the assistant displayed its contents verbatim:

[assistant] [read] /tmp/czk/extern/supraseal-c2/src/lib.rs
<path>/tmp/czk/extern/supraseal-c2/src/lib.rs</path>
<type>file</type>
<content>1: // Copyright Supranational LLC
2: 
3: sppark::cuda_error!();
4: 
5: use std::path::PathBuf;
6: 
7: #[repr(C)]
8: pub struct SRS {
9:     ptr: *const core::ffi::c_void,
10: }
11: 
12: impl Default for SRS {
13:     fn default() -> Self {
14:         Self {
15:             ptr: core::ptr::null(),
16:         }
17:     }
18: }
19: 
20: impl SRS {
21:     pub fn try_new(srs_path: PathBuf, cache: bool) -> Result<Self, ...

On its surface, this is a trivial read operation—the assistant pulled a source file into the conversation. But in the context of the debugging session unfolding around it, this message represents a critical turning point: the moment when the investigation into a persistent out-of-memory (OOM) crash crossed the boundary between Rust's managed memory world and the opaque, C++-controlled allocations happening behind an FFI interface. This single file read crystallized a hypothesis that would ultimately reshape the entire memory management strategy for the CuZK proving engine.

The Debugging Context: A Crash Under Memory Pressure

To understand why this file mattered, we must step back into the broader investigation. The team was running GPU-accelerated proof generation (PoRep 32G) on vast.ai cloud instances. These instances run inside Docker containers with cgroup memory limits—artificial ceilings that prevent a container from using more than its allocated share of the host's RAM. On one particular instance with a 342 GiB cgroup limit, the cuzk daemon was crashing with OOM kills (exit code 137, SIGKILL from the kernel OOM killer) during GPU processing. The system had a 10 GiB safety margin—the memory budget was set to 331 GiB, leaving 11 GiB of headroom below the 342 GiB cgroup limit—yet it was still dying.

The debugging had already gone through several phases. The team had identified that the CUDA pinned memory pool (PinnedPool) was operating partially outside the MemoryBudget accounting system: pinned buffers returned to the pool after GPU work were never freed from actual RSS, creating an accounting discrepancy. They had also identified that kernel/driver overhead (glibc arena fragmentation, page tables, GPU driver allocations) consumed real memory that the budget didn't account for. But the exact trigger for the crash remained elusive.

The User's Hint: "Maybe in SRS-mode it a bit weird"

The immediate predecessor to this message was a crucial hint from the user. In message 3987, the user said simply: "Maybe in SRS-mode it a bit weird." This was a domain-specific observation: the SRS (Structured Reference String) is a 44 GiB blob of elliptic curve points loaded as CUDA-pinned memory at startup. The user suspected that the SRS loading path had some accounting peculiarity—a "weirdness"—that might explain why the budget was insufficient.

The assistant took this hint and began tracing the SRS allocation chain. In message 3988, it read the SRS manager code (srs_manager.rs) to understand how the budget reservation worked. In message 3990, it reconstructed the flow:

The budget reserves 44 GiB (the file size of the .params file)
SuprasealParameters::new(path) loads the file, which internally calls cudaHostAlloc for the SRS data
The reservation is made permanent via into_permanent()
Eviction releases via budget.release_internal() The budget tracked 44 GiB for the SRS, and the actual RSS was also approximately 44 GiB of pinned memory. But the user's hint nagged: was the file size really equal to the actual allocation? What if SuprasealParameters::new() allocated more than the file size? What if there was overhead the Rust side couldn't see?

The FFI Wall: What This File Revealed

This is where message 3993 enters the story. The assistant read supraseal-c2/src/lib.rs—the Rust FFI binding for the C++ supraseal library. What it found was striking in its simplicity and opacity.

The SRS struct is a #[repr(C)] type containing a single field: ptr: *const core::ffi::c_void. This is a raw pointer to an opaque C++ object. The struct has a Default impl that sets the pointer to null, and a try_new method (partially shown) that calls an unsafe extern function create_SRS().

The critical insight is what this struct doesn't reveal. There is no field for the allocation size. There is no method to query how much memory was pinned. The actual cudaHostAlloc call happens deep inside the C++ supraseal library, behind the create_SRS() FFI function. From the Rust side, the SRS is just a pointer—a black box. The Rust memory manager has no way to verify that the actual pinned allocation matches the 44 GiB budget reservation derived from the file size.

This is a classic FFI information asymmetry. The Rust code reserves budget based on a proxy (file size), but the actual allocation is controlled by C++ code that the Rust memory manager cannot inspect. Any mismatch—whether from alignment padding, internal fragmentation, driver overhead, or additional allocations inside create_SRS()—would silently break the budget accounting.

Assumptions and Their Risks

The assistant and the team had been operating under several assumptions that this file read challenged:

Assumption 1: File size equals allocation size. The budget system used file_size as the reservation amount for the SRS. But the file on disk is a serialized representation; the in-memory representation might include additional metadata, alignment padding, or internal data structures. The FFI boundary made it impossible to verify this assumption from Rust code.

Assumption 2: The budget system has full visibility into memory. The entire memory management strategy was built on the premise that the MemoryBudget could track every byte. But the SRS allocation happened through an opaque C++ interface—the budget was tracking a proxy value, not the actual allocation.

Assumption 3: The safety margin covers all overhead. The 10 GiB safety margin was meant to absorb kernel overhead, fragmentation, and accounting inaccuracies. But if the SRS allocation itself was under-accounted by even a few percent, that would consume the margin entirely on a 44 GiB allocation.

The Thinking Process: Systematic Elimination

The assistant's reasoning in the messages leading up to 3993 shows a methodical debugging approach. In message 3986, the assistant walked through the entire memory flow: partition synthesis reserves 14 GiB, releases 13 GiB after GPU submission, the pinned pool recycles buffers without freeing RSS. It considered multiple hypotheses:

GPU memory exhaustion on the RTX PRO 4000's 24 GiB VRAM
The dual-worker interlock failing to prevent simultaneous GPU usage
The PI pacer over-saturating the GPU queue
Glibc arena overhead from 18 parallel synthesis threads
The SRS loading transient spike Each hypothesis was examined and either eliminated or deprioritized. The GPU VRAM hypothesis was discarded because a CUDA OOM would return an error code, not kill the process. The dual-worker hypothesis was noted but seemed unlikely given the Phase 8 interlock design. The user's hint about SRS-mode being "weird" then focused the investigation on the SRS loading path. The assistant traced through the code: srs_manager.rs → supraseal_params.rs → supraseal-c2/src/lib.rs. Each step moved deeper into the allocation chain, and each step revealed less visibility. The SRS manager had clean budget accounting. The SuprasealParameters wrapper was just a thin shim. And finally, the FFI binding revealed the raw pointer—the point where Rust's visibility ended.

Output Knowledge: A New Hypothesis

This message produced a crucial piece of knowledge: the SRS allocation was opaque to the Rust memory manager. The SRS struct was nothing more than a raw pointer. The actual memory allocation happened in C++ code that the Rust side could not instrument or verify.

This led to a new hypothesis: the budget system might be systematically under-accounting for the SRS. If create_SRS() allocated more pinned memory than the file size suggested—due to internal data structures, alignment requirements, or driver overhead—then the 44 GiB reservation was too small. On a system with only 11 GiB of headroom, even a 5% accounting error on the SRS (about 2.2 GiB) would consume 20% of the safety margin.

More broadly, this revealed a fundamental limitation of the memory management architecture: the MemoryBudget system could only track allocations it controlled. Any allocation made through an opaque FFI interface—whether the SRS, GPU driver allocations, or kernel page tables—was invisible and unaccounted. The safety margin was the only defense against these invisible allocations, and 10 GiB was proving insufficient.

The Broader Significance

This message, for all its surface-level simplicity, represents the moment when the debugging team recognized that their memory management system had a blind spot. The elegant Rust-side budget accounting—with its reservations, permanent allocations, and release tracking—could not reach across the FFI boundary into C++ code. The SRS, the single largest allocation in the system, was partially invisible to the very system designed to prevent OOM crashes.

This recognition would drive the subsequent solution: instead of trying to make the budget system more precise (which was impossible across the FFI boundary), the team built memprobe, a utility that empirically measured the true memory overhead by allocating until it neared the cgroup limit. They also added an OOM recovery loop that adaptively reduced the budget after crashes. These were pragmatic solutions that acknowledged the fundamental opacity of the FFI boundary rather than trying to eliminate it.

The raw pointer in supraseal-c2/src/lib.rs—that single *const c_void—was a wall. The debugging session had run into it, and the solution was not to tear it down, but to work around it with empirical measurement and adaptive recovery.