The Full Engineering Cycle: Testing, Deploying, and Validating a Budget-Integrated Pinned Memory Pool

Introduction

In the lifecycle of any significant systems engineering project, there is a critical phase that separates a well-designed implementation from a production-ready system. It is the phase where unit tests are written, operational visibility is added, binaries are built and deployed, and the entire system is validated under real workload. This article chronicles that phase for one of the most critical subsystems in the CuZK zero-knowledge proving engine: the budget-integrated pinned memory pool.

The story begins with a user's three-part request in <msg id=4247>: "Can we unit-test the behaviors? Then deploy on one then more vast nodes. Ideally vast-manager UI would get additional awareness for any internals (pinned pools etc)." This single sentence — testing, staged deployment, and operational visibility — defined the agenda for the entire segment. What followed was a masterclass in disciplined engineering: a carefully reasoned testability refactoring, comprehensive unit and integration tests, UI enhancements, Docker builds, binary extraction, remote deployment, process lifecycle management, and finally, production validation with real SnapDeals proofs.

This article traces the full arc of that work, examining the reasoning behind each decision, the challenges encountered, and the evidence that ultimately confirmed the design was sound. The budget-integrated pinned memory pool, validated on an RTX 5090 test machine with 5 successful proofs, zero failures, and a throughput of ~46 proofs/hour, represents a significant architectural improvement to the CuZK proving engine — replacing guesswork with accounting, caps with natural backpressure, and crashes with graceful budget enforcement.

The Problem That Demanded a Principled Solution

The CuZK proving engine is a GPU-accelerated system for generating Filecoin proofs — the complex zero-knowledge proofs required for the WinningPoSt, WindowPoSt, and SnapDeals proof types. These proofs demand enormous amounts of pinned (page-locked) host memory for efficient GPU transfers. Without pinned memory, GPU-to-CPU transfers suffer from poor bandwidth, directly impacting proof generation throughput.

The system had a PinnedPool component that managed these allocations via CUDA's cudaHostAlloc. But there was a fundamental problem: the pinned pool operated outside the MemoryBudget system. The budget tracked allocations for SRS (Structured Reference String) parameters, PCE (Pre-Compiled Constraint Evaluator) data, and working set memory — but the pinned pool was invisible to it. This led to a recurring failure pattern: the system would allocate pinned buffers up to some hard-coded cap, while the budget believed it had ample free space, and then the Linux OOM killer would terminate the process.

An earlier attempted fix had used arbitrary byte caps — limiting the pool to 40% of the budget for machines under 500 GiB. The assistant had candidly described this as "unprincipled" in <msg id=4246>. The user had rejected this approach. The correct solution, designed and implemented in the preceding segments, was to integrate the pinned pool directly with the memory budget: every allocation would go through budget.try_acquire(), every deallocation would call budget.release_internal(), and the pool would naturally grow only as far as the budget allowed — no caps, no heuristics, just accounting.

But a design is only as good as its validation. The user's three-part request pushed the assistant to go further: not just implement the design, but test it thoroughly, make it visible in the monitoring UI, and deploy it to production.

The Testability Challenge: Mocking CUDA Without a GPU

The central technical challenge was that PinnedPool depends on CUDA runtime APIs (cudaHostAlloc, cudaFreeHost) that are unavailable in a standard test environment. In a CI runner or developer's laptop, these symbols cannot be linked. The assistant's reasoning in <msg id=4248> reveals a careful exploration of alternatives.

The assistant considered making the allocator configurable via function pointers, adding a new_with_allocator() constructor, using conditional compilation, or refactoring to use a generic type parameter. The reasoning shows a back-and-forth deliberation: "I'm going back and forth on the best approach here — using conditional compilation is messy, making it generic changes the API, but adding a test constructor or refactoring to use function pointers could work cleanly."

The final solution was elegant: refactor the CUDA FFI declarations behind a #[cfg(not(test))] module boundary, providing mock implementations using standard Rust Box<[u8]> allocations during testing. This approach kept the production code path entirely unchanged while enabling comprehensive testing without any GPU hardware. A subtle design issue emerged: cuda_host_free originally took only a pointer, but the test mock needed the allocation size to properly deallocate memory. The assistant traced through all call sites — free_buffer(), shrink(), and the Drop implementation — and confirmed they all had access to the buffer size, so the function signature could be updated to accept both pointer and size without breaking anything.

The Tests: Encoding Behavioral Contracts

The assistant dispatched two parallel subagent tasks in <msg id=4253>, recognizing that the test refactoring and the UI updates were independent workstreams. The test-writing subagent produced 11 new unit tests in pinned_pool.rs and 3 integration tests in memory.rs.

The unit tests covered the core behaviors that the budget integration needed to guarantee:

Operational Visibility: Making the Invisible Visible

While the tests validated correctness, the UI update made the system observable. The second subagent task updated the vast-manager web dashboard — a single-file HTML application with embedded JavaScript — to display pinned pool statistics and a stacked memory budget breakdown bar.

The pinned pool statistics panel shows three key metrics: free buffers (buffers in the pool's free list, available for immediate checkout), live buffers (buffers currently checked out to active syntheses), and total bytes (the sum of all buffer allocations tracked by the pool). These numbers provide instant insight into pool health: if live buffers are high and free buffers are low, the system is under memory pressure; if total bytes approaches the budget limit, new allocations will trigger the fallback path.

The memory budget breakdown bar is a horizontal stacked bar with colored segments for each budget consumer: SRS parameters (blue), PCE data (green), pinned pool (orange), working set (yellow), and free memory (gray). This transforms an abstract accounting system into a visual tool that operators can use to understand memory pressure at a glance. If the pinned pool segment grows unexpectedly, or the free segment shrinks to zero, the operator knows immediately that the system is approaching its limits.

The UI changes added 74 lines to ui.html and required corresponding updates to the Go backend's status API. The assistant verified the changes compiled and the tests passed before proceeding to deployment.

The Build and Deployment Pipeline

With tests passing and UI updated, the assistant proceeded to build and deploy. A full Docker image (theuser/curio-cuzk:latest) was built using Docker BuildKit and pushed to Docker Hub in <msg id=4258-4259>. The vast-manager Go binary was rebuilt and deployed to the management host at 10.1.2.104 via a carefully orchestrated SSH sequence: stop the service, copy the binary, restart it, and verify with systemctl status in <msg id=4260-4262>. The first attempt encountered a "Text file busy" error — a classic Unix pitfall where you cannot overwrite a running binary — which the assistant correctly diagnosed and resolved by stopping the service before copying.

The assistant then surveyed the production fleet by querying the dashboard API in <msg id=4263>, revealing three running instances: an RTX 4090 (1008 GB RAM), an A40 (1425 GB RAM), and the RTX 5090 test machine (537 GB RAM). The RTX 5090 was selected as the first deployment target — the same machine used throughout development for validation.

Rather than pulling the full Docker image on the remote machine (which would be slow and bandwidth-intensive), the assistant used a minimal "rebuild" Dockerfile that compiled only the Rust code and extracted the 27 MB binary via docker create and docker cp in <msg id=4266-4268>. The first extraction attempt failed with "no command specified" — the rebuild image lacked a default CMD — but the assistant recovered gracefully by adding /bin/true as the container command. The binary was then copied to the remote machine via SCP in <msg id=4270>.

The Cutover: Killing the Old Process and Waiting for Memory

Before starting the new binary, the assistant checked the status of the running system in <msg id=4271>. The output was revealing: the budget showed 389.9 GiB used out of 400 GiB, but the pinned pool reported 0.0 GiB. This was exactly the problem being fixed — the old pool's memory was invisible to the budget. The assistant noted: "The current binary doesn't report pool stats (it's running the old non-budget-integrated pool). Budget shows 389.9/400 GiB used — the pool memory is invisible, which is exactly the problem we're fixing."

The old process was killed with kill $(pgrep -f cuzk-pitune4) in <msg id=4272>. But when the assistant checked whether the process was truly gone in <msg id=4273>, it found a zombie: 189945 [cuzk-pitune4] <defunct>. The process had exited but its parent had not yet reaped it. More importantly, the ~400 GiB of pinned memory allocated by the old process needed to be freed — a process that can take 90-120 seconds as the CUDA driver unpins memory pages and the kernel releases physical RAM.

The assistant made a deliberate decision to wait 100 seconds in <msg id=4274>, then verified that memory had been released in <msg id=4275>: the system showed 526 GiB free out of 755 GiB total, with no cuzk process running. Only then did the assistant start the new binary in <msg id=4276>:

ssh -p 40612 root@141.0.85.211 'chmod +x /data/cuzk-budget-pool && nohup /data/cuzk-budget-pool \
  --config /tmp/cuzk-memtest-config.toml > /tmp/cuzk-budget-pool.log 2>&1 & echo "started PID: $!"'

The response confirmed: started PID: 678843.

The Verification That Closed the Loop

The SSH command had timed out after 15 seconds (the bash tool's timeout), so the assistant needed independent verification. In <msg id=4277>, it ran a targeted pgrep:

ssh -p 40612 root@141.0.85.211 'pgrep -a cuzk-budget'
678845 /data/cuzk-budget-pool --config /tmp/cuzk-memtest-config.toml

This confirmed three things simultaneously: the process was alive (PID 678845), it was the right binary (/data/cuzk-budget-pool), and it had loaded the correct configuration (--config /tmp/cuzk-memtest-config.toml). The budget-integrated pinned memory pool was running in production.

The assistant then checked the startup logs in <msg id=4278>, confirming a clean start: "cuzk-daemon starting," "configuration loaded," "rayon global thread pool configured." The startup log contained the telltale signs of success: pinned pool: created (budget-integrated, no cap) and CUDA pinned memory pool initialized (budget-integrated). The status API confirmed a clean baseline: Budget: 0.0/400 GiB, Pool: 0.0 GiB, 0 live, 0 free, and zero active synthesis. The deployment was complete.

The Diagnostic Pause: When Complexity Fails

After the clean start was confirmed, the assistant submitted a SnapDeals proof to exercise the full pipeline. The system began ramping up, and the assistant monitored it closely. But at one point, a compound diagnostic command produced confusing output. The assistant ran:

ssh -p 40612 root@[host] 'free -h | head -3; echo "---"; cat /proc/$(pgrep -f cuzk-budget)/status | grep -i "vmrss\|vmsize"'

The output was alarming. Every cat command failed — PIDs were reported as directories or nonexistent. This was deeply confusing: the system was clearly running, proofs were completing, the status API was responding, yet the process query was returning nonsensical results.

The assistant's response was a masterclass in diagnostic decomposition. Instead of adding more complexity to the command, it stripped everything away and asked the simplest possible question:

ssh -p 40612 root@[host] 'pgrep -f cuzk-budget'

The output was clean and unambiguous: two PIDs, 678843 and 678845. The process was running. The confusing output from the previous command was a diagnostic artifact, not a system failure. The compound command had a subtle bash interaction: when pgrep returns multiple PIDs, the command substitution $(...) captures them with newlines, and when those newlines are embedded in a path like /proc/$(...)/status, they create malformed arguments. The first PID becomes part of /proc/PID (a directory, not a file), and the second PID becomes a bare PID/status (a relative path without /proc/).

This moment reveals a debugging philosophy worth articulating: when faced with confusing output from a complex command, the correct response is not to add more complexity — it's to decompose the command into its simplest components and test each one independently. The assistant could have written a more complex command to handle multiple PIDs, but that would have been premature optimization. The first question — "is the process running?" — was unanswered. By asking and answering that simple question first, the assistant avoided adding unnecessary complexity and got immediate clarity.

Watching the Pool Grow: Production Validation Under Load

With the process confirmed healthy, the assistant turned to the real validation: watching the budget-integrated pinned pool behave under genuine SnapDeals load. Over the next several minutes, the status API was queried repeatedly, each response revealing a system that was working exactly as designed.

The First 30 Seconds

After 30 seconds, the pool had grown to 58 GiB with 24 live buffers and 0 free. The assistant correctly inferred that 24 buffers represented 8 partitions × 3 buffers each (the a/b/c buffers for FFT outputs in SnapDeals synthesis), and that "0 free" meant all buffers were checked out to active syntheses. The budget showed 382.7/400 GiB used, with only 17.3 GiB available. This was the first evidence that the budget was correctly tracking pool memory. The assistant noted: "natural backpressure, the budget limits concurrency" — exactly the design goal.

The Early Release Mechanism

One of the key optimizations in the new design is the "early a/b/c budget release." During SnapDeals proof synthesis, each partition reserves budget for three large buffers. When the pinned pool covers these buffers, the partition reservation can be released early, freeing budget for other work. The assistant confirmed this was working by grepping the logs: 7 early releases had occurred early on, each releasing approximately 8 GiB. By the end of the validation period, this counter had grown to 29 — meaning the early release mechanism fired 29 times to free partition reservations, keeping the pipeline flowing.

Buffer Reuse and Efficiency

After another 60 seconds, the pool had grown to 152 GiB with 63 live buffers and 42 free. Two proofs had completed with zero failures. The assistant then checked the allocation counters: 65 new buffer allocations, 73 reuses, and 14 fallbacks to unpinned. The fact that reuses exceeded new allocations was a strong signal that the pool was working efficiently — buffers were being recycled rather than freshly allocated for every synthesis. The 14 fallbacks to unpinned memory occurred only during the very first syntheses, before the pool had "learned" the required buffer sizes through capacity hints. After that, the pool's free list was populated and reuse dominated.

Budget-Full Backpressure

The assistant also counted 7 "budget-full" events — instances where the pool could not allocate a new pinned buffer because the budget was exhausted. In the old system, this would have been a crash. In the new system, it was natural backpressure: the allocation was simply refused, and the system either reused an existing buffer or fell back to unpinned memory. No crashes, no OOM kills, no data loss. The budget-full events were evidence that the design's core mechanism — letting the budget govern pool growth — was working exactly as intended.

System-Level Validation

The assistant also validated at the OS level. The host's physical memory was checked via free -h, showing 755 GiB total with 280 GiB RSS used by the cuzk process. The budget reported 255 GiB used — closely tracking actual RSS. This was the critical validation that the budget accounting was accurate: the pool's internal tracking matched what the OS saw at the process level. The 280 GiB RSS was comfortably within the host's 755 GiB limit, with ample headroom.

The Declaration of Success

After 6.5 minutes of uptime, the assistant checked the final counters: 5 proofs completed, 0 failed. The pool had grown to 181 GiB with 75 buffers (27 free for reuse). The throughput calculation — 5 proofs in 6.5 minutes ≈ 46 proofs/hour — confirmed that the new pool was not only safe but performant.

The assistant's declaration was understated but definitive:

5 proofs in 6.5 minutes = ~46 proofs/hour — good throughput. 0 failures. Pool at 181 GiB (75 buffers, 27 free for reuse). The deployment is working well.

Each metric in that summary answers a specific question:

The Design Philosophy Validated

The success of this deployment validates a core design principle that guided the entire effort: eliminate arbitrary caps and let the memory budget naturally govern pool growth. The old system used a hard capacity cap — a brittle guess at how much pinned memory was safe. The new system has no cap at all, because it doesn't need one. The budget itself provides the upper bound. When the budget is full, new allocations are simply refused, and the system adapts by reusing existing buffers or falling back to unpinned memory.

This principle extends beyond the pinned pool. The same budget system governs SRS parameters, PCE (Pre-Compiled Evaluator) data, working sets, and now pinned buffers. All memory consumers are charged against the same global budget, creating a unified view of memory utilization. The budget becomes a single source of truth for memory accounting, eliminating the blind spots that caused the original OOM crashes.

The validation also confirmed that the design's specific mechanisms work correctly:

Lessons in Engineering Discipline

This segment of the opencode session offers several lessons for production engineering:

1. Testability is a design property, not an afterthought. The assistant recognized that the pinned pool's CUDA dependency made it untestable and invested in a #[cfg(test)] mock allocator before writing tests. This refactoring was the foundation for all subsequent validation.

2. Verification at every step. The assistant did not assume success. It checked test output, checked cargo check, checked binary size with ls -lh, checked process existence with pgrep, checked memory with free -h, and checked startup logs. Each verification step caught a potential failure mode before it could propagate.

3. Staged deployment with risk awareness. The assistant deployed to the test machine first, not the production instances. It waited 100 seconds for memory to drain rather than starting the new binary immediately. It verified preconditions before acting.

4. Parallel execution when possible. The test refactoring and UI updates were dispatched as parallel subagent tasks, recognizing their independence. The Docker push and Go binary build ran concurrently. This maximized throughput without sacrificing quality.

5. Graceful error recovery. The "Text file busy" error and the "no command specified" Docker error were both handled calmly and correctly. The assistant did not panic or retry blindly — it diagnosed the root cause and adjusted the approach.

6. Decompose complexity when debugging. When a compound diagnostic command produced confusing output, the assistant stripped it down to the simplest possible question — "is the process running?" — and got immediate clarity. This principle applies broadly: when faced with confusing output, reduce complexity before adding more.

Conclusion

The budget-integrated pinned memory pool represents the full engineering cycle: from problem diagnosis through design, implementation, testing, operational visibility, deployment, and production validation. The segment covered in this article captures the critical transition from "it works in tests" to "it works in production."

The assistant's methodical approach to testability, verification, staged deployment, and error recovery transformed a principled design into a running, validated system. The 11 unit tests and 3 integration tests provided confidence that the budget integration was correct. The UI enhancements made the system observable. The careful deployment protocol — kill, wait, verify, start — ensured a clean transition. And the production validation under real SnapDeals load confirmed every aspect of the design: budget tracking, early release, buffer reuse, graceful fallback, and natural backpressure.

In the end, the pool grew organically to 181 GiB, the budget tracked it accurately, 29 early releases freed partition reservations, 73 buffer reuses dominated 65 new allocations, 7 budget-full events provided natural backpressure without crashing, and 5 proofs completed at ~46 proofs/hour with zero failures. This was the ultimate validation that arbitrary caps had been replaced by natural, budget-governed self-regulation.

For anyone following the CuZK project's development, this segment marks the moment when the budget-integrated pinned memory pool transitioned from "should work" to "does work" — and the assistant, true to form, marked the occasion not with fanfare, but with a grep command, a few lines of JSON, and a quiet update to the todo list.