From Deployment Firefighting to Systematic OOM Prevention: The Full Arc of Production Engineering in GPU Proving Infrastructure

Introduction

In the span of a single coding session segment, an AI assistant and its human collaborator transformed a fragile, crash-prone GPU proving deployment into a robust, self-monitoring infrastructure. Segment 28 of this opencode conversation—spanning over 150 messages from initial configuration fixes through the design, implementation, and production deployment of a comprehensive memory safety system—captures the full arc of production engineering. The work traverses four distinct phases: the discovery and correction of critical configuration errors found during live testing, the restructuring of benchmark methodology to eliminate measurement artifacts, the diagnosis and resolution of a cluster-wide SSH meltdown, and the architectural response to systematic out-of-memory (OOM) failures through a full-stack memory awareness system.

What emerges is a case study in how distributed systems are hardened not through grand architectural redesigns but through the accumulation of small, correct decisions, each grounded in observation and verified against real hardware. The segment demonstrates that reliability is not a feature you add at the end—it is a property that emerges from the disciplined application of observation-driven engineering across every layer of the stack.

Part I: The Deployment Firefight — Four Critical Configuration Fixes

The segment opens with the assistant facing four issues discovered during live deployment testing of the CuZK proving daemon on vast.ai GPU instances. These were not theoretical problems—they were concrete failures observed on machines running real Filecoin proofs, and each required immediate correction before the system could be considered production-ready.

Fix 1: Synthesis Concurrency. The synthesis_concurrency parameter, which controls how many proof synthesis tasks run concurrently on the GPU pipeline, was defaulting to 4. Through extensive empirical tuning in earlier segments, the assistant had established that 18 concurrent syntheses was the sweet spot for DDR5 systems with 64 cores—any lower left the GPU underutilized, any higher caused contention. The fix was applied to both run.sh (the production daemon startup script) and benchmark.sh (the benchmarking script), ensuring consistency across operational modes [1].

Fix 2: Benchmark Concurrency Floor. The entrypoint.sh script calculated benchmark concurrency by dividing available memory by per-proof requirements times a safety factor. On 251 GB machines, this calculation produced a value of 3—too low to saturate the GPU pipeline and produce meaningful throughput measurements. The assistant added a floor of 4, a pragmatic override that ensured even memory-constrained machines would run enough concurrent proofs for valid measurement [1].

Fix 3: Missing Status Listener. The benchmark script's daemon configuration template omitted the status_listen directive, meaning the HTTP status API—which powers the vast-manager's real-time pipeline visualization—was unavailable during benchmarks. Operators were flying blind during the very activity designed to validate performance. The fix added status_listen = "0.0.0.0:9821" to the template, ensuring the status endpoint was active during both benchmark and production modes [1].

Fix 4: ANSI Escape Code Stripping. The vast-manager web UI was rendering raw ANSI escape codes in its log displays, making logs unreadable. The assistant added a stripAnsi() JavaScript helper function and applied it in both renderInstanceLogs and renderManagerLogs before the existing HTML-escaping step, cleaning up the display without breaking existing formatting [1].

These four fixes, applied across run.sh, benchmark.sh, entrypoint.sh, and ui.html, represent the difference between a system that works in development and one that works in production. Each fix was small—a number change, a floor clamp, a config key addition, a regex substitution—but collectively they addressed the gap between architectural design and operational reality. The Docker image was rebuilt and pushed after each set of changes, ensuring that the fixes were immediately available for deployment.

Part II: Restructuring the Benchmark — The Three-Phase Model

With the four fixes applied and a new Docker image deployed, the user raised a deeper concern about measurement methodology. The existing benchmark design involved starting the daemon in a special "warmup mode" with reduced concurrency for PCE (Pre-Compiled Constraint Evaluator) extraction, then restarting it with full production settings for the actual measurement. The restart forced the daemon to reload multi-gigabyte SRS files, adding minutes of overhead that dominated total benchmark time [1].

The user's insight was concise and transformative: "No restart after PCE warmup, for actual bench round let's do 5 proofs dispatched for warmup, 10 timed for throughput measurement, 3 cooldown." This three-phase model—warmup, timed, cooldown—eliminated the expensive daemon restart and introduced a rigorous measurement methodology that accounted for pipeline cold-start behavior and drain effects [1].

The assistant's implementation of this model required careful restructuring of benchmark.sh. The warmup mode was removed entirely; the daemon now starts once with full settings and runs all 18 proofs (5+10+3) in a single session. The timed phase captures only the middle 10 proofs, discarding both the ramp-up and ramp-down periods. This approach ensures that the measured throughput reflects steady-state behavior, not transient pipeline dynamics. The entrypoint.sh was updated to set BENCH_PROOFS=10 (the timed count) instead of the previous total of 12, and a new Docker image was built and pushed [1].

The three-phase model is a textbook example of how measurement methodology must evolve to match system behavior. A naive benchmark that measures from first proof to last includes both the cold-start penalty (SRS loading, GPU warmup) and the cooldown period (draining in-flight proofs), neither of which reflects the system's sustainable throughput. By explicitly accounting for these phases, the assistant and user created a benchmark that produces meaningful, reproducible results—results that can be trusted to guide configuration decisions.

Part III: The SSH Crisis — Exit Status 255 Across All Nodes

Just as the benchmark restructuring was completing, the user reported a catastrophic failure: the vast-manager's SSH connections to all remote GPU instances had stopped working, returning the cryptic error "cuzk: ssh exec failed: exit status 255" for every node [1].

Exit code 255 is SSH's universal "I couldn't even start the connection" error, distinct from command failure (exit code 1) or authentication errors (which SSH typically reports with specific messages). The fact that all nodes were failing simultaneously immediately ruled out per-instance problems and pointed to a systemic issue on the manager side.

The assistant's diagnostic approach was methodical. Rather than guessing at the root cause, it first examined the Go code in main.go to understand exactly how the SSH proxy worked. This revealed a critical flaw: the code used cmd.Output(), which captures only stdout. SSH's error messages—the very evidence needed to diagnose exit code 255—were being written to stderr and silently discarded. All the operator could see was the exit code, with no context about why the connection failed [1].

The fix was twofold. First, the Go code was modified to capture SSH stderr and include it in error messages, so future failures would be self-diagnosing. Second, a retry mechanism was added: on exit code 255, the code removes the stale ControlMaster socket and retries once, handling the common case where a dead master connection leaves a stale socket file that blocks new connections [1].

But the stderr capture revealed the true root cause: the vast-manager host's SSH public key had never been added to the vast.ai instances' authorized_keys files. And when the assistant attempted to add it, a second bug emerged—the original authorized_keys file lacked a trailing newline, causing the new key to be concatenated onto the same line as the existing key, producing a malformed entry that SSH rejected [1].

The assistant fixed this on four running instances, using printf with explicit newlines to ensure each key occupied its own line. The new vast-manager binary was built, copied to the manager host via SCP, and deployed via systemd restart. Verification confirmed that SSH connectivity was restored across all nodes [1].

The SSH debugging session is a textbook example of systematic diagnosis. Faced with a cryptic error code affecting all nodes, the assistant resisted the urge to guess. Instead, it examined the code to understand the mechanism, identified the stderr capture gap, fixed the diagnostic infrastructure first, and then used the improved diagnostics to find the root cause. This approach—understand before changing, fix diagnosis before fixing symptoms—is the essence of professional debugging.

Part IV: The Memcheck System — From Reactive Firefighting to Proactive Prevention

With SSH connectivity restored, the user reported a deeper problem: CuZK was being OOM-killed on production nodes, including the largest 256 GB machines. Benchmarks would pass, but production loads consumed all available memory and triggered the kernel's OOM killer. The user's request was specific and architecturally ambitious: build a utility that "understands memory constraints, including cgroups (docker) very well," understands pinning constraints, reports to vast-manager, and surfaces data in the UI [1][2].

The Root Cause

The assistant's investigation confirmed that CuZK's detect_system_memory() function read total RAM from /proc/meminfo, which reports the host's physical memory. Inside Docker containers, this value is meaningless—the real constraint is the cgroup memory limit, which is invisible to /proc/meminfo. A 256 GB machine running a Docker container with a 64 GB limit would appear to CuZK as having 256 GB available, causing it to allocate far beyond what the container could provide. The kernel would then enforce the cgroup limit, killing the process with a SIGKILL when it tried to allocate more memory than the container was allowed [1][2].

The Utility Design

The assistant's response was a comprehensive planning session that produced the memcheck.sh utility—a cgroup-aware memory analysis tool that became the cornerstone of OOM prevention across the proving fleet. The script detects cgroup v1 and v2 memory limits by reading /sys/fs/cgroup/memory/memory.limit_in_bytes (v1) or /sys/fs/cgroup/memory.max (v2), checks RLIMIT_MEMLOCK for pinning capability via ulimit -l, gathers GPU information via nvidia-smi, and calculates safe concurrency levels. The effective memory limit is computed as min(MemTotal, cgroup_limit)—taking the minimum of the host-reported value and the container-imposed constraint [1][2].

Full-Stack Integration

The memcheck system was not just a shell script. The assistant built a complete feedback loop spanning four architectural layers, each depending on the one before it [2]:

Layer 1: Data Collection (Shell Script). The memcheck.sh script runs on each instance, reading cgroup files and GPU information to produce a structured JSON report. Without this layer, there is no data.

Layer 2: Data Transmission and Storage (Go API + SQLite). The vast-manager's POST /memcheck endpoint receives the report and stores it in the database alongside the instance record. The DashboardInstance struct was extended with MemcheckJSON and MemcheckAt fields, and the dashboard query was updated to load memcheck data alongside instance records. Without this layer, the data cannot be stored or retrieved.

Layer 3: Data Visualization (HTML/JS Dashboard). The renderMemcheck function displays the report in the instance detail view, giving operators visibility into memory constraints. The assistant added the memcheck panel HTML to the instance detail view, implemented the renderMemcheck JavaScript function with appropriate CSS styling, and ensured the panel had a distinct visual treatment. Without this layer, operators cannot see what the system decided.

Layer 4: Data-Driven Configuration (Entrypoint Integration). The BUDGET and BENCH_CONCURRENCY variables are set based on the memcheck report, dynamically adapting the proving engine's behavior to the actual container limits. The entrypoint script runs memcheck immediately after instance registration, POSTs the results to the manager, and uses the reported cgroup-aware memory limits to configure the proving pipeline. Without this layer, the data never influences runtime behavior [2].

The assistant built these layers in dependency order, ensuring that each layer was complete before moving to the next. This disciplined approach prevented the common pitfall of building a tool that produces data but has no mechanism to consume it.

The Entrypoint Integration

The critical integration point is the container entrypoint script (entrypoint.sh), which orchestrates the entire lifecycle of each proving instance—from registration with the vast-manager, through parameter fetching and benchmarking, to the production supervisor loop. The assistant made four key changes [2]:

Run memcheck.sh after registration, POST to manager: The memcheck invocation is inserted early in the lifecycle, immediately after the instance registers with the vast-manager. This timing ensures that the instance's identity is established in the management database before the memcheck results are POSTed, and that the benchmark phase runs with memory-aware concurrency settings.
Use memcheck results to set correct BUDGET and BENCH_CONCURRENCY: This is the heart of the integration. By deriving these values from the cgroup-aware memory limit rather than the host's /proc/meminfo, the system automatically adapts to whatever container configuration the vast.ai rental provides. A 64 GB container gets different settings than a 128 GB container, without any manual tuning.
Add CURIO_NODE_NAME: Setting this environment variable to the container's hostname makes each proving instance identifiable in the Curio cluster management interface. Without it, operators would see anonymous nodes and struggle to correlate on-chain activity with specific instances.
Trust the existing supervisor loop: The assistant recognized that the existing supervisor loop already managed restarting cuzk and curio daemons after crashes. Adding another restart mechanism would be redundant and potentially conflicting. The correct approach was to fix the inputs to the daemon so that crashes happened less frequently in the first place. This fourth point is a particularly insightful piece of "negative work"—recognizing what not to change. The assistant could have added a new restart wrapper around the benchmark phase, but that would have introduced a second restart pathway with different timing and error semantics, creating a maintenance burden and potential race conditions. Instead, the assistant trusted the existing infrastructure and focused on preventing the conditions that triggered crashes [2].

Packaging and Deployment

With the code changes complete, the assistant faced a critical packaging problem: the memcheck.sh script existed on disk but was not included in the Docker image's runtime layer. Without this step, the entrypoint integration would fail at runtime because the script would not be available inside the container [2].

The assistant added COPY docker/cuzk/memcheck.sh /usr/local/bin/memcheck.sh to the Dockerfile and updated the chmod layer to make the new script executable. The Docker build succeeded, confirming that the Dockerfile edit was syntactically correct and all referenced files existed [2].

The deployment was completed in two steps: the Docker image was pushed to the registry with docker push theuser/curio-cuzk:latest, and the updated vast-manager binary was deployed to the production server via SCP and systemd restart. The service started successfully, confirming that the Go backend changes were compatible with the production environment. The memcheck system was now fully operational across the entire stack [2].

Part V: Themes and Engineering Philosophy

Several themes recur throughout this segment, revealing the engineering philosophy that guided the work.

Observation-Driven Development

Every fix in this segment was grounded in observation from live deployment testing. The synthesis concurrency value of 18 was not guessed—it was discovered empirically through earlier benchmarking. The three-phase benchmark model emerged from watching the system's actual warmup behavior. The SSH key concatenation bug was found by examining the actual authorized_keys file on running instances. The memcheck system was a direct response to observed OOM kills. This pattern—observe, hypothesize, test, implement—is the hallmark of disciplined engineering.

The Power of Small Fixes

The most impactful changes in this segment were often the smallest. A floor of 4, a missing config key, a regex substitution, a single environment variable. None of these changes would be noteworthy in isolation, but together they transformed a system from barely functional to production-ready. This is a reminder that in complex systems, reliability is achieved not through grand architectural changes but through the accumulation of small, correct decisions, each verified against real hardware.

Systematic Debugging

Full-Stack Thinking

The memcheck system exemplifies full-stack engineering. The assistant did not stop at writing a shell script. It designed an API endpoint, a database schema, a UI component, and an integration point in the entrypoint script. This end-to-end thinking ensured that the utility's output would be visible, persistent, and actionable—not buried in a log file that no one reads. Each layer was built in dependency order, ensuring that the system was complete and functional at every stage [2].

The Art of Negative Work

One of the most subtle skills demonstrated in this segment is the ability to recognize what not to change. When integrating memcheck into the entrypoint, the assistant recognized that the existing supervisor loop already handled daemon restart, and adding a second restart mechanism would create maintenance burden and potential race conditions. This "negative work"—the deliberate decision to not add code—is a hallmark of experienced engineers who understand that every line of code is a liability.

Part VI: Coordination and Workflow

Beyond the technical fixes, this segment reveals a sophisticated pattern of human-AI collaboration. The assistant maintained a persistent todo list using the todowrite tool, updating it after each logical unit of work. This served multiple functions simultaneously: it externalized the assistant's working memory, communicated progress to the user, created an audit trail, and provided a recovery point if the conversation were interrupted [1][2].

The user's messages were equally strategic. The two-sentence "continue if you have next steps, or stop and ask for clarification" message at the beginning of the segment was a masterclass in delegation—authorizing execution while providing a safety valve for uncertainty. The three-phase benchmark proposal demonstrated how a single observation-driven insight could reshape an entire measurement methodology. The memcheck mandate was not just a bug report but an architectural specification, decomposing a complex problem into implementable components [1].

This pattern of interaction—the assistant proposing, the user validating, the assistant executing, the user observing—created a tight feedback loop that enabled rapid iteration. Each cycle of the loop produced new knowledge that informed the next cycle. The SSH debugging session, for example, started with a cryptic error code, progressed through code examination and stderr capture, and ultimately revealed both a missing SSH key and a concatenation bug—findings that would have been impossible to discover without the systematic approach.

Conclusion: The Architecture of Reliability

Segment 28 of this opencode session tells a story that is common in production engineering but rarely documented: the story of how a system goes from "works on my machine" to "works in production." The path is not a straight line. It involves configuration fixes that seem trivial in retrospect, measurement methodology that evolves as you understand the system's actual behavior, infrastructure failures that reveal hidden assumptions, and architectural responses that transform reactive firefighting into proactive prevention.

The CuZK proving system that emerged from this segment was fundamentally different from the one that entered it. The synthesis concurrency was tuned to match the hardware. The benchmark methodology accounted for pipeline dynamics. The SSH proxy was hardened with stderr capture and retry logic. The memory detection was made cgroup-aware. The dashboard was cleaned up. The processes were set to auto-restart. And crucially, a complete feedback loop was established: instances self-report their memory constraints, the management layer stores and displays the data, and the instances use that same data to configure themselves, all without human intervention.

In the end, reliability is not a feature you add—it is a property that emerges from the accumulation of correct decisions, each grounded in observation and verified against real systems. This segment of the opencode session is a testament to that principle, and a case study in how AI-assisted engineering can accelerate the journey from fragile to robust.

References

[1] "From Configuration Fixes to Systematic OOM Prevention: A Full-Stack Engineering Sprint in GPU Proving Infrastructure" — Chunk 0 article covering the four configuration fixes, three-phase benchmark restructuring, SSH crisis diagnosis, and memcheck design.

[2] "The Full-Stack Pipeline: How a Memory-Awareness System Was Wired from Shell Script to Production Deployment" — Chunk 1 article covering the UI integration, entrypoint wiring, Docker packaging, and production deployment of the memcheck system.