The Full-Stack Pipeline: How a Memory-Awareness System Was Wired from Shell Script to Production Deployment

Introduction

In the world of distributed GPU proving infrastructure, the gap between a diagnostic tool and an operational solution is measured not in lines of code but in integration layers. A shell script that detects memory limits is a curiosity. A Go API that stores those limits is a database. A dashboard panel that displays them is a visualization. But a system that acts on those limits — that dynamically configures itself to prevent out-of-memory (OOM) kills without human intervention — is an engineering achievement.

This article examines a single chunk of an opencode coding session (segment 28, chunk 1) in which the assistant completed the full-stack integration of the memcheck utility — a cgroup-aware memory analysis system designed to prevent OOM kills on GPU proving instances running inside Docker containers on vast.ai. The chunk spans 24 messages (indices 3847 through 3870) and covers the final integration of memcheck across four architectural layers: the dashboard UI, the container entrypoint script, the Docker image build, and the production deployment pipeline. By the end of the chunk, the memcheck system is fully operational: the shell script runs on instances, data flows through the API into SQLite, is surfaced in the dashboard UI, and directly influences runtime configuration to prevent OOM kills.

The Problem: When Memory Detection Lies

The root cause of the OOM problem was a subtle but devastating mismatch. The CuZK proving engine's detect_system_memory() function read total RAM from /proc/meminfo, which reports the host machine's physical memory. On a 256 GB vast.ai instance running a Docker container with a 64 GB memory limit, the engine would believe it had 256 GB available and configure concurrency accordingly. The kernel would then enforce the cgroup limit, killing the process with a SIGKILL when it tried to allocate more memory than the container was allowed.

The solution was memcheck.sh — a shell script that detects cgroup v1 and v2 memory limits by reading /sys/fs/cgroup/memory.max or /sys/fs/cgroup/memory/memory.limit_in_bytes, checks RLIMIT_MEMLOCK for GPU pinning capability, gathers GPU information via nvidia-smi, and calculates safe concurrency levels. But a script on disk is not a solution. It needed to be wired into every layer of the system.

Layer 1: The Dashboard UI — Making Memory Visible

The first task in this chunk was to surface memcheck data in the vast-manager dashboard. The assistant began with a reconnaissance grep across ui.html to find where instance details are rendered ([msg 3847]). The search patterns — renderInstances, cuzk-$, expandedUUID, instance-detail — reveal a deliberate strategy: find the main rendering function, locate an existing per-instance panel pattern to use as a template, understand the expansion/collapse state management, and identify where to insert the new panel relative to existing detail content.

This grep is a textbook example of informed editing. Rather than reading the entire HTML file, the assistant used targeted searches to build a mental model of the UI architecture. The results confirmed that renderInstances() at line 442 is the central function that builds the HTML for the instance table, that expandedUUID tracks which instance is expanded, and that the existing CUZK panel (<div id="cuzk-${inst.uuid}">) provides a structural template for the new memcheck panel.

The assistant then read the UI file to examine the detail grid rendering code (<msg id=3848-3849>), identifying the exact location where the memcheck panel should be inserted. The edit in [msg 3850] added the memcheck panel HTML to the instance detail view, and [msg 3851] added the renderMemcheck JavaScript function with appropriate CSS styling. The function parses the memcheck_json field from the instance object — which the Go backend had already been extended to include — and renders a structured panel showing memory limits, GPU information, pin check status, and recommended concurrency.

What makes this integration elegant is that the assistant did not need to modify the Go backend's API response format. The DashboardInstance struct had already been extended with MemcheckJSON and MemcheckAt fields in earlier messages (<msg id=3838-3844>), and the dashboard query already loaded memcheck data alongside instance records. The UI work was purely a presentation layer change — consuming data that was already flowing through the pipeline.

The CSS styling in [msg 3854] added the final polish, ensuring the memcheck panel had a distinct visual treatment that differentiated it from the surrounding detail content. With the UI complete, the assistant updated the todo list ([msg 3855]) to mark the task as "in_progress" and pivoted to the next layer.## Layer 2: The Entrypoint — Making Memory Actionable

The dashboard UI makes memcheck data visible to human operators, but visibility alone does not prevent OOM kills. The critical integration layer is the container entrypoint script (entrypoint.sh), which orchestrates the entire lifecycle of each proving instance — from registration with the vast-manager, through parameter fetching and benchmarking, to the production supervisor loop.

The assistant's approach to this integration is captured in [msg 3856], which enumerates four key changes:

Run memcheck.sh after registration, POST to manager: The memcheck invocation is inserted early in the lifecycle, immediately after the instance registers with the vast-manager. This timing ensures that the instance's identity is established in the management database before the memcheck results are POSTed, and that the benchmark phase — which follows shortly after — runs with memory-aware concurrency settings.
Use memcheck results to set correct BUDGET and BENCH_CONCURRENCY: This is the heart of the integration. The BUDGET variable controls the memory budget passed to the cuzk proving engine, and BENCH_CONCURRENCY controls how many parallel proofs the benchmark attempts. By deriving these values from the cgroup-aware memory limit rather than the host's /proc/meminfo, the system automatically adapts to whatever container configuration the vast.ai rental provides. A 64 GB container gets different settings than a 128 GB container, without any manual tuning.
Add CURIO_NODE_NAME: Setting this environment variable to the container's hostname makes each proving instance identifiable in the Curio cluster management interface. Without it, operators would see anonymous nodes and struggle to correlate on-chain activity with specific instances.
The supervisor loop already handles auto-restart: The assistant recognized that the existing supervisor loop at line 278 of entrypoint.sh already managed restarting cuzk and curio daemons after crashes. Adding another restart mechanism would be redundant and potentially conflicting. The correct approach was to fix the inputs to the daemon so that crashes happened less frequently in the first place. This fourth point is a particularly insightful piece of "negative work" — recognizing what not to change. The assistant could have added a new restart wrapper around the benchmark phase, but that would have introduced a second restart pathway with different timing and error semantics, creating a maintenance burden and potential race conditions. Instead, the assistant trusted the existing infrastructure and focused on preventing the conditions that triggered crashes. The subsequent messages (<msg id=3857-3860>) show the assistant methodically applying each change: updating the registration section to run memcheck and POST results, fixing the benchmark invocation to pass the budget, updating the production run.sh call, and adding CURIO_NODE_NAME. Each edit was verified by the tool's success response, and the todo list was updated in [msg 3861] to reflect the completed integration.

Layer 3: The Docker Image — Packaging the Solution

With the code changes complete, the assistant faced a critical packaging problem: the memcheck.sh script existed on disk but was not included in the Docker image's runtime layer. Without this step, the entrypoint integration would fail at runtime because the script would not be available inside the container.

The assistant discovered this gap in [msg 3862], immediately after the todo update. The Dockerfile read in [msg 3863] confirmed the existing pattern: runtime scripts like benchmark.sh, run.sh, and monitor.sh were copied to /usr/local/bin/ in the Docker image. The assistant added COPY docker/cuzk/memcheck.sh /usr/local/bin/memcheck.sh to the Dockerfile in [msg 3864], and updated the chmod layer to make the new script executable.

The Docker build in [msg 3866] shows the result. Layer #40 (COPY docker/cuzk/memcheck.sh) completed in 0.1 seconds — a deceptively fast step that represented the culmination of dozens of messages of design, coding, and integration work. The chmod layer (#41) was also updated to include the new script. The build succeeded, confirming that the Dockerfile edit was syntactically correct and all referenced files existed.

The assistant chose to build the Docker image and the vast-manager Go binary separately rather than in a multi-stage Docker build. The Go binary was compiled natively on the development machine (go build -o /tmp/czk/vast-manager . in [msg 3865]), while the Docker image was built with docker build. This two-step approach is pragmatic: the Go toolchain and its C dependencies are heavy and would bloat the Docker build cache. By compiling the binary outside Docker and deploying it via SCP, the assistant avoided rebuilding the entire Docker image for every Go code change.

Layer 4: Production Deployment — The Final Push

The deployment phase in <msg id=3867-3868> completed the end-to-end implementation. The Docker image was pushed to the registry with docker push theuser/curio-cuzk:latest, and the updated vast-manager binary was deployed to the production server via SCP and systemd restart.

The deployment of the vast-manager binary is particularly noteworthy. The assistant used scp to copy the binary to the production server and then restarted the systemd service. This pattern — build locally, deploy via SCP, restart via systemd — is a common but effective deployment strategy for single-server Go services. It avoids the complexity of containerizing the management service while still enabling rapid iteration.

The service started successfully, confirming that the Go backend changes (the new API endpoint, the SQLite column, the dashboard query updates) were compatible with the production environment. The memcheck system was now fully operational across the entire stack.

The Todo Checkpoint: Marking the Milestone

Throughout the chunk, the assistant used the todowrite tool to maintain a structured task list. Message [msg 3869] is the final todo update, marking all memcheck-related tasks as completed:

"Write memcheck.sh: cgroup-aware memory analysis, pin check, GPU info, JSON output" — completed
"Add vast-manager memcheck API endpoint + DB storage" — completed
"Add memcheck display to vast-manager UI" — completed
"Integrate memcheck into entrypoint.sh (run before bench/prod, POST to manager)" — completed This todo update is more than administrative overhead. It is a milestone marker that signals the transition from development to operational readiness. The assistant's disciplined use of structured task tracking creates an audit trail and keeps the development process transparent. Message [msg 3870] is an empty user message — a silence that speaks volumes. In the context of the tool-calling architecture, an empty message from the user signals that no new instructions are needed; the assistant is expected to continue with its autonomous workflow. This silence confirms that the memcheck integration was the intended next step and that the user had no corrections or additional requirements.

Architectural Insights: The Full-Stack Pattern

The memcheck integration exemplifies a pattern that appears repeatedly in distributed systems engineering: a diagnostic tool must be wired through every layer of the stack to become an operational solution. The pattern has four stages:

Data collection (shell script): The memcheck.sh script runs on each instance, reading cgroup files and GPU information to produce a structured JSON report.
Data transmission and storage (Go API + SQLite): The vast-manager's POST /memcheck endpoint receives the report and stores it in the database alongside the instance record.
Data visualization (HTML/JS dashboard): The renderMemcheck function displays the report in the instance detail view, giving operators visibility into memory constraints.
Data-driven configuration (entrypoint integration): The BUDGET and BENCH_CONCURRENCY variables are set based on the memcheck report, dynamically adapting the proving engine's behavior to the actual container limits. Each layer depends on the one before it. Without the shell script, there is no data. Without the API, the data cannot be stored or retrieved. Without the UI, operators cannot see what the system decided. Without the entrypoint integration, the data never influences runtime behavior. The assistant built these layers in dependency order, ensuring that each layer was complete before moving to the next.

Conclusion

This chunk of the opencode session demonstrates what it takes to turn a diagnostic tool into an operational system. The memcheck utility — a shell script that reads cgroup memory limits — was integrated across four architectural layers: a dashboard UI for visibility, an entrypoint script for automatic execution, a Docker image for packaging, and a production deployment pipeline for delivery. By the end of the chunk, the system was fully operational: instances self-report their memory constraints, the management layer stores and displays the data, and the instances use that same data to configure themselves, all without human intervention.

The work is a testament to the importance of integration in distributed systems. A shell script that detects memory limits is a curiosity. A Go API that stores those limits is a database. A dashboard panel that displays them is a visualization. But a system that acts on those limits — that dynamically configures itself to prevent OOM kills — is an engineering achievement. And that achievement was completed in this chunk, one layer at a time.