The Broken Pipe That Told the Truth: A Benchmark Failure Reveals Hidden OOM Dynamics

Introduction

In the course of deploying a high-performance GPU proving engine (CuZK) onto memory-constrained vast.ai cloud instances, a single user message arrived that encapsulated both the triumph and the tragedy of the endeavor. Message <msg id=3983> is a benchmark failure report from a 342 GiB cgroup-limited instance running an RTX PRO 4000 GPU. On its surface, it appears to be a routine crash log: a "transport error" caused by a "broken pipe," the daemon killed, the benchmark rate recorded as zero. But beneath this mundane exterior lies a rich diagnostic story — one that validated months of work on cgroup-aware memory detection while simultaneously revealing the next layer of the OOM puzzle that would drive the remainder of the development effort.

Context: The Deployment of Cgroup-Aware Memory Detection

To understand the significance of this message, one must first understand what had just been deployed. The assistant had spent the preceding chunks implementing a cgroup-aware memory detection system for the CuZK proving engine. The problem was straightforward: inside Docker containers on vast.ai, the /proc/meminfo file reports the host machine's total RAM, not the container's cgroup limit. A machine with 503 GiB of host RAM but a 342 GiB cgroup limit would previously cause the engine to budget 493 GiB of memory — far exceeding what was actually available, leading to inevitable OOM kills.

The fix involved rewriting the detect_system_memory() function to read cgroup v2 (memory.max) and v1 (memory.limit_in_bytes) limits, returning the minimum of host RAM and the cgroup constraint. This was deployed alongside a memcheck utility, updated Docker scripts, and a 10 GiB safety margin. The first test on a 961 GiB cgroup-limited instance (RTX 4090) had succeeded beautifully, achieving ~63.9 proofs/hour with no OOM kills.

The user then deployed a second test instance — a 256 GiB target that actually resolved to a 342 GiB cgroup limit on a 503 GiB host (machine ID 55891, an RTX PRO 4000 in Norway). This was the instance whose benchmark failure is reported in message <msg id=3983>.

Anatomy of a Crash: What the Logs Reveal

The message begins with what looks like a success story. The synthesis pipeline is humming along:

2026-03-15T09:27:33.467611Z  INFO synthesize_partition{...}: cuzk_core::pipeline: partition synthesis complete partition=6 synth_ms=232414 is_pinned=true

Partition 6 completed synthesis in 232 seconds — a reasonable time for a proving workload. The is_pinned=true flag confirms that the pinned memory pool was functioning, allocating host-pinned buffers for fast GPU transfers. The partition is then pushed to the GPU queue and immediately picked up by worker 0:

TIMELINE,502653,GPU_PICKUP,2dc50399-...,worker=0,partition=6

The GPU timing log shows zero overhead for the hot path pre-prove checks:

GPU_TIMING hot_path pre-prove overhead worker_id=0 partition=Some(6) status_ms=0 fail_check_ms=0 mark_busy_ms=0 total_overhead_ms=0

Then the crash. No GPU computation errors, no CUDA errors, no out-of-memory messages from the engine itself. Just:

[benchout]Error: Prove RPC failed
[benchout]Caused by:
[benchout]    0: status: Unknown, message: "transport error", details: [], metadata: MetadataMap { headers: {} }
[benchout]    1: transport error
[benchout]    2: connection error
[benchout]    3: stream closed because of a broken pipe

The "broken pipe" is the critical clue. In a gRPC-based system where the benchmark client communicates with the cuzk daemon over a local TCP connection, a broken pipe means the remote end of the connection (the cuzk daemon process) was terminated abruptly. The daemon did not shut down gracefully — it was killed.

The OOM Signature

The instance metadata confirms the kill:

Killed: 3/15/2026, 10:29:04 AM
Kill Reason: bench_rate 0.0 below min_rate 20.0

The vast.ai supervisor detected that the benchmark produced zero proofs and killed the instance for falling below the minimum rate of 20 proofs/hour. But the real cause of death was the OOM killer. The "broken pipe" is the hallmark of the Linux out-of-memory killer (OOM killer) terminating a process mid-execution. When the OOM killer strikes, the process is sent SIGKILL — no warning, no graceful shutdown, no error message logged by the application itself. The only evidence is the severed connection.

The timing is particularly telling. The crash occurred after synthesis completed and after the partition was handed off to the GPU worker. This suggests that the GPU proving phase — which involves additional memory allocations for GPU command buffers, CUDA streams, and kernel execution — pushed the system over the cgroup memory limit. The synthesis phase (CPU-bound, using the pinned memory pool) had already consumed a significant portion of the budget, and the GPU phase's additional allocations tipped the scales.

Memcheck: The Validation Within the Failure

Crucially, the message also contains the memcheck report, which serves as a powerful validation of the cgroup-aware detection system:

Memcheck
Effective RAM: 342GiB
Host MemTotal: 504GiB
cgroup (v2): 342GiB
Budget: 331GiB
Pin: OK (ulimit=64)
Max PoRep: 6
Max Snap: 9
Bench Conc: 4

The system correctly identified the cgroup limit of 342 GiB, set a budget of 331 GiB (342 - 10 GiB safety margin), and configured appropriate concurrency limits (6 concurrent PoReps, 4 benchmark concurrency). The warning about the 161 GiB discrepancy between cgroup and host RAM confirms that the detection logic is working as intended — it is no longer being fooled by /proc/meminfo.

This is a bittersweet validation. The memory detection is correct, but the budget of 331 GiB was still insufficient. The system crashed despite having what should have been a generous 10 GiB safety margin. Something else was consuming memory beyond what the budget accounted for.

What This Message Enabled: The Next Phase of Investigation

The failure reported in this message was the catalyst for the investigation that followed in the subsequent chunks. The user and assistant would go on to identify two additional memory consumption sources that the budget system did not account for:

  1. The pinned memory pool accounting gap: Pinned buffers returned to the pool after GPU work were never freed from actual RSS (Resident Set Size). The pool grew over time, consuming memory that the budget system believed was available.
  2. Kernel and driver overhead: The glibc memory allocator (ptmalloc) maintains internal arenas, the GPU driver reserves its own memory for command submission and synchronization, and page tables consume physical memory proportional to the process's virtual address space. These overheads can amount to several gigabytes on a 342 GiB machine. The 10 GiB safety margin was empirically insufficient for these hidden costs. This message directly motivated the creation of the memprobe utility — a C program that allocates 1 GiB chunks via mmap/memset until it nears the cgroup limit, providing a data-driven measurement of the true available memory after kernel overhead. It also motivated the OOM recovery loop in benchmark.sh: if the daemon is killed (exit code 137), the budget is reduced by 10% and the benchmark is retried up to three times.

Assumptions and Knowledge Required

To fully understand this message, one needs several pieces of context:

Output Knowledge Created

This message created several critical pieces of knowledge:

  1. Empirical confirmation that cgroup-aware detection works: The memcheck report proves the system correctly identifies the cgroup limit. The detection logic is validated on a real production instance.
  2. Empirical evidence that the 10 GiB safety margin is insufficient: Despite correct detection and budgeting, the instance still OOM-killed. This falsified the assumption that a 10 GiB margin would be adequate for all cases.
  3. A diagnostic pattern for OOM during GPU proving: The sequence of "synthesis complete → GPU pickup → broken pipe" is a reliable signature for OOM kills during the GPU proving phase, distinguishable from other failure modes (e.g., CUDA errors, synthesis crashes).
  4. A concrete target for the next iteration: The failure mode directly motivated the memprobe utility and OOM recovery loop, which became the focus of the subsequent development effort.

Conclusion

Message <msg id=3983> is a study in contrasts. It reports a failure — a benchmark that produced zero proofs, an instance killed, a deployment that did not meet its performance target. Yet within that failure is a wealth of diagnostic information that validated one system (cgroup-aware detection) while exposing the limitations of another (the budget model's incomplete accounting for kernel overhead and pinned pool RSS). The broken pipe, far from being a mere error message, was a truthful signal that pointed the development effort in exactly the right direction. In the complex world of GPU proving on memory-constrained cloud instances, sometimes the most informative message is the one that reports a crash.