Testing Before Deploying: How a Bash Helper Function Replaced a Fragile Pipeline Pattern

In the midst of a grueling debugging session spanning dozens of messages, the assistant reached a critical juncture. After identifying four distinct bugs in a production bash script, rewriting the entire benchmark.sh file, and verifying its syntax, the assistant paused to test a single helper function before deploying the fix to a remote instance. Message [msg 4081] captures this moment of deliberate, incremental verification — a small but telling example of disciplined engineering under pressure.

The Debugging Context

The assistant had been chasing a baffling crash on a remote vast.ai instance running on an RTX 5090 GPU. The benchmark script (benchmark.sh) would complete its Phase 1 warmup successfully, then crash with a syntax error at line 346 — a line that bash -n had already confirmed was syntactically valid. The crash appeared alongside an OOM (out-of-memory) kill, but the root cause turned out to be far more subtle than a simple memory exhaustion.

Through careful log analysis and experimentation (see [msg 4071] through [msg 4076]), the assistant uncovered a tangled interaction between several bash features: set -euo pipefail, the if ! cmd | tee pipeline pattern, and the behavior of $? after an if compound command. The assistant identified four bugs in the original script:

  1. Double daemon start: The daemon was started both outside and inside the run_benchmark() function, meaning OOM recovery could never properly restart it.
  2. The if ! cmd | tee pattern: Using if ! ... | tee; then inside a function with pipefail created confusing exit code semantics. The ! operator inverts the pipeline exit code, and tee's exit code (always 0 under normal operation) could mask failures from the actual benchmark binary.
  3. $? after if: The OOM retry loop used rc=$? immediately after if run_benchmark; then, but $? after an if compound command is the exit status of the if itself (always 0 or 1), not of run_benchmark. This completely broke OOM detection.
  4. Output buffering: When stdout was redirected to a file (as it was by the Docker entrypoint), echo statements used block buffering, causing log lines to appear out of order and confusing the debugging effort.

The run_bench_phase Helper

The fix for bug #2 was to replace every instance of the if ! cmd | tee pattern with a robust helper function that uses the || phase_rc=${PIPESTATUS[0]} idiom. This pattern leverages bash's PIPESTATUS array, which captures the exit codes of each command in a pipeline individually. By using || (which only fires when the overall pipeline exit code is non-zero under pipefail) combined with ${PIPESTATUS[0]} (which specifically captures the exit code of the first command in the pipeline), the helper correctly distinguishes between a genuine benchmark failure and a tee failure.

The run_bench_phase function, shown in the subject message, encapsulates this pattern cleanly:

run_bench_phase() {
    local logfile="$1"; shift
    local desc="$1"; shift

    local phase_rc=0
    "$BENCH_BIN" "$@" 2>&1 | tee "$logfile" || phase_rc=${PIPESTATUS[0]}

    if [[ $phase_rc -ne 0 ]]; then
        echo "ERROR: $desc failed (exit=$phase_rc)" >&2
        return 1
    fi
    return 0
}

This function takes three arguments: a log file path, a human-readable description of the phase, and the arguments to pass to the benchmark binary. It runs the binary with its output piped through tee to the log file, captures the exit code of the binary (not of tee), and returns success or failure accordingly. The local keyword ensures the helper is self-contained and doesn't leak variables into the calling scope.

The Test Script

Message [msg 4081] shows the assistant writing a standalone test script to verify this helper function works correctly in both success and failure scenarios. The test script is carefully constructed:

Test 1 (Success Path): The assistant sets BENCH_BIN=/bin/echo and calls run_bench_phase with arguments that simulate a real benchmark invocation (--addr http://localhost:1234 batch -t porep). Since /bin/echo always exits 0, this tests that the function correctly identifies success. The output confirms this: the echoed arguments appear, followed by "PASS".

Test 2 (Failure Path): The assistant sets BENCH_BIN=/bin/false and calls the same function. /bin/false always exits 1, simulating a benchmark failure. The output shows the error message "ERROR: test phase failed (exit=1)" printed to stderr, and the function returns 1, causing the || branch to print "FAIL rc=1".

Both tests pass exactly as expected. The function correctly captures exit codes, prints appropriate error messages, and propagates the failure status to the caller.

The Thinking Process

This message reveals several aspects of the assistant's thinking process:

Incremental confidence building. Rather than deploying the entire rewritten script at once and hoping for the best, the assistant tests each component in isolation. First syntax was verified (bash -n in [msg 4078]). Then the || phase_rc=${PIPESTATUS[0]} pattern was tested in isolation ([msg 4079]). Then local variable scoping was verified ([msg 4080]). Now the helper function itself is tested. Each step builds confidence before moving to the next.

Isolation of concerns. The test script deliberately uses /bin/echo and /bin/false as stand-ins for the actual cuzk-bench binary. This is a classic unit testing technique: test the bash plumbing logic independently of the actual application binary. If the test failed, the assistant would know the bug was in the helper function itself, not in the benchmark binary's behavior.

Exhaustive case coverage. The assistant tests both the success and failure paths explicitly. This is not a quick smoke test — it's a deliberate verification that the error handling works correctly. The failure test is especially important because the whole point of the helper is to replace a pattern that was silently masking failures.

Attention to detail. The assistant uses 2>&1 in the failure test call to redirect stderr to stdout, ensuring the error message appears in the test output. This shows careful thought about what the test output should look like and how to make it visible.

What This Message Teaches About Debugging Methodology

The subject message, though only a single bash command, embodies several principles of effective debugging:

  1. Fix the root cause, not the symptom. The assistant didn't just patch the syntax error — it identified the underlying pattern that made the script fragile and replaced it with a robust alternative.
  2. Test your fix before deploying. It's tempting, after hours of debugging, to push the fix immediately and see if it works. The assistant resists this temptation and verifies the fix incrementally.
  3. Use the right tool for the job. The || phase_rc=${PIPESTATUS[0]} pattern is a well-known idiom in bash scripting for handling pipeline exit codes. The assistant chose this over more complex alternatives (like set -o pipefail with explicit $? checks or subshell workarounds) because it's simple, correct, and well-understood.
  4. Build confidence in layers. Each verification step — syntax check, pattern test, variable scoping test, helper function test — eliminates a class of potential failures before the script is deployed to production.

Conclusion

Message [msg 4081] captures a quiet but essential moment in the debugging process. After the drama of identifying the crash, tracing the bash execution flow, and uncovering the subtle interactions between pipefail, if, and $?, the assistant takes a deliberate pause to verify that the replacement code works correctly. This is the kind of discipline that separates a rushed fix from a reliable one. The test passes, the helper function is confirmed correct, and the assistant can proceed to deploy with confidence — knowing that at least one class of bugs has been thoroughly eliminated.