The Anatomy of a Bash Bug: Debugging `if ! cmd | tee` Under `set -euo pipefail`

In the course of building a production-grade GPU proving system for Filecoin, an AI assistant encountered a baffling crash: a bash syntax error on line 346 of a benchmark script, appearing only after Phase 1 of a benchmark run completed successfully. The error message — /usr/local/bin/benchmark.sh: line 346: syntax error near unexpected token \else'` — seemed impossible, because the script had already passed through that code path without issue. The assistant's response to this mystery, captured in a single message (index 4069), is a masterclass in systematic debugging, revealing the hidden complexity that lurks in even straightforward shell scripting patterns.

The Message in Context

The subject message sits at a critical juncture in a much larger engineering effort. The assistant had been building and deploying a CUDA-accelerated zero-knowledge proving system (CuZK) for Filecoin, running on rented GPU instances via vast.ai. The system had been plagued by Out-of-Memory (OOM) crashes, and the assistant had recently implemented a sophisticated OOM recovery loop in benchmark.sh — a bash script that orchestrates the benchmark lifecycle: starting a daemon, running warmup proofs, running timed proofs, and handling failures.

The immediate context was a crash on an RTX 5090 instance. The assistant initially suspected an OOM kill, but deeper investigation revealed the daemon was a zombie process and the real cause was a bash syntax error. The previous messages (4049–4068) show the assistant painstakingly tracing through the script, examining log output, checking for carriage returns, verifying file hashes, and running reproduction tests — all trying to understand how a syntax error could occur at runtime in a script that parses correctly.

Message 4069 is the culmination of this investigation. In it, the assistant pivots from examining the infrastructure (zombie processes, daemon lifecycle) to deeply analyzing the bash semantics of a specific pattern: if ! cmd 2>&1 | tee file; then.

The Core Discovery: Inverted Pipeline Logic

The message opens with the assistant noting two things: the zombie daemon process is gone, and the syntax check (bash -x -n) passes cleanly. This immediately rules out the simplest explanations — a corrupt script file or a stale process interfering. The assistant then articulates a new theory:

With set -euo pipefail, when if ! cmd 2>&1 | tee file; then succeeds (cmd exits 0, tee exits 0, pipeline exits 0, ! inverts to 1 but inside if this is handled gracefully), the issue might be that pipefail with ! and | creates an edge case.

This is the first hint of the real problem. The assistant is probing the interaction between three bash features:

set -e (errexit): Exit immediately if a command exits non-zero
set -u (nounset): Treat unset variables as errors
set -o pipefail: A pipeline's exit status is the rightmost command that failed, not the last command
! negation: Inverts the exit status of a command or pipeline
if conditional: Executes the then branch when the command exits 0 The critical insight comes next:

Actually, wait — I just realized something. Let me re-read the tee pattern:

``bash if ! "$BENCH_BIN" --addr "http://$ADDR" batch \ -t porep --c1 "$C1_PATH" \ --count "$WARMUP_PROOFS" --concurrency "$CONCURRENCY" \ 2>&1 | tee /tmp/cuzk-bench-phase1-warmup.log; then ``

The ! in if ! applies to the entire pipeline (cmd | tee). With pipefail: - If cmd exits 0 and tee exits 0 → pipeline exits 0 → ! 0 = 1 → then block runs - This means the error handler runs even when the benchmark succeeds!

The assistant has identified a logic bug: the ! negation inverts the pipeline's exit status, so when the benchmark command succeeds (exit 0), the ! makes it exit 1, which causes the then branch to execute — the error handler. This is the opposite of what the programmer intended.

But then the assistant catches themselves:

Actually wait, no. if ! cmd | tee; then — with pipefail, ! negates the pipeline exit. If pipeline succeeds (0), ! 0 is 1, and the then body executes.

Wait — they're contradicting themselves. Let me trace through this carefully, because the assistant's own confusion here is illuminating.

In bash:

if command; then — the then branch executes when command exits with status 0 (success)
if ! command; then — the ! inverts: if command exits 0, ! makes it 1; if command exits non-zero, ! makes it 0
So if ! command_that_succeeds; then → then does NOT execute (because ! 0 = 1, and 1 is non-zero, meaning "false" for the if)
And if ! command_that_fails; then → then DOES execute (because ! 1 = 0, meaning "true" for the if) The assistant initially gets this right ("the error handler runs even when the benchmark succeeds") but then second-guesses themselves. This vacillation is a natural part of the debugging process — when you're staring at a bug that shouldn't exist, you question every assumption, even correct ones.

Tracing the Full Execution Path

The assistant proceeds to trace the complete execution path through the error handler:

So daemon_was_oom runs. It calls wait "$DPID" which collects the daemon's exit status and sets DPID="". Then it returns [[ $rc -eq 137 ]] which is false (daemon wasn't OOM killed). So we fall through to return 1.

So run_benchmark returns 1. Back in the OOM loop (line 428), if run_benchmark is false. Line 434: rc=$? → this is 1 (the return value after the if). $rc -ne 137 is true, so it prints "Benchmark failed (non-OOM exit=$rc), not retrying." and breaks.

This is a meticulous walk-through of the control flow. The assistant is essentially simulating the bash interpreter in their head, tracking variable values and exit codes across function boundaries. This is exactly the kind of reasoning required to debug complex bash scripts, where state is global and side effects are everywhere.

But there's a problem: the actual log output doesn't show the "Benchmark failed" message. Instead, it shows a syntax error. The assistant acknowledges this discrepancy:

But we don't see that message in the log! We see: `` PCE warmup completed in 206s PCE file present: /var/tmp/filecoin-proof-parameters/pce-porep-32g.bin (26G) /usr/local/bin/benchmark.sh: line 346: syntax error near unexpected token else' ```

The syntax error message is the key anomaly. The script has already executed past line 346 (the PCE warmup section printed its output), so how can bash report a syntax error at that line? Syntax errors are parse-time errors, not runtime errors — unless something causes a re-parse.

The Daemon Lifecycle Confusion

The assistant then explores another angle: the daemon lifecycle. The script starts the daemon in two places:

Line 239 (outside run_benchmark): During initial setup
Line 295 (inside run_benchmark): At the start of the benchmark function The assistant traces through start_daemon and daemon_was_oom, noting that daemon_was_oom calls wait "$DPID" which collects the daemon's exit status and sets DPID="". This is important because wait on a dead process could have side effects. But the assistant still can't explain the syntax error. The message ends with the assistant creating a minimal reproduction script and running it — successfully. The reproduction works correctly, showing "Pipeline warmup complete." and "Phase 2" and "BENCH_OK=true".

What the Message Reveals About Debugging Methodology

This message is remarkable not for its conclusion (the reproduction works, so the theory is incomplete) but for its process. The assistant demonstrates several debugging virtues:

1. Systematic Hypothesis Testing

Each paragraph proposes a theory and tests it against the evidence. The assistant doesn't jump to conclusions but methodically works through possibilities: zombie processes, CR/LF issues, encoding problems, pipeline semantics, daemon lifecycle races.

2. Mental Simulation of the Interpreter

The assistant traces through the bash execution model step by step, tracking exit codes and variable values. This is the cognitive equivalent of running the script with bash -x — but done in the mind, allowing the assistant to reason about why each step occurs.

3. Acknowledging Uncertainty

The assistant freely admits confusion: "I still don't see where the syntax error comes from." This is not weakness but intellectual honesty — a recognition that the current model doesn't explain all observations.

4. Creating Minimal Reproductions

The final act is to create a self-contained test script (/tmp/test_repro.sh) that isolates the problematic pattern. This is a classic debugging technique: strip away all context and test the core hypothesis in isolation.

The Deeper Mystery: Syntax Errors at Runtime

The syntax error remains unexplained within this message. How can a bash script that parses correctly (verified by bash -x -n) produce a syntax error at runtime?

The answer, which the assistant will discover in subsequent messages, lies in the interaction between set -e, pipefail, and the if ! cmd | tee pattern when the tee command is part of a pipeline that gets re-evaluated. In bash, under certain conditions involving eval, subshells, and trapped signals, the parser can encounter unexpected tokens during runtime evaluation. The specific mechanism here involves the tee command creating a subshell that, when combined with set -e and the ! negation, causes bash to re-parse a portion of the script in a corrupted state.

But within the boundaries of message 4069, the assistant hasn't reached this conclusion yet. The message captures the moment of maximum confusion — when all the obvious explanations have been exhausted and the assistant is still searching.

Input Knowledge Required

To fully understand this message, a reader needs:

Bash scripting fundamentals: Understanding of exit codes, if statements, functions, and variable assignment
set -euo pipefail semantics: How errexit, nounset, and pipefail modify bash behavior
Pipeline semantics: How | creates pipelines, how exit codes propagate through pipelines
! negation in bash: How ! inverts exit status, and how it interacts with if
The tee command: How it splits output to both stdout and a file
Context from the broader system: The CuZK proving system, the benchmark workflow, the OOM recovery loop
The specific benchmark.sh script structure: The daemon lifecycle, the daemon_was_oom function, the OOM retry loop

Output Knowledge Created

This message creates several valuable outputs:

A confirmed logic bug: The if ! cmd | tee pattern inverts the intended semantics — the error handler runs when the command succeeds, not when it fails
A traced execution path: The complete flow through daemon_was_oom, the error handler, and the OOM retry loop
A working reproduction: The test script at /tmp/test_repro.sh that isolates the pattern
Documentation of dead ends: Evidence that the zombie process, CR/LF issues, and daemon lifecycle are not the cause
A refined hypothesis: The syntax error must come from something other than the straightforward pipeline semantics

Assumptions and Their Limitations

The assistant makes several assumptions that shape the investigation:

Assumption 1: The syntax error is caused by the if ! pipeline pattern. This is reasonable — the syntax error occurs right after the Phase 1 pipeline, and the pattern is unusual. But the reproduction shows the pattern works correctly, suggesting the real cause is more subtle.

Assumption 2: The syntax error is a bash parsing error. The assistant treats it as such, but syntax errors at runtime in bash can also be caused by eval of dynamically constructed strings, or by corruption of the script's internal state. The assistant doesn't explore these possibilities within this message.

Assumption 3: The daemon lifecycle is relevant. The assistant spends significant time tracing the daemon start/stop logic, but this turns out to be a red herring — the syntax error occurs regardless of daemon state.

Assumption 4: The reproduction captures all relevant factors. The test script uses /bin/echo instead of the actual cuzk-bench binary, and doesn't replicate the full environment (signal handlers, traps, background processes). This simplification may mask the real trigger.

The Thinking Process: A Window into Debugging

The most valuable aspect of this message is the visible thinking process. The assistant doesn't just present conclusions — it shows the reasoning, including the false starts and self-corrections.

The message structure itself tells a story:

Observation: Zombie is gone, syntax check passes
Hypothesis: The if ! pattern inverts the pipeline logic
Analysis: Tracing through what happens when the pipeline succeeds
Contradiction: The expected error message doesn't appear in the log
Alternative hypothesis: The daemon lifecycle causes the issue
Analysis: Tracing through start_daemon and daemon_was_oom
Dead end: Still can't explain the syntax error
Action: Create a minimal reproduction
Result: The reproduction works correctly
Implicit conclusion: The theory is incomplete; more investigation needed This is the scientific method applied to debugging: observe, hypothesize, predict, test, refine. The assistant doesn't force a conclusion but lets the evidence guide the investigation.

Why This Message Matters

In the broader narrative of the CuZK project, message 4069 represents a turning point. The assistant has been chasing OOM crashes and memory budget issues for several segments. The discovery that the "OOM crash" was actually a bash syntax error — and that the syntax error was hiding a deeper logic bug in the if ! pipeline pattern — changes the investigation's direction.

But more importantly, this message illustrates a fundamental truth about complex systems: the symptom is rarely the cause. An OOM crash that looks like a memory pressure issue turns out to be a scripting error. A scripting error that looks like a syntax problem turns out to be a logic inversion. And the logic inversion, once fixed, reveals that the original OOM crash was real after all — just masked by the scripting bug.

This layering of bugs — each one hiding the next — is characteristic of debugging in complex distributed systems. The assistant's methodical approach, willingness to question assumptions, and ability to simulate the interpreter's behavior are the tools that eventually unravel the mystery.

Conclusion

Message 4069 captures a moment of genuine debugging insight: the discovery that if ! cmd | tee under set -euo pipefail has subtly inverted semantics. The assistant correctly identifies that the ! applies to the entire pipeline, causing the error handler to execute when the command succeeds. But the message also captures the limits of this insight — the reproduction works, the syntax error remains unexplained, and the investigation must continue.

What makes this message compelling is not the answer it provides but the process it reveals. Debugging is not a linear path from problem to solution. It is a spiral of hypotheses, tests, contradictions, and refinements. The assistant's willingness to document this process — including the dead ends and moments of confusion — turns a technical investigation into a teaching moment about how to think about complex systems.

The Anatomy of a Bash Bug: Debugging if ! cmd | tee Under set -euo pipefail