The Phantom Binary: Debugging Overlay Filesystem Caching in a Containerized Deployment

The Message

[assistant] The `cp` keeps writing the old size! The file system might have the old binary cached. The `rm` succeeds but `cp` writes old content. Let me try differently:
[bash] ssh -p 40612 root@141.0.85.211 'md5sum /tmp/cuzk-orderedsyn /usr/local/bin/cuzk' 2>&1
0353080709b0ea71930723d1deaa1059  /tmp/cuzk-orderedsyn
d2d9bed586e52f7f722bcd5a8a22952a  /usr/local/bin/cuzk

This brief message, sent by the AI assistant in the middle of a complex deployment and debugging session, captures a moment of genuine discovery. It is the pivot point in a troubleshooting chain that had been spinning its wheels on a seemingly simple problem: why won't the new binary replace the old one? The answer, once uncovered, would lead the assistant deep into the arcane semantics of Docker overlay filesystems, container layer caching, and the subtle ways that containerized environments deviate from normal Linux behavior.

Context: The Ordered Partition Scheduling Fix

To understand why this message matters, we must first understand what the assistant was trying to accomplish. The broader session (Segment 20 of the opencode conversation) focused on deploying and refining a "vast-manager cuzk status panel" — a monitoring dashboard for a GPU-based proof generation pipeline called CuZK. In the chunk preceding this message (Chunk 1 of Segment 20), the assistant had implemented a critical architectural fix: replacing the existing partition scheduling mechanism with an ordered, FIFO channel-based system.

The original design had a flaw: each partition from each proof pipeline was spawned as an independent Tokio task, and all these tasks raced on a shared memory budget via budget.acquire(). This meant partitions were processed in effectively random order. A nearly-finished pipeline could stall waiting for GPU proving while other pipelines, which had no synthesis work left, continued to consume resources. The fix replaced the per-partition tokio::spawn pattern with a shared mpsc (multi-producer, single-consumer) channel. Partitions were enqueued in FIFO order — earlier pipelines first, lower partition indices first — and a pool of synthesis workers pulled from the channel sequentially. This ensured predictable, efficient processing that minimized gaps in synthesis work.

The assistant had successfully compiled this change, built a new release binary via Docker (cuzk-rebuild:orderedsyn), extracted it from the container image, and uploaded it to the remote test machine (IP 141.0.85.211, SSH port 40612) via scp. The binary landed in /tmp/cuzk-orderedsyn. Then the deployment began.

The Mysterious Stale Binary

The first attempt to deploy the new binary followed a straightforward script: kill the running daemon, remove the old binary at /usr/local/bin/cuzk, copy the new one from /tmp/cuzk-orderedsyn into place, and restart. But when the assistant queried the status endpoint, the synth_max field showed 4 instead of the expected 44. The assistant had recently changed the status API to compute synth_max dynamically from the memory budget (total_bytes / min_partition_size), and with a 400 GiB budget and 9 GiB per partition, the result should have been 44. The fact that it showed 4 strongly suggested the old binary was still running.

A size check confirmed the discrepancy: /usr/local/bin/cuzk was 27,475,224 bytes (the old build), while /tmp/cuzk-orderedsyn was 27,494,064 bytes (the new build). The cp had apparently failed. The assistant tried again in message 2800, explicitly running rm -f /usr/local/bin/cuzk && cp /tmp/cuzk-orderedsyn /usr/local/bin/cuzk && chmod +x /usr/local/bin/cuzk — a sequence that should be bulletproof on any normal Linux system. Yet in message 2801, the file size remained unchanged: still 27,475,224 bytes, still the old binary.

The Moment of Insight

This brings us to the subject message (message 2802). The assistant writes:

The cp keeps writing the old size! The file system might have the old binary cached. The rm succeeds but cp writes old content. Let me try differently.

The exclamation mark is telling. It signals a shift in the assistant's mental model. Up to this point, the assistant had been operating under the assumption that the deployment commands were executing correctly on a normal Linux filesystem. The rm appeared to succeed (no error), the cp appeared to succeed (no error), and yet the result was wrong. The assistant had been stuck in a loop of retrying the same commands, each time expecting a different outcome — a classic debugging trap.

The insight is the hypothesis that "the file system might have the old binary cached." This is a sophisticated leap. Rather than assuming a bug in the shell commands or a network issue with SSH, the assistant correctly identifies that the filesystem itself might be serving stale data. The cp might be writing to a layer that is being shadowed by a cached copy in a lower layer.

To confirm this hypothesis, the assistant runs md5sum on both files. Cryptographic hashes are the gold standard for file identity — if two files have different MD5 hashes, they are genuinely different, regardless of what ls -la or any other metadata might suggest. The result is unambiguous:

0353080709b0ea71930723d1deaa1059  /tmp/cuzk-orderedsyn
d2d9bed586e52f7f722bcd5a8a22952a  /usr/local/bin/cuzk

Different hashes. The two files are genuinely different binaries. The new binary exists on disk at /tmp/cuzk-orderedsyn, but the old binary stubbornly persists at /usr/local/bin/cuzk despite repeated deletion and copy attempts.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of contextual knowledge:

  1. The deployment pipeline: The assistant builds a Rust binary inside a Docker container, extracts it with docker create + docker cp, uploads it via scp to the remote machine's /tmp directory, and then attempts to install it to /usr/local/bin.
  2. The remote environment: The test machine (141.0.85.211) is a vast.ai instance, which typically runs workloads inside Docker containers. This means the filesystem visible to SSH is actually a Docker overlay filesystem, not a bare-metal Linux filesystem.
  3. Overlay filesystem semantics: Docker overlay filesystems consist of multiple layers (lowerdir, upperdir, merged). Files in lower layers are read-only and can be shadowed by the upper layer. When you delete a file that exists in a lower layer, Docker creates a "whiteout" file in the upper layer to mask it. However, when you then write a new file to the same path, the overlay may serve the old file from the lower layer if the copy-up operation doesn't work as expected.
  4. The synth_max calculation: The assistant had recently changed the status API to compute max_concurrent synthesis slots as total_bytes / min_partition_size. With 400 GiB total and 9 GiB per SnapDeals partition, the expected value was 44. The fact that the running binary showed 4 indicated it was the old build.
  5. The partition ordering fix: The assistant had just implemented a significant refactor of the synthesis pipeline, replacing tokio::spawn with mpsc channel-based ordered scheduling. This was the feature being deployed, and its absence (if the old binary was running) would mean the fix wasn't in effect.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. Definitive proof of binary mismatch: The MD5 hashes provide irrefutable evidence that /tmp/cuzk-orderedsyn and /usr/local/bin/cuzk are different files. This eliminates any possibility that the cp command was silently succeeding but the files happened to be identical.
  2. A falsified hypothesis: The assistant had previously assumed that the cp command was simply not being executed or was failing silently. The md5sum output disproves this — the new binary exists at /tmp/cuzk-orderedsyn, so the upload step worked. The problem is specifically with the replacement of the installed binary at /usr/local/bin/cuzk.
  3. A new hypothesis about filesystem caching: The assistant now suspects that the filesystem itself is caching the old binary. This is a fundamentally different class of problem than a command-line error. It shifts the debugging from "what command did I run?" to "what is this filesystem doing?"
  4. The foundation for the eventual solution: This insight leads directly to the subsequent investigation. In the following messages, the assistant checks mount output and discovers the overlay filesystem, tries mv instead of cp (which also fails due to cross-filesystem copy-up semantics), tries scp directly to /usr/local/bin (which also fails), and eventually discovers that the overlay filesystem has the old binary cached in a lower layer. The workaround — deploying to /data/cuzk-ordered, a path that doesn't exist in any lower layer — succeeds immediately.

Assumptions and Their Failure

The assistant made several assumptions that turned out to be incorrect:

  1. POSIX filesystem semantics: The assistant assumed that rm followed by cp on a Linux system would reliably replace a file. On a normal ext4 or XFS filesystem, this is trivially true. But Docker overlay filesystems violate this assumption: a file that exists in a lower layer cannot be truly deleted; it can only be masked by a whiteout entry in the upper layer. And when a new file is written to the same path, the overlay's copy-up operation may serve the old content from the lower layer if the whiteout is not properly handled.
  2. That the remote machine is a "normal" Linux server: The assistant treated the SSH session as a standard Linux environment. While the remote machine does run Linux, it runs inside a Docker container with an overlay filesystem. This is a common deployment pattern for vast.ai and similar GPU cloud platforms, but it introduces filesystem behaviors that differ from bare metal.
  3. That cp errors would be visible: The assistant expected that if cp failed, it would produce an error message. But the cp command did not fail — it completed successfully, writing the new binary to the overlay's upper layer. The problem was that reads from the merged view continued to return the lower layer's copy. This is a silent failure mode with no error output.
  4. That file size alone was sufficient evidence: In message 2800, the assistant compared file sizes (27,494,064 vs 27,475,224) and concluded the cp "failed." But file sizes can be coincidental or misleading. The md5sum in this message provides cryptographic certainty.

The Thinking Process

The reasoning visible in this message reveals a methodical debugging approach. The assistant has moved through several stages:

Stage 1 (message 2799): Observe the symptom. The status shows synth: 0/4 instead of the expected synth: 0/44. The assistant performs a sanity check on the calculation (400 GiB / 9 GiB = 44.4 → 44) and confirms the math is correct. Then checks if the old binary was deployed instead.

Stage 2 (message 2800): Compare file sizes. Discover that /usr/local/bin/cuzk (27,475,224 bytes) is smaller than /tmp/cuzk-orderedsyn (27,494,064 bytes). Conclude that "The cp earlier failed." Attempt to fix by running rm + cp again.

Stage 3 (message 2801): Verify the fix. Find that the file size is still the old value. The rm and cp commands executed without error, but the result is unchanged.

Stage 4 (message 2802, the subject): Formulate a new hypothesis. The assistant realizes that the commands are executing correctly but the filesystem is returning stale data. The phrase "The cp keeps writing the old size!" with the exclamation mark signals this breakthrough. The assistant then uses md5sum to obtain definitive evidence.

This progression — from symptom observation, to command-level debugging, to filesystem-level investigation — is a textbook example of escalating abstraction in debugging. Each stage tests a different layer of the system: first the application logic (the synth_max calculation), then the shell commands (rm, cp), then the filesystem itself (overlay caching).

The Broader Significance

This message is more than just a diagnostic step in a single debugging session. It illustrates several enduring lessons about software deployment in containerized environments:

  1. Containers are not lightweight VMs: A Docker container shares the host kernel but has its own filesystem semantics. Overlay filesystems behave differently from traditional filesystems, and operations that are atomic on ext4 (like rm + cp) may not be atomic on overlay.
  2. Cryptographic hashes are your friend: When files don't behave as expected, md5sum (or sha256sum) provides definitive evidence that bypasses any filesystem caching or metadata quirks.
  3. The importance of deployment path hygiene: The eventual solution — deploying to /data/cuzk-ordered, a path that didn't exist in any lower layer — worked because it avoided the overlay shadowing problem entirely. This is a practical lesson: if you're deploying binaries to a containerized environment, use paths that are guaranteed to be in the writable layer.
  4. Debugging requires escalating abstraction: When a fix at one level doesn't work, the problem may be at a lower level. The assistant's progression from application logic to shell commands to filesystem internals is a model of systematic debugging.

Conclusion

Message 2802 is a small but pivotal moment in a complex deployment session. In just a few lines, the assistant moves from confusion to insight, from retrying the same failing commands to formulating a new hypothesis about filesystem caching. The md5sum output provides the evidence needed to confirm that hypothesis and sets the stage for the eventual workaround. It's a reminder that in modern cloud-native environments, even the simplest operations — copying a file — can behave unexpectedly, and that the most effective debuggers are those who can recognize when their assumptions about the underlying infrastructure are wrong.