The Cached Image Trap: A Microcosm of Infrastructure Debugging

Introduction

In the midst of an intensive debugging session for a Filecoin Gateway (FGW) cluster deployment pipeline, a single message from an AI assistant captures one of the most fundamental yet easily overlooked pitfalls in containerized infrastructure work: the cached Docker image. The message, brief and action-oriented, represents a moment of meta-cognitive clarity—the realization that a fix didn't work not because the fix was wrong, but because the testing environment was stale.

The Message

The subject message, from the assistant at index 1643 in the conversation, reads:

The systemd-user-sessions.service removal didn't work - the Docker images are cached. Let me force rebuild:

```bash cd /home/theuser/gw/ansible/test/docker && docker compose build --no-cache kuri-01 kuri-02 s3-fe-01 2>&1 | tail -30 #18 [kuri-01] exporting to image #18 exporting layers #18 exporting layers 1.0s done #18 writing image sha256:00dac310b34f88d51859a534dbc7dd1270306c20ac37add5c8b6e186a4094b6d done #18 naming to docker.io/library/docker-kuri-01 done #18 DONE 1.1s

#19 [s3-fe-01] exporting to image #19 exporting layers 1.0s done #19 writing image sha256:c19bbaea8e39dbe8dca38a51be701b58318635ca8556fe6e0eaa6d8d859df885 done #19 naming to docker.io/library/docker-s3-fe-01 done #19 DONE 1.1s

#20 [kuri-02] exporting t... ```

On its surface, this is a simple command invocation: force-rebuild three Docker images. But the reasoning behind it reveals a layered debugging process that is the essence of infrastructure-as-code development.

The Context: A Cascade of Deployment Bugs

To understand why this message matters, one must appreciate the debugging marathon that preceded it. The assistant had been iteratively fixing a suite of Ansible deployment scripts for a distributed S3 storage system built on the Filecoin Gateway architecture. The test harness used Docker containers running systemd to simulate production Ubuntu servers, with Ansible driving SSH-based configuration management.

The session had already uncovered and resolved several subtle bugs:

Environment file syntax: systemd's EnvironmentFile directive rejects export prefixes, but the Jinja template settings.env.j2 included them, causing silent configuration failures.
Log level format: The configuration used *:* as a log level pattern, which the Go regex parser rejected as "missing argument to repetition operator."
Wallet dotfiles: Hidden files like .gitkeep in the wallet directory caused binary parsing errors when kuri tried to read them as wallet data.
Duplicate table creation: Both the yugabyte_init Ansible role and kuri init tried to run database migrations, causing CREATE TABLE to fail on the second attempt.
Non-existent Ansible filter: The s3_frontend role referenced format_backend_url, a filter that didn't exist in the installed Ansible collection. Each of these had been diagnosed through log analysis, fixed in the source code, and tested. But one stubborn problem remained: the pam_nologin issue.

The nologin Problem

When systemd boots a Linux system, it creates a /run/nologin file (and its symlink /var/run/nologin) during the boot sequence. This file causes the Pluggable Authentication Module (PAM) to reject login attempts with the message "System is booting up. Unprivileged users are not permitted to log in yet." In a production server, this is sensible behavior—it prevents SSH access before the system is fully initialized. But in a Docker container running systemd for testing purposes, it becomes a persistent obstacle.

The assistant's initial approach was pragmatic: remove the nologin files manually after container startup. This worked, but it was a manual workaround, not a proper fix. The better solution was to prevent the nologin file from being created in the first place by disabling the systemd-user-sessions.service in the Dockerfile.

The assistant edited Dockerfile.target to add a systemctl disable systemd-user-sessions command, then ran the test setup script (setup.sh) which builds the Docker images and starts the containers. But when the tests ran again, the nologin file was still present.## The Core Insight: Cached Builds

The subject message begins with the crucial realization: "The systemd-user-sessions.service removal didn't work - the Docker images are cached." This is the moment where the assistant recognized that the edit to the Dockerfile was correct, but the build process was not incorporating it. Docker's layer caching, a feature designed to speed up development by reusing unchanged build steps, had become the enemy of debugging.

When docker compose build runs without --no-cache, Docker checks each instruction in the Dockerfile against its cache. If the instruction text and its context (the files it references, the base image layers) haven't changed, Docker reuses the cached layer instead of executing the instruction. The assistant had edited the Dockerfile to add a RUN systemctl disable systemd-user-sessions line, but if this was added after a RUN instruction that already existed, Docker might have considered the earlier layers still valid and skipped the new instruction—or, more likely, the build process was using cached images from a previous build that predated the edit.

The command chosen—docker compose build --no-cache—is the nuclear option. It tells Docker to ignore all cached layers and rebuild every instruction from scratch. The --no-cache flag ensures that the new systemctl disable line is actually executed, and that the resulting image contains the disabled service.

The Output: Confirmation and Ambiguity

The build output shown in the message is telling. All three images (kuri-01, kuri-02, s3-fe-01) rebuild successfully, each producing a new image SHA. The build times are remarkably short—around 1.1 seconds for the exporting phase—suggesting these are relatively lightweight images built on top of Ubuntu 24.04 with systemd and SSH installed.

But the output is truncated. The tail -30 command shows only the last 30 lines of the build output, and what we see is the final "exporting to image" phase. We don't see the critical middle section where the RUN systemctl disable systemd-user-sessions instruction would have been executed. This truncation is itself a subtle decision: the assistant chose to show only the tail end to confirm the build completed, rather than the full output to verify the fix was applied. In a debugging session, this is a reasonable trade-off—the assistant is moving fast, iterating quickly, and trusts that --no-cache will do its job.

Assumptions and Their Validity

The message rests on several assumptions, some explicit and some implicit:

The fix was correct: The assistant assumes that disabling systemd-user-sessions.service will prevent the nologin file from being created. This is a reasonable assumption based on systemd's behavior, but it's not guaranteed. The nologin file can also be created by other systemd units or by the pam_nologin module directly. The assistant is betting that systemd-user-sessions is the culprit, based on common systemd behavior in Docker containers.

Caching was the problem: The assistant assumes that the Docker build cache was the reason the fix didn't take effect. This is the most likely explanation, but there are alternatives: the edit might have been applied to the wrong file, the Dockerfile might have a syntax error that caused the instruction to be silently ignored, or the systemctl disable command might have failed silently. The assistant doesn't verify these possibilities before proceeding with the rebuild.

The build will succeed: The assistant assumes that docker compose build --no-cache will complete successfully. The output confirms this, but only partially—we see the export phase complete, but not the intermediate steps.

The test will pass afterward: There's an implicit assumption that once the images are rebuilt with the fix, the nologin problem will be resolved and the Ansible tests will pass. This is optimistic—there could be other issues lurking, and the nologin fix might not be sufficient (for instance, if the container's systemd boot sequence creates the nologin file before the service is disabled).

The Thinking Process

The message reveals a clear chain of reasoning:

Observation: The systemd-user-sessions.service removal didn't work—the nologin file still appears after container startup.
Hypothesis: The Docker images are cached, so the Dockerfile edit wasn't incorporated into the running containers.
Action: Force rebuild with --no-cache to ensure the edit is applied.
Verification: Run the build command and check the output for successful completion. This is classic scientific method applied to infrastructure debugging. The assistant doesn't panic, doesn't assume the fix was wrong, and doesn't try a different approach. Instead, it questions the testing environment itself—a mature debugging instinct that separates experienced operators from novices.

Input Knowledge Required

To understand this message fully, the reader needs knowledge of:

Docker build caching: How Docker uses layer caching to speed up builds, and why --no-cache is sometimes necessary.
Docker Compose: How docker compose build works with service definitions in a compose.yml file.
systemd and PAM: The role of systemd-user-sessions.service in creating /run/nologin, and how PAM's pam_nologin module uses this file to block logins during boot.
The broader debugging context: The nologin issue was blocking Ansible's SSH-based connectivity checks, which are the first test in the run-tests.sh script.
The test architecture: The test harness uses Docker containers with systemd to simulate production servers, and Ansible runs from a separate controller container.

Output Knowledge Created

This message creates several pieces of knowledge:

A confirmed build process: The images rebuilt successfully with new SHA hashes, meaning the Dockerfile changes were incorporated.
A debugging technique: The message documents the technique of using --no-cache when infrastructure changes don't seem to take effect.
A decision point: The assistant has committed to the systemd-user-sessions fix and is now testing it. The next message will reveal whether this was the correct approach.
Build timing data: The export phase took about 1.1 seconds per image, providing a baseline for future build performance comparisons.

The Broader Significance

This message, while seemingly mundane, is a microcosm of the entire debugging session. It represents the moment when the assistant shifted from fixing code to fixing the testing environment—a critical skill in infrastructure development. The cached image problem is universal: every developer who works with Docker has been burned by stale caches at some point. The assistant's response—calm diagnosis, targeted action, and verification—is the gold standard.

Moreover, the message highlights a tension in infrastructure-as-code development: the desire for fast iteration (via caching) versus the need for deterministic results (via clean builds). The assistant chose --no-cache, sacrificing speed for correctness. In a production CI/CD pipeline, this trade-off would be unacceptable for every build, but in a debugging session, it's exactly the right call.

Conclusion

The subject message at index 1643 is a small but revealing window into the art of infrastructure debugging. It captures the moment of insight when a developer realizes that the problem isn't in their code but in their testing environment. The Docker build cache, a feature designed to accelerate development, had become a bug—silently preserving an old state and masking the effect of a fix. By recognizing this, the assistant demonstrates the kind of systems thinking that separates effective infrastructure engineering from mere scripting.

The message also serves as a reminder that in complex distributed systems, the debugging process is never linear. Each fix reveals new assumptions, each test exposes new failures, and each layer of the stack—from Ansible playbooks to Docker builds to systemd services—must be questioned. The cached image trap is just one of many pitfalls, but learning to recognize it is essential for anyone building and testing infrastructure at scale.