The Moment of Reckoning: Restarting Containers After a Debugging Marathon

Introduction

In the trenches of infrastructure-as-code development, there comes a moment when all the fixes have been applied, the configuration files have been corrected, the Docker images have been rebuilt, and the only thing left to do is recreate the containers and hope everything works. Message 1644 in this coding session captures exactly that moment — a single docker compose up --force-recreate command that represents the culmination of an intense, iterative debugging session spanning dozens of messages and touching nearly every layer of a distributed S3 storage deployment pipeline.

The message itself is deceptively simple:

[assistant] [bash] cd /home/theuser/gw/ansible/test/docker && docker compose up -d kuri-01 kuri-02 s3-fe-01 --force-recreate 2>&1
time="2026-01-31T18:58:04+01:00" level=warning msg="No services to build"
 Container ansible-test-yb Running 
 Container ansible-test-kuri-02 Recreate 
 Container ansible-test-kuri-01 Recreate 
 Container ansible-test-kuri-01 Recreated 
 Container ansible-test-kuri-02 Recreated 
 Container ansible-test-s3-fe-01 Recreate 
 Container ansible-test-s3-fe-01 Recreated 
 Container ansible-test-yb Waiting 
 Container ansible-test-yb Waiting 
 Container ansible-test-yb Healthy

A developer runs a Docker Compose command, three containers are recreated, the YugabyteDB database container remains healthy. On the surface, it looks routine. But to understand why this message matters — why it was written at all — requires tracing the tangled thread of bugs, misconfigurations, and architectural surprises that led to this precise invocation.

The Debugging Journey That Preceded This Command

The context leading up to message 1644 is a story of incremental discovery. The assistant had been building and debugging Ansible playbooks for deploying a Filecoin Gateway (FGW) cluster — a horizontally scalable S3-compatible storage system with Kuri storage nodes, stateless S3 frontend proxies, and a YugabyteDB backend. The test harness used Docker containers running systemd to simulate real Ubuntu servers, with Ansible running from a controller container to deploy software onto these target hosts.

What began as a straightforward test run had unraveled into a cascade of failures, each revealing a subtle but critical issue:

The environment file format problem. Systemd's EnvironmentFile directive silently rejects files containing export prefixes, but the Ansible template settings.env.j2 was generating lines like export LOG_LEVEL=debug. The error was invisible — systemd would simply refuse to load the file, and the service would start with default values.

The log level regex mismatch. The configuration used *:* as a log level pattern, but the application expected the standard logging syntax .*:.*=debug. This caused the service to ignore the configured log level entirely.

The wallet dotfile trap. The test wallet directory contained a .gitkeep file (a common trick to keep empty directories in Git). When the Ansible wallet role copied this directory to the target hosts, it included the hidden dotfile, which the Kuri binary then tried to parse as a wallet — failing with a binary format error.

The duplicate migration conflict. The yugabyte_init Ansible role was creating CQL tables (like S3Objects) during database initialization, but the Kuri application also ran its own migrations on startup. The result was a CREATE TABLE error when Kuri tried to create a table that already existed — a race condition between infrastructure provisioning and application initialization.

The missing Ansible filter. The s3_frontend role referenced a custom Jinja2 filter called format_backend_url that didn't exist in the Ansible environment, causing playbook failures during S3 proxy configuration.

The pam_nologin blockade. Most critically, the Docker target containers ran systemd, which creates a /run/nologin file during boot. SSH daemon (via PAM) reads this file and refuses login for non-root users — meaning Ansible's SSH-based connectivity checks would fail with the message "System is booting up. Unprivileged users are not permitted to log in yet." This was the showstopper that prevented any playbook from executing.## The Specific Fixes That Made This Command Necessary

Each of these bugs had been diagnosed and fixed in the messages immediately preceding message 1644. The assistant had:

  1. Removed export prefixes from the settings.env.j2 template so systemd could parse it.
  2. Changed the log level format from *:* to .*:.*=debug.
  3. Updated the wallet role to exclude dotfiles (files starting with .) during the copy task.
  4. Stripped table creation from the yugabyte_init role, leaving only keyspace creation, so that Kuri's own migrations could run without conflict.
  5. Fixed the s3_frontend role to use a valid backend URL format.
  6. Added systemctl disable systemd-user-sessions.service to the Dockerfile to prevent the pam_nologin blockade. But fixing the Dockerfile wasn't enough — the Docker images had been built with the old configuration and were cached. The assistant had already tried running docker compose build --no-cache to force a rebuild of the three target images (kuri-01, kuri-02, s3-fe-01). Message 1644 shows the next logical step: recreating the containers from those freshly built images. The --force-recreate flag is the key detail here. Without it, Docker Compose would see that containers already exist and simply start the existing ones — which would still have the old pam_nologin issue baked in. The flag ensures that Docker discards the old container instances and creates new ones from the updated images, picking up the systemd service fix.

What the Output Reveals

The output tells a story of its own. The warning "No services to build" appears because the command only requested up (start containers), not build. The images were already built in a previous step. The three containers are sequentially recreated: kuri-01, kuri-02, s3-fe-01. The YugabyteDB container (ansible-test-yb) is already running and healthy — it wasn't touched because it wasn't in the service list and doesn't need rebuilding.

The "Waiting" status for the YugabyteDB container is Docker Compose's dependency resolution: the S3 frontend and Kuri nodes depend on the database being healthy, so Compose waits for the health check to pass before considering those services fully started. The final "Healthy" line confirms the database is ready.

The Assumptions Embedded in This Command

This message makes several implicit assumptions that are worth examining:

Assumption 1: The image rebuild was successful. The assistant had run docker compose build --no-cache in message 1643, but the output was truncated. The assumption here is that the build completed without errors and the new images contain the systemd-user-sessions.service fix. This is a reasonable assumption given the partial output showed image export succeeding, but it's not verified.

Assumption 2: The pam_nologin fix is sufficient. Disabling systemd-user-sessions.service should prevent the creation of /run/nologin, but there could be other systemd components that create this file. The assistant is betting that this single change resolves the SSH login blockade.

Assumption 3: No other bugs remain. The assistant had fixed six distinct issues, but there's no guarantee that fixing these six reveals all the bugs. New errors could surface once the connectivity blockade is removed.

Assumption 4: The container recreation is safe. Using --force-recreate with -d (detached mode) means the assistant won't see real-time logs. If a container fails to start, the error would need to be caught separately with docker compose logs.

Input Knowledge Required to Understand This Message

To fully grasp what's happening here, a reader needs to understand:

Output Knowledge Created

This message doesn't produce a deliverable in the traditional sense — no configuration file, no code change, no documentation. What it produces is a state: a running test cluster with freshly recreated containers that should (hopefully) accept SSH connections. The output confirms:

The Thinking Process Behind the Command

The reasoning visible in the surrounding messages shows a developer working through a systematic debugging process:

  1. Observe failure: The connectivity check fails with "System is booting up."
  2. Diagnose root cause: The /run/nologin file exists, created by systemd during boot.
  3. Attempt quick fix: Remove the file manually with rm -f. This works temporarily but won't persist across container restarts.
  4. Implement permanent fix: Add systemctl disable systemd-user-sessions.service to the Dockerfile.
  5. Rebuild images: Use --no-cache to ensure the fix is baked into fresh images.
  6. Recreate containers: Use --force-recreate to discard old container instances that still have the nologin problem. The progression from manual intervention (step 3) to permanent fix (step 4) to clean restart (steps 5-6) is characteristic of mature infrastructure debugging. The assistant doesn't just work around the problem — they trace it to its source and eliminate it at the configuration level.

What Happens Next

The messages following 1644 (visible in the context) show the test run proceeding: the connectivity check passes, the YugabyteDB initialization succeeds, the Kuri nodes deploy and pass health checks, and the S3 frontend deploys successfully. The entire pipeline works. The commit 806c370 with 19 file changes captures all the fixes.

But message 1644 itself is the turning point — the moment when debugging stops and validation begins. It's the breath before the plunge, the restart that could either reveal a clean pipeline or surface a new layer of bugs. In infrastructure work, these moments are where the quality of the preceding diagnosis is tested. If the fixes were correct, the containers come up clean. If something was missed, the errors start all over again.

Conclusion

Message 1644 is a study in what makes infrastructure-as-code debugging unique: the long chain of dependencies, the subtle interactions between system components (systemd, PAM, SSH, Docker, Ansible), and the need to trace failures across multiple layers of abstraction. A single docker compose up --force-recreate command looks trivial in isolation, but in context, it represents the resolution of hours of detective work — a developer's bet that they've found and fixed all the hidden traps in the deployment pipeline.

The message is also a reminder that in complex systems, the most important commands are often the simplest ones: restart, retry, recreate. The sophistication lies not in the command itself, but in knowing when to run it and what to expect when you do.