The Systemd Boot Race: Debugging Container SSH Failures in Ansible Test Infrastructure

Introduction

In the course of building a complex distributed storage system called the Filecoin Gateway (FGW), a developer encountered a frustrating but instructive problem: SSH connections to Docker containers were being rejected with the message "System is booting up. Unprivileged users are not permitted to log in yet." The message under analysis—a single Bash command issued by an AI coding assistant—represents a precise, minimal fix to this infrastructure timing problem. While the command itself is only a few lines, the reasoning behind it reveals a deep understanding of Linux boot sequencing, systemd's state machine, Docker container lifecycle, and the subtle ways these systems interact during automated testing.

The message reads:

Systemd is still booting. Let me wait longer and retry: ``bash cd /home/theuser/gw/ansible/test/docker && for host in kuri-01 kuri-02 s3-fe-01; do echo "Waiting for $host..." docker exec "ansible-test-$host" systemctl is-system-running --wait || true done && echo "All hosts ready" 2>&1 ` Output: ` Waiting for kuri-01... running Waiting for kuri-02... running Waiting for s3-fe-01... running All hosts ready ``

This article examines why this message was written, what assumptions it makes, what knowledge it requires and produces, and what it reveals about the challenges of testing infrastructure-as-code in containerized environments.

The Context: Ansible Deployment Testing with Docker

To understand this message, one must understand the broader project. The developer was building Ansible playbooks to automate the deployment of a horizontally scalable S3-compatible storage cluster called the Filecoin Gateway. This cluster has three layers: stateless S3 frontend proxies, Kuri storage nodes, and a YugabyteDB database. The Ansible scripts needed to be tested before they could be used on real hardware, so a Docker-based test harness was created.

The test harness consisted of:

Why This Message Was Written

The immediate trigger was a failed test run in the previous message (index 1565). The Ansible connectivity check had produced this error:

[ERROR]: Task failed: Data could not be sent to remote host "172.28.0.22".
Make sure this host can be reached over ssh:
Warning: Permanently added '172.28.0.22' (ED25519) to the list of known hosts.
"System is booting up. Unprivileged users are not permitted to log in yet.
Please come back later. For technical details, see pam_nologin(8)."
Connection closed by 172.28.0.22 port 22

This error occurs because systemd, during early boot, creates a file called /run/nologin (or /etc/nologin on some systems). The Pluggable Authentication Modules (PAM) framework checks for this file during SSH authentication. If it exists, non-root users are denied login with the message "System is booting up." This is a security feature designed to prevent services from starting before the system is fully initialized.

The developer had already encountered this issue earlier (message 1547) and attempted a workaround: manually removing the nologin files with rm -f /run/nologin /etc/nologin. At that time, the files didn't exist yet, so the workaround appeared to succeed—but the real problem was that systemd hadn't even gotten far enough in its boot sequence to create them. The SSH connection succeeded briefly (message 1548 confirmed "SSH works") but after containers were restarted in message 1562 (to fix the read-only volume issue), the race condition returned.

The key insight in message 1566 is that the developer recognized the root cause: the containers were restarted, systemd was still booting, and the Ansible test suite was trying to connect before boot completed. Instead of another fragile workaround, the developer used systemd's own API to wait for boot completion.## The Reasoning: Why systemctl is-system-running --wait Is the Right Tool

The command chosen by the developer is elegant in its simplicity. systemctl is-system-running queries systemd's state machine and returns one of several states: running, initializing, starting, degraded, maintenance, or stopping. The --wait flag is the crucial addition—it causes the command to block until the boot process completes and the system reaches either the running or degraded state.

This approach is superior to the earlier workaround of deleting nologin files for several reasons:

  1. It is race-condition-free. Instead of guessing a sleep duration or polling for a side effect, it directly asks systemd when the boot is complete. The command will not return until the condition is met.
  2. It respects systemd's design. The nologin mechanism exists for a reason—it prevents services from starting before dependencies are ready. Bypassing it by deleting files could lead to services starting in an incomplete environment.
  3. It is portable across systemd versions. The is-system-running verb has been stable since systemd v220 (released in 2015) and is available in Ubuntu 24.04.
  4. It provides clear feedback. The loop prints "Waiting for $host..." before each check and "All hosts ready" at the end, giving the developer visibility into which containers are slow to boot. The || true at the end of the docker exec command is also a deliberate choice. If systemctl is-system-running --wait fails for some reason (e.g., the container doesn't have systemd, or the command is not found), the || true prevents the shell from aborting the entire loop with a non-zero exit code. This makes the script resilient to unexpected container states.

Assumptions Made by the Developer

Every debugging step rests on assumptions, and this message is no exception:

  1. The containers have systemd installed. This is a safe assumption because the Dockerfiles for the target hosts were built from Ubuntu 24.04 with systemd as the init system. However, it's worth noting that many minimal Docker images do not include systemd—they run a single process. The test harness explicitly chose a systemd-based image to simulate real server behavior.
  2. systemctl is-system-running --wait will eventually return. This assumes that the boot process will complete within a reasonable time. In practice, Docker containers boot much faster than physical or virtual machines, so the wait is typically a few seconds. But if a container has a broken systemd unit or a missing dependency, the command could block indefinitely. The developer's earlier check in message 1544 had already confirmed that systemd reaches the "running" state after a 10-second sleep, so this assumption is grounded in empirical evidence.
  3. The SSH service is started by systemd during boot. The pam_nologin error implies that SSH is running but PAM denies access. This means sshd starts early in the boot sequence, before systemd considers the system fully "running." The developer correctly reasoned that waiting for systemd to reach the running state would be sufficient time for SSH to accept logins.
  4. Docker exec can reach the containers. The developer uses docker exec to run the systemd check, which requires that the Docker daemon can communicate with the container. This is a reasonable assumption since the containers were just started with docker compose up -d and are in the running state from Docker's perspective (even if systemd is still booting).

Mistakes and Incorrect Assumptions

The earlier approach of deleting nologin files (message 1547) was a mistake in two ways:

  1. It treated the symptom, not the cause. The nologin file is a symptom of an incomplete boot, not the cause of the problem. Deleting it allowed SSH access temporarily, but after container restarts, the file would be recreated by systemd during the next boot.
  2. It was fragile. The developer checked for the nologin files and found they didn't exist yet (message 1547 output: ls: cannot access '/run/nologin': No such file or directory). This was because the containers had been running for a while and boot had already completed. After the containers were restarted in message 1562, the race condition returned because the nologin file was present during early boot. The developer also initially attempted a fixed 10-second sleep (message 1544: sleep 10 && for host in ...). While this worked in that specific instance, it's a brittle solution that could fail on slower machines or when containers are under load. The --wait approach is strictly better because it adapts to the actual boot time. Another subtle mistake in the earlier approach was checking systemctl is-system-running without --wait. In message 1544, the command was:
systemctl is-system-running --wait

This actually does wait, so it was correct. But the developer had added sleep 10 before it, which was unnecessary if --wait was already being used. The message 1566 version removes the sleep and relies entirely on --wait, which is cleaner.

Input Knowledge Required

To understand this message fully, a reader needs:

  1. Systemd boot sequencing. Understanding that systemd has a state machine with phases like initializing, starting, and running, and that it enforces login restrictions during early boot via PAM and the nologin file.
  2. Docker container lifecycle. Knowing that docker compose up -d returns immediately after containers start, not after they are fully booted. The container's process (systemd) may still be initializing.
  3. SSH authentication flow. Understanding that SSH uses PAM for authentication, and PAM checks for the nologin file before allowing non-root logins.
  4. Ansible's connection mechanism. Ansible connects to target hosts via SSH and runs Python modules. If SSH fails, the entire playbook fails. The connectivity test is the first step in the test suite.
  5. Bash scripting idioms. The || true pattern to suppress errors, the for loop over host names, and the && chaining to ensure sequential execution.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A reliable wait-for-boot pattern. The command systemctl is-system-running --wait is a reusable pattern for any scenario where Docker containers with systemd need to be accessed via SSH after startup. This is directly applicable to CI/CD pipelines, integration tests, and any infrastructure-as-code testing.
  2. Confirmation that all three target hosts boot correctly. The output shows "running" for each host, proving that the Docker Compose configuration, the Dockerfiles, and the systemd configurations are all correct. This is a validation milestone.
  3. A foundation for further debugging. With the SSH connectivity issue resolved, the developer can now focus on the actual Ansible playbook execution. The next test run (which would follow this message) would be able to progress past the connectivity check to the YugabyteDB initialization and Kuri deployment steps.
  4. Documentation of the race condition. By writing this fix explicitly in the conversation, the developer creates a record of the issue and its solution. This is valuable for future maintenance and for other developers working on similar infrastructure.## The Thinking Process Visible in the Message The message reveals a clear chain of reasoning:
  5. Observation: The previous test run failed with a pam_nologin error on host 172.28.0.22 (kuri-02). The error message explicitly states "System is booting up."
  6. Diagnosis: The developer immediately identifies the cause: "Systemd is still booting." This is stated plainly in the first line of the message. No additional investigation is needed because the error message is unambiguous.
  7. Solution selection: Rather than re-attempting the earlier workaround of deleting nologin files, the developer chooses to wait for systemd to complete its boot sequence. The tool systemctl is-system-running --wait is the standard way to do this.
  8. Implementation: The developer wraps this in a loop over the three target hosts, with echo statements for progress visibility. The || true handles edge cases where the command might fail.
  9. Verification: The output shows all three hosts report "running," confirming that the boot is complete and SSH should now accept connections. The thinking is notably efficient. The developer doesn't over-engineer the solution—no complex retry logic, no polling loops with sleep intervals, no fallback mechanisms. Just a direct use of systemd's own API to wait for the exact condition needed. This is a hallmark of experienced systems engineering: use the tool designed for the job rather than building a custom workaround.

Broader Implications for Infrastructure Testing

This message, while small, illustrates several important principles for testing infrastructure-as-code:

Containers are not instant. Docker containers start quickly, but "started" from Docker's perspective means the init process is running, not that the system is ready. This distinction is critical when containers run full operating systems with systemd.

Systemd is your friend, not your enemy. The nologin mechanism that caused the SSH failure is a safety feature. Working with it (by waiting for boot to complete) is better than working against it (by deleting files or disabling PAM checks).

Test harnesses must model production behavior. The decision to use systemd-based containers rather than minimal images was deliberate—it simulates real server behavior more accurately. But this fidelity comes with costs, like the boot race condition, that must be handled.

Iterative debugging converges on simple solutions. The developer's path from "delete the nologin file" to "sleep 10 seconds" to "wait for systemd" shows how each iteration reveals a better understanding of the problem. The final solution is the simplest and most robust.

Conclusion

The message at index 1566 is a small but perfect example of systems debugging at its best. A single command—systemctl is-system-running --wait—replaces a fragile workaround with a robust solution, eliminating a race condition in the Ansible test harness. The developer's reasoning shows a deep understanding of systemd's boot sequence, Docker container lifecycle, and SSH authentication, all synthesized into a minimal fix.

In the broader context of the Filecoin Gateway project, this fix was essential. Without reliable SSH connectivity, the Ansible playbooks could not be tested, and the deployment automation would remain untrusted. By solving this timing issue, the developer cleared the path for the remaining tests to run, ultimately leading to a validated deployment system for the horizontally scalable S3 architecture.

The message also serves as a reminder that in infrastructure engineering, the most effective solutions are often those that work with the system's design rather than against it. Systemd provides is-system-running --wait precisely for this use case. The developer knew to reach for it.