The Ten-Second Wait That Wasn't Enough: Debugging pam_nologin in Docker-Based Ansible Testing

The Message

[assistant] The system is still booting. There's a `pam_nologin` check that prevents login during boot. Let me wait a bit and retry:
[bash] sleep 10 && cd /home/theuser/gw/ansible/test && ./run-tests.sh 2>&1
=== FGW Ansible Playbook Tests ===


=== Test 1: Connectivity Check ===
[ERROR]: Task failed: Data could not be sent to remote host "172.28.0.22". Make sure this host can be reached over ssh: Warning: Permanently added '172.28.0.22' (ED25519) to the list of known hosts.
"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
Connection closed by 172.28.0.22 port 22
Origin: <adhoc 'ping' task>

{'action': 'ping...

At first glance, this message from a coding session appears to be a trivial moment—a simple retry after a brief pause. But in the context of infrastructure-as-code debugging, this message represents a critical inflection point. It is the moment when an engineer, having already fixed half a dozen subtle and frustrating bugs in an Ansible deployment pipeline, encounters a new class of problem that cannot be solved by waiting. This message captures the transition from configuration-level debugging to system-level debugging, and it reveals deep truths about the nature of infrastructure testing, the assumptions we make about containerized environments, and the hidden complexity of simulating production systems in Docker.

The Context: A Pipeline Under Siege

To understand why this message was written, we must first understand the journey that led to it. The assistant had been building and debugging an automated deployment pipeline for the Filecoin Gateway (FGW) project—a horizontally scalable S3-compatible storage system built on a distributed architecture of stateless S3 frontend proxies, Kuri storage nodes, and a YugabyteDB backend. The Ansible playbooks were designed to deploy this cluster across multiple nodes, handling everything from binary distribution and wallet creation to database initialization and service management.

The debugging session had been relentless. The assistant had already diagnosed and fixed:

  1. Systemd EnvironmentFile syntax: The settings.env.j2 template used export prefixes, which systemd's EnvironmentFile directive rejects. The fix was to remove the export keywords.
  2. Invalid log level format: The configuration used *:*=debug as a log level pattern, but the application's regex parser requires .*:.*=debug. The asterisk is not a valid regex character without a preceding dot.
  3. Wallet dotfile contamination: The wallet directory contained a .gitkeep file that was being copied to the target nodes and parsed as a wallet key file, causing binary parsing errors. The fix involved excluding dotfiles from the copy operation.
  4. Duplicate CQL table creation: Both the yugabyte_init role and kuri init attempted to create database tables, causing conflicts. The fix was to remove table creation from the init role.
  5. Non-existent Ansible filter: The s3_frontend role used a filter called format_backend_url that didn't exist. Each fix had been applied, the Docker images rebuilt, and the test harness re-run. The assistant had just finished cleaning up the test environment, rebuilding the Docker containers, and verifying that binaries were properly installed. The stage was set for a triumphant full test run.

The First Failure and the Reasoning Behind the Retry

When the assistant first ran the test suite after all the fixes, the connectivity check failed with the pam_nologin error (visible in message 1626, the immediate predecessor to our subject message). The error message was clear: "System is booting up. Unprivileged users are not permitted to log in yet."

The assistant's response in our subject message reveals a specific reasoning process. The key phrase is: "The system is still booting. There's a pam_nologin check that prevents login during boot. Let me wait a bit and retry."

This reasoning rests on a reasonable assumption: that the Docker containers running systemd are in a transient boot state, and that given enough time, the boot process will complete, the pam_nologin file will be automatically removed by systemd, and SSH logins will succeed. In a normal Linux system, this is exactly what happens—systemd creates /run/nologin during early boot and removes it once boot is complete. The pam_nologin module checks for the existence of this file and refuses login if it exists.

The decision to wait 10 seconds is a classic debugging heuristic: before diving into complex root cause analysis, try the simplest possible intervention. It's the digital equivalent of "turn it off and on again." In many cases, transient timing issues resolve themselves with a brief wait. The assistant had already experienced the frustration of chasing phantom bugs that turned out to be timing-related—the earlier chmod error during binary copying turned out to be a race condition where docker cp hadn't finished before chmod ran.

The Assumption That Failed

The 10-second wait was not enough. The error persisted, identical in every detail except the target IP address (172.28.0.22 instead of 172.28.0.31 from the previous run, reflecting a different container in the cluster). This persistence is deeply informative. It tells us that the pam_nologin condition is not a transient boot delay but a persistent state. Something in the Docker container configuration is causing systemd to either never complete its boot sequence, or to leave the nologin file in place indefinitely.

The assistant's assumption—that the boot process is merely slow—turns out to be incorrect. The real issue is more fundamental. In a normal Ubuntu system, systemd removes /run/nologin as part of its boot sequence, specifically when the systemd-user-sessions service starts. But in a Docker container running systemd, the boot sequence can be incomplete or misconfigured. The systemd-user-sessions service may not start at all, or it may fail silently, leaving the nologin file in place permanently.

This is a classic pitfall of running systemd in containers. Systemd is designed to manage the full lifecycle of a physical or virtual machine, and it makes assumptions about hardware, initramfs, and service ordering that don't hold in containerized environments. The pam_nologin mechanism is a security feature designed to prevent logins during the boot process, but in a container that never fully "boots" in the traditional sense, it becomes a permanent barrier.

Input Knowledge Required

To understand this message fully, one needs knowledge spanning several domains:

Systemd internals: Understanding that pam_nologin checks for /run/nologin (or /var/run/nologin) and that this file is created during early boot and removed by systemd-user-sessions.service later in the boot sequence.

Docker container behavior with systemd: Knowing that running systemd as PID 1 inside a Docker container is a common pattern for testing infrastructure code, but that it comes with quirks—including incomplete boot sequences, missing dependencies, and services that don't start properly.

SSH authentication flow: Understanding that the SSH server uses PAM (Pluggable Authentication Modules) for authentication, and that pam_nologin.so is one of the modules in the auth stack. When it detects /run/nologin, it refuses all non-root logins.

Ansible connection mechanics: Knowing that Ansible uses SSH to connect to target hosts and run tasks. The first test in the suite is a simple "ping" module that verifies SSH connectivity. If SSH fails, the entire pipeline stops.

The Docker test harness architecture: Understanding that the test environment consists of multiple Docker containers—an Ansible controller (running Python and Ansible), two Kuri storage nodes (running systemd with SSH), and an S3 frontend proxy node (also running systemd). The containers are connected via a Docker network, and Ansible connects to them over SSH.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. The nologin error is persistent, not transient: Waiting does not resolve the issue. This eliminates the simplest possible fix and forces deeper investigation.
  2. The fix must be applied at the container level: Since the nologin file persists across the container's lifetime, the solution must involve either removing the file after boot or preventing its creation in the first place.
  3. The Dockerfile needs modification: The proper fix is to add a command to the Dockerfile that either removes the nologin file after systemd starts, or disables the systemd-user-sessions service that creates it.
  4. The test harness has a systemic weakness: The Docker containers are not fully simulating production behavior. In production, servers boot once and stay up. In the test harness, containers are restarted frequently, and each restart triggers a fresh boot sequence with the nologin barrier.

The Thinking Process Revealed

The assistant's thinking process in this message is a textbook example of systematic debugging. The sequence is:

  1. Observe the error: The connectivity check fails with a specific error message about pam_nologin.
  2. Interpret the error: The assistant recognizes that pam_nologin is a boot-time check and correctly identifies that the system is in a booting state.
  3. Form a hypothesis: The system needs more time to complete booting. The nologin file will be removed automatically when boot completes.
  4. Test the hypothesis: Wait 10 seconds and retry the full test suite.
  5. Evaluate the result: The error persists. The hypothesis is rejected. This is the scientific method applied to debugging. The assistant doesn't jump to conclusions or try random fixes. They formulate a specific, testable hypothesis and design a minimal experiment to test it. When the experiment fails, they have new information: the problem is not timing-related. The choice to wait 10 seconds specifically is also telling. It's long enough to account for normal boot delays but short enough to maintain momentum. In a debugging session where each iteration can take minutes (rebuilding Docker images, re-running tests), a 10-second wait is a negligible investment for potentially valuable information.

What Follows

The messages immediately after our subject message show the assistant pivoting to the correct solution. In message 1628, the assistant checks for the nologin file directly:

[bash] cd /home/theuser/gw/ansible/test/docker && \
docker compose exec -T kuri-01 ls -la /var/run/nologin /run/nologin 2>&1 && \
docker compose exec -T kuri-01 rm -f /var/run/nologin /run/nologin 2>&1 && \
docker compose exec -T kuri-02 rm -f /var/run/nologin /run/nologin 2>&1 && \
docker compose exec -T s3-fe-01 rm -f /var/run/nologin /run/nologin 2>&1 && \
echo "Removed nologin files"

This confirms the file exists and removes it manually. Then in message 1630, the assistant identifies the root cause—the systemd-user-sessions.service creates the nologin file—and adds a fix to the Dockerfile to disable that service. The final fix is to rebuild the Docker images with systemd-user-sessions disabled, preventing the nologin file from being created at all.

The Deeper Lesson

This message, for all its apparent simplicity, teaches a profound lesson about infrastructure testing: containers are not virtual machines. When you run systemd inside a Docker container to simulate a production server for Ansible testing, you inherit all the complexity of systemd's boot sequence without the guarantees that come with a full VM or physical server. The boot process may be incomplete, services may fail silently, and security mechanisms designed for real hardware can become permanent obstacles.

The assistant's 10-second wait was not a mistake—it was a necessary step in the diagnostic process. Every experienced engineer has learned the hard way that sometimes the simplest fix (waiting, restarting, re-running) actually works. The skill is knowing when to abandon that approach and dig deeper. This message captures the exact moment when the assistant makes that transition, and it's a beautiful example of disciplined, methodical debugging in action.