The pam_nologin Wall: When Systemd Says "Running" But SSH Says "Not Yet"

In the middle of a long and increasingly frustrating debugging session, a developer runs a test script and watches it fail — again. The error message is familiar to anyone who has worked with systemd-based containers, but it still stings: "System is booting up. Unprivileged users are not permitted to log in yet." This single message, exchanged between an AI coding assistant and its human partner, captures a moment where infrastructure automation meets the messy reality of Linux boot sequences, and where a subtle race condition in systemd's state machine derails an otherwise carefully constructed test harness.

The Message

The message in question is a straightforward bash command execution:

[assistant] [bash] cd /home/theuser/gw/ansible/test && ./run-tests.sh 2>&1
=== FGW Ansible Playbook Tests ===


=== Test 1: Connectivity Check ===
[ERROR]: Task failed: Data could not be sent to remote host "172.28.0.21". Make sure this host can be reached over ssh: Warning: Permanently added '172.28.0.21' (ED25519) to the list of known hosts.
"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
Connection closed by 172.28.0.21 port 22
Origin: <adhoc 'ping' task>

{'action': 'ping...

The assistant runs the test suite, which begins with an Ansible connectivity check (essentially a ping module test against the target hosts). The test fails because the SSH server on host 172.28.0.21 (one of the Kuri storage nodes in the Docker test cluster) rejects the connection with a pam_nologin error.

The Context: Building an Ansible Deployment Test Harness

To understand why this message matters, we need to step back and look at the larger context. The project is building a horizontally scalable S3 storage system with a three-layer architecture: stateless S3 frontend proxies, Kuri storage nodes, and a YugabyteDB backend. The assistant has been working on Ansible deployment automation — writing roles, playbooks, and inventory configurations to automate the setup of production clusters.

The current segment focuses on validating these Ansible scripts using a Docker-based test harness. The test environment consists of:

Why This Message Was Written: The Iteration Cycle

The message is the latest attempt in a cycle of "fix, run, fail, diagnose, fix again." In the immediately preceding message (index 1566), the assistant ran a command to wait for systemd to finish booting on all three target hosts:

Waiting for kuri-01...
running
Waiting for kuri-02...
running
Waiting for s3-fe-01...
running
All hosts ready

Satisfied that the hosts were fully booted, the assistant then ran the test suite. The assumption was clear: if systemd reports "running," then the system is ready for SSH connections. This assumption turned out to be incorrect.

The error message reveals that the SSH server is still enforcing pam_nologin, a Pluggable Authentication Module (PAM) that prevents non-root logins while the system is booting. The module checks for the existence of /run/nologin (or /etc/nologin) and refuses authentication if the file is present. This mechanism exists to prevent users from logging in before the system has completed its startup sequence — imagine someone logging in and starting processes before critical services or filesystems are ready.

The Incorrect Assumption: Systemd "Running" ≠ SSH Ready

The core mistake here is subtle but important. The assistant assumed that systemctl is-system-running --wait returning "running" was a sufficient condition for SSH accessibility. In reality, there is a gap between systemd declaring the system as "running" and the removal of the pam_nologin guard.

The boot sequence works roughly like this:

  1. Early boot: systemd creates /run/nologin to block logins
  2. System initialization: services start, filesystems mount, network comes up
  3. Late boot: systemd removes /run/nologin and transitions to the "running" state
  4. Post-boot: SSH logins are permitted The race condition occurs because systemd can report "running" before the nologin file is cleaned up in all cases, or the file may be recreated by another process. In Docker containers running systemd, the timing can be especially unpredictable because the container's init process may not follow the exact same boot sequence as a physical machine. Furthermore, the assistant had already encountered a similar issue earlier in the session (message 1547), where they checked for and removed nologin files on the target hosts. At that time, the files didn't exist, but the SSH connection was still failing because the hosts hadn't fully booted. After waiting for systemd to report "running," the assistant assumed the problem was resolved — but the pam_nologin issue resurfaced because the containers had been recreated (via docker compose down &amp;&amp; docker compose up -d in message 1562), starting a fresh boot cycle.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several interconnected systems:

Systemd boot sequencing: Understanding that is-system-running reports the overall system state but doesn't guarantee that all boot-time guards have been removed. The pam_nologin module is part of the PAM stack and operates independently of systemd's state machine.

PAM and SSH authentication flow: SSH uses PAM for authentication, and the pam_nologin module is typically configured to deny access if /run/nologin exists. This is a deliberate safety mechanism, not a bug.

Docker with systemd: Running systemd inside Docker containers is a common pattern for testing infrastructure automation, but it introduces complexities. The container's PID 1 is systemd, which manages the boot sequence, but the container environment may not perfectly replicate a physical or virtual machine's boot process.

Ansible connectivity testing: The ping module in Ansible tests not just network connectivity but also the ability to execute Python on the remote host. A failure at the SSH layer (before Ansible can even attempt to run Python) produces the kind of error seen here.

The project architecture: Understanding that the test cluster has three target hosts (kuri-01, kuri-02, s3-fe-01) with static IPs, and that the test suite runs sequentially through connectivity checks, database initialization, and service deployment.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. A clear diagnostic signal: The error message explicitly identifies pam_nologin as the cause. The text "System is booting up. Unprivileged users are not permitted to log in yet. For technical details, see pam_nologin(8)." is the SSH server's direct output, not an Ansible abstraction. This makes the root cause unambiguous.
  2. A timing constraint: The message establishes that waiting for systemctl is-system-running --wait is not sufficient. The test harness needs either a longer delay, a manual removal of the nologin files, or a fix in the Docker image to prevent the issue from occurring.
  3. A pattern of recurring issues: The fact that this is the second time pam_nologin has appeared (the first was in message 1541, though with a different manifestation) suggests that the Docker images need a permanent fix rather than a runtime workaround.
  4. A boundary condition: The message reveals the edge case where systemd's "running" state and SSH accessibility diverge. This is valuable knowledge for anyone building systemd-based Docker containers for testing.

The Immediate Aftermath

The assistant's response to this failure (message 1568) shows clear understanding of the cause:

The pam_nologin module is blocking logins. The issue is that systemd reports running but the nologin is still in place. Let me manually remove them and also fix the Dockerfile to not have this issue.

The assistant then removes /run/nologin and /var/run/nologin from all three target hosts, verifies that SSH works, and re-runs the test suite. This time, the connectivity check passes for all three hosts, and the test progresses to the next stage (YugabyteDB initialization).

However, the deeper lesson is that the Docker images themselves should be fixed to prevent this issue. The nologin file is created during early boot and should be cleaned up automatically, but in the container environment, something is preventing the cleanup from happening reliably. A proper fix would involve modifying the Dockerfile to either remove the nologin file after systemd starts, configure systemd to skip the nologin mechanism, or add a startup script that ensures the file is removed before SSH is made available.

Broader Implications for Infrastructure Testing

This single error message illustrates a broader challenge in infrastructure automation: the gap between "the system is running" and "the system is ready." In production deployments, similar issues appear with database readiness (a PostgreSQL container might report "running" before it's ready to accept connections), load balancer health checks, and service mesh endpoints.

The pam_nologin issue is a small but instructive example of why test harnesses need to verify behavioral readiness rather than process state. Checking that systemd reports "running" is checking process state. Verifying that an SSH connection succeeds and can execute a command is checking behavioral readiness. The latter is always more reliable, but it's also slower and more complex to implement.

The assistant's iterative approach — fix one issue, run the tests, find the next issue, fix again — is characteristic of infrastructure development. Each failure reveals a new assumption that was incorrect, and each fix strengthens the overall system. The pam_nologin message is just one step in this chain, but it's a step that teaches an important lesson about the difference between "booted" and "ready."