The pam_nologin Problem: When "System Is Running" Doesn't Mean "Ready for SSH"

In the course of building and debugging an Ansible-based deployment system for a horizontally scalable S3 storage architecture, a single message captures a moment of diagnostic insight that resolved a recurring test failure. The message, indexed at 1568 in the conversation, is deceptively brief — a few lines of shell commands and a verification step — but it represents the culmination of a multi-step debugging process that had been frustrating progress for several iterations. This article examines that message in depth: why it was written, what assumptions it corrected, and what knowledge it produced.

The Message

Here is the exact content of the subject message:

The pam_nologin module is blocking logins. The issue is that systemd reports running but the nologin is still in place. Let me manually remove them and also fix the Dockerfile to not have this issue: [bash] cd /home/theuser/gw/ansible/test/docker && for host in kuri-01 kuri-02 s3-fe-01; do docker exec "ansible-test-$host" rm -f /run/nologin /var/run/nologin done && docker compose exec -T ansible-controller bash -c "sshpass -p [REDACTED] ssh -o StrictHostKeyChecking=no fgw@172.28.0.21 'echo SSH works'" 2>&1 SSH works

The Context: Building an Ansible Test Harness

To understand why this message was written, one must understand the broader project. The assistant was building Ansible deployment scripts for FGW (Filecoin Gateway) clusters — a distributed, horizontally scalable S3 storage system. The architecture involved three layers: stateless S3 frontend proxies, Kuri storage nodes, and a YugabyteDB database. The assistant had created a comprehensive Docker-based test harness to validate the Ansible playbooks before they would be used in production.

The test environment consisted of several Docker containers: an Ansible controller container (the machine that runs playbooks), three target hosts (two Kuri nodes and one S3 frontend) running Ubuntu 24.04 with systemd and SSH servers, and a YugabyteDB container. The test script would run a series of playbook tests: connectivity checks, database initialization, Kuri node deployment, and S3 frontend deployment.

The problem was that the test suite kept failing at the very first step — the connectivity check. The Ansible controller could not SSH into the target hosts. The error message was consistent: "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."

The Debugging Trail: What Led to This Message

The preceding messages (1541 through 1567) show a classic debugging progression. Initially, the target containers were unreachable because they were still booting — the SSH daemon hadn't started yet. The assistant added a sleep and a systemd wait (systemctl is-system-running --wait). Then there was a Python interpreter issue on the YugabyteDB host, which was excluded from the ping test. Then the pam_nologin error appeared.

The assistant's first response to the pam_nologin error was to wait for systemd to finish booting. In message 1566, they ran systemctl is-system-running --wait on each target host, and all three reported "running." Confident that the system was ready, they ran the tests again. But in message 1567, the same pam_nologin error returned. Systemd said "running," but SSH logins were still blocked.

This is the critical moment that led to message 1568. The assistant realized that systemd reporting "running" and the pam_nologin file being removed are not perfectly synchronized events. There is a gap — a window of time where systemd considers itself "running" but the /run/nologin file (which the pam_nologin module checks before allowing logins) has not yet been cleaned up. The assistant's insight was that waiting for systemd state was not sufficient; they needed to directly ensure the nologin file was absent.## The Root Cause: pam_nologin and Systemd Boot Sequencing

The pam_nologin module is a Pluggable Authentication Module (PAM) that checks for the existence of /run/nologin or /var/run/nologin. During system boot, systemd creates this file to prevent non-root users from logging in while the system is still initializing services. Once all services have started and the system reaches a stable state, systemd removes the file, allowing normal user logins.

The subtlety that the assistant identified is that systemctl is-system-running returns "running" before the nologin file is removed. There is a race condition in the boot sequence: systemd considers the system "running" when all units have been started, but the removal of the nologin file happens as part of a late boot process that can lag behind the systemd state transition. This means that waiting for is-system-running to return "running" is not a reliable indicator that SSH logins will succeed.

The assistant's decision to manually remove the nologin files was a pragmatic workaround for the test environment. Rather than trying to fix the boot sequencing in the Docker containers (which would require modifying systemd configuration or the Dockerfile), they chose to directly delete the blocking files. This approach is appropriate for a test harness where the goal is to validate Ansible playbooks, not to test systemd boot behavior.

Assumptions Made and Corrected

Several assumptions were challenged in this debugging process. The first assumption was that systemd's "running" state would be sufficient for SSH access. This turned out to be incorrect — there is a gap between systemd reporting "running" and the pam_nologin file being removed. The second assumption was that waiting for systemd to finish booting would eliminate the nologin file. The assistant's repeated attempts (messages 1564, 1566, 1567) showed that even after systemd reported "running," the nologin file persisted.

Another implicit assumption was that the Docker containers' boot process would complete quickly and deterministically. In practice, the boot time varied, and the nologin file could remain for an unpredictable duration after systemd reported "running." This variability made the test suite unreliable — sometimes it would pass, sometimes it would fail, depending on timing.

The assistant also assumed that the fix should be applied both immediately (by manually removing the files) and permanently (by fixing the Dockerfile). The message mentions "also fix the Dockerfile to not have this issue," indicating an understanding that the workaround should be made permanent so future test runs don't encounter the same problem.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains. Linux system administration knowledge is essential: understanding what pam_nologin is, how it works, and where the nologin file is located (/run/nologin and /var/run/nologin). Knowledge of systemd's boot sequence and state reporting is also required — specifically, the distinction between systemd considering itself "running" and all boot processes being complete.

Docker knowledge is needed to understand the test infrastructure: how Docker containers are started, how volumes are mounted, and how docker exec can be used to run commands inside containers. Ansible knowledge helps understand why SSH connectivity is the first test — Ansible uses SSH to execute tasks on remote hosts, so if SSH doesn't work, nothing else will.

The broader architectural context is also important: the three-layer S3 architecture (S3 proxies, Kuri storage nodes, YugabyteDB), the Ansible roles and playbooks being tested, and the test harness structure. Without this context, the message appears to be about a trivial SSH configuration issue, when in fact it was a critical blocker for validating the entire deployment system.## Output Knowledge Created

This message produced several valuable pieces of knowledge. First, it established a reliable workaround for the pam_nologin issue in Docker-based test environments: manually removing /run/nologin and /var/run/nologin after systemd reports "running." This is a reusable technique that can be applied to any Docker container running systemd where SSH access is needed for testing.

Second, the message documented the specific locations of the nologin file on Ubuntu 24.04. The assistant checked both /run/nologin and /var/run/nologin, covering the common locations where systemd places this file. This is useful information for anyone debugging similar SSH login issues.

Third, the verification step — using sshpass to SSH into one of the target hosts and run echo SSH works — confirmed that the fix was effective. This created a clear, repeatable verification procedure that could be incorporated into the test harness setup script.

Fourth, the message created the knowledge that the Dockerfile needed to be modified to prevent this issue from recurring. While the message doesn't show the Dockerfile change itself, it establishes the requirement and the direction of the fix. The assistant understood that a permanent solution was needed, not just a manual workaround.

The Thinking Process: Diagnostic Reasoning in Action

The thinking process visible in this message is a textbook example of diagnostic reasoning. The assistant started with a symptom (SSH connection failures with pam_nologin errors), formed a hypothesis (the nologin file is blocking logins even though systemd reports running), tested the hypothesis (by manually removing the files and attempting SSH), and confirmed the fix worked.

What's particularly interesting is the progression of hypotheses across the preceding messages. The assistant initially thought the issue was that the containers were still booting (message 1541). When that didn't fully resolve the problem, they considered that the Python interpreter was missing (message 1546). When the pam_nologin error appeared, they first tried waiting for systemd (message 1566). Only when that failed did they arrive at the correct diagnosis: the nologin file persists after systemd reports "running."

This progression shows a narrowing of hypotheses based on evidence. Each failed attempt eliminated one possible cause and pointed toward the next. The assistant's willingness to try different approaches — from waiting, to installing packages, to modifying the test inventory, to directly manipulating files — demonstrates a flexible and persistent debugging style.

Mistakes and Incorrect Assumptions

The primary mistake was relying on systemctl is-system-running as a proxy for SSH readiness. This is a reasonable assumption — systemd's "running" state should mean the system is fully operational — but it proved incorrect in practice. The assistant learned that there is a gap between systemd's state machine and the actual removal of the nologin file.

A secondary issue was the repeated attempts to fix the problem through the test harness rather than addressing the root cause directly. The assistant spent several iterations (messages 1541 through 1567) working around the issue — modifying the inventory, installing packages, waiting for boot — before finally identifying the nologin file as the direct cause. This is a natural part of debugging, but it highlights the value of understanding the underlying mechanism rather than treating symptoms.

Another subtle assumption was that the Docker containers would behave identically to physical or virtual machines in their boot sequencing. Docker containers running systemd can have quirks in how boot processes are handled, especially around PAM modules and login management. The assistant's eventual fix — directly removing the nologin file — is a pragmatic acknowledgment that the test environment has different characteristics than production.

Broader Implications for Infrastructure Testing

This message, while focused on a specific technical issue, illustrates several broader principles about infrastructure testing. First, test harnesses must account for the full lifecycle of the systems they test, including boot sequencing and service readiness. A test that assumes a system is ready based on one indicator (systemd state) may fail if another indicator (PAM login state) is not aligned.

Second, Docker-based test environments, while convenient, can introduce subtle differences from production environments. Systemd in Docker containers may not behave identically to systemd on bare metal or virtual machines. Test harnesses need to account for these differences or use alternative approaches (such as running SSH servers without PAM restrictions in test containers).

Third, the most effective debugging often involves understanding the mechanism behind the symptom. The assistant could have continued trying to work around the pam_nologin error — increasing wait times, retrying connections, or modifying Ansible settings — but identifying the specific file that was blocking logins allowed for a direct, targeted fix. This is the difference between treating symptoms and curing the disease.

Conclusion

Message 1568 is a small but instructive moment in a larger engineering effort. It captures the moment when a frustrating, recurring test failure was finally understood and resolved. The assistant's diagnosis — that the pam_nologin file persists after systemd reports "running" — required both system administration knowledge and careful observation of the evidence. The fix — manually removing the nologin files and planning a permanent Dockerfile change — was pragmatic and effective.

For anyone building Ansible deployment systems, Docker-based test harnesses, or infrastructure automation, this message offers a valuable lesson: trust but verify. Systemd may say "running," but that doesn't mean SSH will work. Sometimes the most reliable test is the simplest one: try to connect and see what happens.