The pam_nologin Wall: When Systemd Blocks SSH in Docker Containers

A Single Error Message That Exposed the Fragility of Infrastructure Testing

On the surface, message 1565 of this coding session appears to be nothing more than a routine test failure — an Ansible connectivity check that went wrong. The assistant runs a test script, and the output is a single, dense error block:

=== FGW Ansible Playbook Tests ===


=== Test 1: Connectivity Check ===
[ERROR]: Task failed: Data could not be sent to remote host "172.28.0.22". Make sure this host can be reached over ssh: Warning: Permanently added '172.28.0.22' (ED25519) to the list of known hosts.
"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
Connection closed by 172.28.0.22 port 22
Origin: <adhoc 'ping' task>

But beneath this seemingly mundane failure lies a rich story about infrastructure-as-code testing, the hidden complexities of containerized systemd environments, and the iterative debugging process that characterizes real-world DevOps work. This message captures a pivotal moment in a larger effort to validate Ansible deployment scripts for a horizontally scalable S3 storage system called the Filecoin Gateway (FGW), and it reveals how even a well-prepared test harness can be defeated by something as subtle as a boot-time security mechanism.

The Context: Building a Test Harness for Ansible Deployments

To understand why this message was written, we must first understand what the assistant was trying to accomplish. The broader session involved committing and validating Ansible deployment scripts for FGW clusters. The assistant had created seven Ansible roles, five playbooks, and a complete inventory structure for deploying a three-layer architecture: stateless S3 frontend proxies, Kuri storage nodes, and a YugabyteDB backend.

To validate these scripts without deploying to real hardware, the assistant built a comprehensive Docker-based test harness. This harness consisted of:

The Immediate Trigger: A Reboot and a Race Condition

The message at index 1565 was the result of running ./run-tests.sh after a significant amount of preparatory work. Looking at the messages immediately preceding it, we can see the assistant had just:

  1. Fixed a read-only volume mount issue in the Docker Compose configuration (message 1561)
  2. Restarted all containers with docker compose down &amp;&amp; docker compose up -d (message 1562)
  3. Waited 15 seconds, then copied binaries into the target containers (message 1563)
  4. Set up the Ansible environment in the controller container, installing packages like sshpass, postgresql-client, ansible, and cqlsh (message 1564) The critical detail is in step 2: the containers had just been restarted. When a systemd-based Docker container boots, it goes through the full Linux boot sequence — or at least a simulation of it. During this boot process, systemd creates a file called /run/nologin (and sometimes /etc/nologin). The PAM (Pluggable Authentication Modules) framework, which handles authentication on Linux, checks for this file. If it exists, the pam_nologin module blocks all non-root login attempts with the message: "System is booting up. Unprivileged users are not permitted to log in yet." This is a security feature designed to prevent users from logging in while the system is still initializing services. In a production server, this makes sense — you don't want users connecting before all services are ready. But in a Docker test container, this mechanism becomes an obstacle: the containers are technically "booting" every time they start, and the Ansible controller tries to connect via SSH before systemd has finished its initialization.

Assumptions Made and Broken

The assistant made several assumptions that this error message would expose:

Assumption 1: Containers are ready immediately after startup. The assistant had already shown awareness of boot timing issues earlier in the session — in message 1544, they explicitly waited for systemctl is-system-running --wait on the target hosts. But after the container restart in message 1562, the assistant only waited 15 seconds before copying binaries (message 1563) and then immediately proceeded to set up the Ansible environment and run tests. The 15-second sleep was insufficient for systemd to complete its boot sequence.

Assumption 2: The SSH connection would work if the container was running. Docker containers can be in a "running" state from Docker's perspective (the main process is alive) while systemd is still initializing. The docker compose up -d command returns as soon as the containers are started, not when they're fully booted. This creates a window where SSH is technically running but refusing connections due to pam_nologin.

Assumption 3: The test script's connectivity check would pass after the previous fixes. The assistant had just resolved several issues — missing packages, read-only volumes, missing group_vars — and may have assumed that the remaining problems were all addressed. The pam_nologin issue was a new class of problem that hadn't been encountered yet in this session.

The Thinking Process: What the Assistant's Response Reveals

The assistant's response to this error is instructive. Rather than showing a direct reply in the same message, the assistant's next action (message 1566) demonstrates their diagnostic reasoning:

Systemd is still booting. Let me wait longer and retry:

This one-line diagnosis reveals that the assistant immediately recognized the root cause. The error message itself was quite specific — it quoted the pam_nologin message verbatim — which made the diagnosis straightforward for someone familiar with systemd behavior. The assistant correctly identified that:

  1. The SSH connection was being established (the host key was accepted)
  2. But the authentication was being rejected by PAM
  3. The specific PAM module responsible was pam_nologin
  4. The solution was to wait for systemd to finish booting The fix was simple: run systemctl is-system-running --wait on each target host, which blocks until systemd reports that the system is fully operational. This command was already used earlier in message 1544, showing that the assistant had prior knowledge of this pattern but had forgotten to apply it after the latest container restart.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Understanding of Ansible's ping module: The error originates from an ad-hoc ping task, which is Ansible's way of testing SSH connectivity to target hosts. The error format shows Ansible's structured output.
  2. Knowledge of PAM and pam_nologin: The pam_nologin module is a standard Linux PAM component that checks for the existence of /run/nologin or /etc/nologin files. These files are created by systemd during boot and removed when the system reaches a running state.
  3. Understanding of systemd boot sequence in containers: Systemd-based Docker containers go through an initialization process that mimics a real system boot. The systemctl is-system-running command reports the system's current state (initializing, starting, running, degraded, etc.).
  4. Familiarity with Docker Compose lifecycle: The docker compose up -d command starts containers in detached mode and returns immediately. It does not wait for the containers' init systems to complete their boot sequence.
  5. Knowledge of the FGW architecture: The broader context involves a horizontally scalable S3 storage system with Kuri storage nodes, S3 frontend proxies, and YugabyteDB. The test harness validates Ansible playbooks that deploy this architecture.

Output Knowledge Created

This message, despite being a failure, created valuable knowledge:

  1. A documented failure mode: The pam_nologin issue is now captured as part of the development history. Anyone reviewing this session will understand that systemd boot timing is a critical consideration when testing Ansible roles against Docker containers.
  2. Validation of the test harness design: The fact that the SSH connection was established (the host key was added) but authentication was rejected confirms that the network configuration and SSH setup were correct. The only issue was timing.
  3. A pattern for future testing: The assistant's response in message 1566 — using systemctl is-system-running --wait — establishes a reusable pattern for ensuring Docker containers are fully booted before running Ansible tests.
  4. Confirmation of the debugging methodology: The iterative approach of running tests, observing failures, diagnosing root causes, and applying fixes was validated. Each failure eliminated a class of problems, narrowing down to the remaining issues.

The Broader Significance

This message represents a classic tension in infrastructure testing: the desire for realism versus the need for reliability. The assistant chose to use full systemd-based containers to closely simulate production environments, which is admirable for testing Ansible roles that depend on systemd services. But this realism introduces timing dependencies that don't exist in lighter-weight container approaches.

The alternative would have been to use minimal containers without systemd, running SSH directly as the main process. This would eliminate the pam_nologin issue entirely, but it would also miss testing the systemd service management aspects of the Ansible roles. The assistant's choice to use systemd containers was deliberate and correct for the testing goals — it just required additional handling for the boot sequence.

The error message in message 1565 is a reminder that infrastructure testing is never straightforward. Every layer of abstraction — Docker, systemd, PAM, SSH, Ansible — introduces its own behaviors and failure modes. The skill lies not in avoiding these failures entirely, but in recognizing them quickly, understanding their root causes, and applying targeted fixes. The assistant's immediate recognition of the pam_nologin issue and the swift resolution in the following message demonstrate exactly this kind of expertise.