The pam_nologin Wall: When Systemd Blocks SSH in a Docker Container

A Single Debugging Message That Reveals the Hidden Complexity of Infrastructure Automation

In the middle of an intense debugging session for a distributed Filecoin Gateway (FGW) cluster deployment, the assistant types a seemingly optimistic message: "Everything looks good." Then runs the test suite. And hits a wall. The error is not about missing binaries, malformed configuration, or broken Ansible roles—the problems that had consumed the previous thirty-seven messages. Instead, it is something far more subtle: a Linux Pluggable Authentication Module (PAM) rule, triggered by systemd's boot lifecycle, preventing SSH logins into a Docker container.

This message, indexed as message 1626 in the conversation, captures a single moment in an iterative debugging marathon. It is the point where surface-level fixes have been applied, confidence is high, and a completely unexpected system-level interaction derails the entire pipeline. To understand why this message matters, one must understand the full arc of debugging that preceded it, the assumptions baked into the test harness, and the peculiarities of running systemd inside Docker containers for infrastructure testing.

The Exact Message

The assistant writes:

Everything looks good. The chmod error was just because docker cp hadn't completed when chmod ran. Let me run the tests now:

``` [bash] cd /home/theuser/gw/ansible/test && ./run-tests.sh 2>&1 === FGW Ansible Playbook Tests ===

> >

=== Test 1: Connectivity Check === [ERROR]: Task failed: Data could not be sent to remote host "172.28.0.31". Make sure this host can be reached over ssh: Warning: Permanently added '172.28.0.31' (ED25519) to the list of known hosts. "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)." Connection closed by 172.28.0.31 port 22 Origin: <adhoc 'ping' task>

{'action': 'ping... ```

The message is deceptively short. It contains a verdict ("Everything looks good"), a dismissed warning (the chmod error was a timing issue), a test invocation, and a failure. But the failure is not a code bug—it is a policy enforcement mechanism embedded deep in the Linux operating system's authentication stack. The assistant had no reason to expect this. The containers were running. The binaries were in place. The Ansible roles had been updated. And yet, SSH—the fundamental transport for Ansible's remote execution—was being rejected.

The Debugging Arc That Led Here

To appreciate the significance of this message, one must trace the preceding thirty-seven messages (1589 through 1625). The assistant had been iteratively fixing a suite of Ansible deployment scripts for the Filecoin Gateway project. The problems were numerous and varied:

The wallet dotfile problem. The Ansible files/wallet/ directory contained a .gitkeep file, a common Git trick to preserve empty directories in version control. When the wallet distribution role copied this directory to target nodes, the .gitkeep file was treated as a wallet key file by the kuri binary, causing binary parsing errors. The assistant had to modify the wallet role to exclude dotfiles after copying.

The log level format mismatch. The environment variable RIBS_LOGLEVEL was set to *:*=debug, but the application's log parsing library (Go's log package using regex-based filtering) rejected the asterisk as an invalid regex repetition operator. The correct format was .*:.*=debug. This required updating both the test inventory and the production defaults.

The wallet file name format. The test wallet directory contained a file named t3test1234567890abcdefghijklmnop, which was not a valid Filecoin address format. The kuri binary expected base32-encoded key filenames. The assistant eventually realized that kuri init could create its own wallet if the directory was empty, leading to a decision to leave the test wallet directory empty.

The chmod timing issue. During container setup, the docker cp command and the subsequent chmod command were racing. The assistant correctly diagnosed this as a non-issue—the binaries were already in place from the volume mount.

Each of these fixes was applied, the test environment was cleaned and rebuilt, and the assistant reached a point where the infrastructure appeared healthy. The docker compose ps output showed all three target containers (kuri-01, kuri-02, s3-fe-01) and the ansible-controller container all in "Up" status. The binaries were present. The configuration had been updated. Everything looked good.

The pam_nologin Surprise

The error message reveals a systemd mechanism: pam_nologin(8). This PAM module checks for the existence of /run/nologin or /etc/nologin files. During system boot, systemd creates /run/nologin to prevent unprivileged users from logging in until the system has finished starting up. Once boot is complete, systemd removes this file, and normal login resumes.

In a Docker container running systemd as its init system (PID 1), the boot sequence is real. The container goes through the full systemd startup: mounting filesystems, starting services, and eventually reaching the default target. During this boot window, the pam_nologin module blocks all non-root SSH logins. Ansible, which connects as the fgw user (an unprivileged account), is locked out.

The assistant's assumption was that a running container equals a ready container. The docker compose ps command showed "Up 34 seconds" for kuri-01, which seemed sufficient. But 34 seconds is well within systemd's boot window, especially for a container that includes service startup, network configuration, and SSH daemon initialization. The SSH daemon (sshd) starts early in the boot process, but PAM's nologin check happens at authentication time, which is later in the boot sequence.

This is a classic race condition between infrastructure provisioning tools and operating system initialization. Ansible's first test—the connectivity check using a raw ping module—sends an SSH connection attempt. If it arrives during the nologin window, the connection is rejected with the "System is booting up" message. The error is not intermittent in the traditional sense; it is deterministic based on timing. Run the test immediately after container start, and it fails. Wait long enough for systemd to finish booting, and it succeeds.

The Reasoning Process Visible in the Message

The message reveals several layers of the assistant's thinking:

Optimism grounded in evidence. The assistant says "Everything looks good" based on observable facts: the docker compose ps output showed all containers running, the earlier chmod warning was understood as a harmless timing issue, and the previous test run had progressed further than before. This is not blind optimism—it is a reasoned assessment based on the debugging progress made.

Prioritization of the test suite. Rather than manually probing individual components, the assistant immediately runs the full run-tests.sh script. This indicates confidence that the foundational issues (wallet, log level, configuration) are resolved, and the next step is end-to-end validation.

Interpretation of the error. The assistant does not panic or revert changes. The error output is captured verbatim, including the full PAM message. This is diagnostic behavior—the assistant is gathering information to understand the new failure mode.

The unstated realization. Although the message ends with the error output and does not include explicit analysis, the assistant's subsequent actions (visible in later messages) show the correct diagnosis: the Docker images need to disable the pam_nologin behavior, specifically by ensuring systemd-user-sessions service is enabled or by masking the nologin file. The fix involves rebuilding the Docker images with a systemd configuration that skips the nologin check, typically by removing /run/nologin early in the boot sequence or by configuring PAM to ignore it.

Assumptions Made and Broken

The assistant made several assumptions that this message exposes:

Container "Up" means ready for SSH. This is the critical broken assumption. Docker's "Up" status only indicates that the container's main process (systemd) is running, not that all boot-time services have completed or that the system is in a steady state. The assistant implicitly assumed that 34 seconds was enough time for systemd to complete boot, which was incorrect for this container configuration.

The chmod error was the only remaining issue. By dismissing the chmod warning as a timing artifact, the assistant implicitly assumed that no other timing-sensitive problems existed. The pam_nologin error proved otherwise—it was also a timing issue, but of a completely different nature.

Previous fixes would cascade to success. The assistant had fixed wallet parsing, log level format, and configuration syntax. The expectation was that these fixes would clear the path for the test suite to pass. While each fix was correct, they addressed different layers of the problem stack. The pam_nologin issue was an orthogonal concern—an infrastructure layer problem that existed independently of the application configuration fixes.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

Ansible's transport mechanism. Ansible uses SSH for remote execution by default. The first test in any Ansible playbook run is typically a connectivity check (the ping module), which establishes an SSH connection and verifies Python availability on the target.

Systemd's boot lifecycle. Systemd goes through multiple phases: initial boot, basic system initialization, multi-user target, and graphical target (if applicable). During early boot, it creates /run/nologin to prevent unprivileged logins. This file is removed when the system reaches a stable state.

PAM architecture. The Pluggable Authentication Modules system allows Linux to apply modular authentication policies. The pam_nologin.so module checks for the existence of nologin files and rejects non-root logins if they exist. This is a security measure to prevent users from accessing a partially-initialized system.

Docker with systemd as init. Running systemd inside a Docker container is a common pattern for testing infrastructure code that depends on systemd services (like Ansible managing services via systemctl). However, it introduces the full system boot sequence, including PAM checks, which are normally bypassed in simpler Docker containers that run a single process.

The FGW test harness architecture. The test environment uses Docker containers with systemd as PID 1, SSH servers for Ansible connectivity, and a shared Ansible controller container that orchestrates deployments. The containers are built with specific Dockerfiles that install systemd, SSH, and other dependencies.

Output Knowledge Created

This message creates several pieces of output knowledge:

A documented failure mode. The pam_nologin error is now captured as a known issue in the FGW deployment pipeline. Future developers encountering this error will recognize it as a systemd boot timing problem rather than a configuration or code bug.

A diagnostic trace. The full error output, including the PAM message and the connection details, provides a complete diagnostic trace. The IP address (172.28.0.31), the SSH warning about host key verification, and the exact PAM error text are all preserved.

A debugging milestone. This message marks the transition from application-layer debugging (wallet files, log levels, configuration syntax) to infrastructure-layer debugging (systemd boot sequence, PAM policies, Docker image configuration). The debugging effort must now address the container image itself, not just the Ansible roles.

A timing sensitivity insight. The message implicitly documents that the test harness has a timing dependency on systemd boot completion. This is valuable architectural knowledge—any future changes to the Docker images that affect boot time (additional services, complex initialization scripts) will impact test reliability.

The Resolution Path

Although the message itself does not contain the fix, the assistant's subsequent actions (visible in the broader session context) follow a clear path: the Docker images must be rebuilt to disable the pam_nologin behavior. The typical fix involves either:

Removing /run/nologin early in the container startup, perhaps via a systemd service that runs before SSH is available.
Configuring PAM to skip the nologin check for the SSH service, by editing /etc/pam.d/sshd to comment out or remove the pam_nologin.so line.
Ensuring the systemd-user-sessions service is enabled, which is responsible for removing /run/nologin once boot is complete. If this service is masked or disabled, the nologin file persists indefinitely.
Adding a startup delay or polling loop in the test script to wait for SSH connectivity before running Ansible. The session summary indicates that the eventual fix was to disable systemd-user-sessions in the Docker images—or more precisely, to ensure it was properly configured to remove the nologin file. The assistant rebuilt the Docker images with this fix, and the test harness subsequently passed all stages.

Why This Message Matters

Message 1626 is a microcosm of infrastructure debugging. It captures the moment when a developer has fixed all the obvious problems, only to discover a hidden one at a completely different layer of the stack. The wallet dotfiles were an application concern. The log level format was a configuration concern. The chmod timing was a scripting concern. But the pam_nologin error was an operating system policy concern—something that most developers never think about because it is invisible in normal operation.

This message also illustrates the peril of "everything looks good" in complex systems. The assistant's assessment was correct based on the available evidence: containers were running, binaries were present, configuration was updated. But the assessment was incomplete because it did not account for systemd's boot lifecycle. The error was not a failure of the fixes—it was a failure of the testing methodology itself. The test suite ran too early, before the infrastructure was fully initialized.

The lesson is universal in infrastructure automation: a running process is not the same as a ready system. Whether it's systemd's nologin file, database warm-up time, cache population, or service discovery propagation, the gap between "running" and "ready" is where many automation failures hide. Message 1626 is a vivid reminder of that gap, captured in a single SSH error message.