The Persistence of pam_nologin: A Case Study in Iterative Infrastructure Debugging

The Message

sleep 5 && cd /home/theuser/gw/ansible/test && ./run-tests.sh 2>&1
=== FGW Ansible Playbook Tests ===


=== Test 1: Connectivity Check ===
[ERROR]: Task failed: Data could not be sent to remote host "172.28.0.21". Make sure this host can be reached over ssh: Warning: Permanently added '172.28.0.21' (ED25519) to the list of known hosts.
"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
Connection closed by 172.28.0.21 port 22
Origin: <adhoc 'ping' task>

{'action': 'ping...

At first glance, this is a failure message—another one. The Ansible connectivity check has failed because SSH access to the target host is blocked by pam_nologin, a Pluggable Authentication Module (PAM) mechanism that prevents unprivileged users from logging in while a Linux system is still booting. But this message is far more than a simple error report. It is a snapshot of a debugging process at a critical inflection point, where multiple attempted fixes have been applied, assumptions are being tested against reality, and the same stubborn issue keeps resurfacing. To understand why this message was written—and what it reveals about the craft of infrastructure-as-code debugging—we must trace the journey that led to this moment.

The Context: An Ansible Deployment Pipeline Under Construction

The assistant is building and debugging a suite of Ansible playbooks for deploying Filecoin Gateway (FGW) clusters—a horizontally scalable S3-compatible storage system backed by IPFS and Filecoin. The test harness uses Docker containers running systemd to simulate production Ubuntu servers, against which Ansible playbooks are validated. This is a sophisticated testing strategy: by running real systemd inside containers, the test environment closely mirrors production behavior, catching issues that would otherwise only surface during real deployments.

The session leading up to message 1645 has been a marathon of iterative debugging. The assistant has already identified and fixed several distinct classes of bugs:

• Environment file syntax: systemd's EnvironmentFile directive rejects export prefixes in shell-style variable assignments, requiring a template fix.
• Log level format: The Java-style *:* pattern was invalid for the logging framework, requiring .*:.* regex syntax.
• Wallet dotfiles: Hidden files like .gitkeep in the wallet directory caused binary parsing errors when kuri tried to read them as wallet data.
• Duplicate table creation: Both the yugabyte_init role and kuri init tried to create the same CQL tables, causing migration failures.
• Missing controller tools: The Ansible controller container lacked psql and cqlsh for database initialization.
• Non-existent Ansible filter: The s3_frontend role referenced format_backend_url, a filter that didn't exist. Each fix was applied with surgical precision: edit the file, verify the logic, rebuild, retest. Yet one issue has proven remarkably resilient: the pam_nologin blockade.

The Assumption That Failed

Message 1645 is the direct result of a specific assumption made in the preceding messages. In message 1630, the assistant edited the Dockerfile to add a line disabling systemd-user-sessions.service, the systemd unit responsible for creating the /run/nologin file during boot. The reasoning was sound: if the service doesn't create the nologin file, PAM won't block SSH logins. In messages 1643 and 1644, the assistant took the additional steps of forcing a full rebuild with --no-cache and recreating containers with --force-recreate, ensuring the modified Dockerfile would actually be used.

The sleep 5 command at the start of message 1645 reveals a further assumption: that the containers simply need a few seconds to finish booting before they become accessible. This is a reasonable heuristic—systemd containers do take time to initialize all services—but it misunderstands the nature of the pam_nologin problem. The nologin file isn't a transient condition that clears after boot completes; in a Docker context, systemd may never reach the "finished booting" state that would trigger its removal. The file persists indefinitely, permanently blocking SSH access.

What This Message Reveals About the Debugging Process

Message 1645 is, in essence, a reality check. The assistant has applied what appeared to be the correct fix—modify the Dockerfile to disable the offending service, rebuild from scratch, recreate containers—yet the symptom persists unchanged. This is a common pattern in infrastructure debugging: the fix that seems obvious on paper fails in practice, forcing a deeper investigation.

Several details in the message hint at the underlying dynamics:

The changing IP address. The error message references 172.28.0.21, whereas earlier connectivity failures (in messages 1626 and 1627) referenced 172.28.0.31 and 172.28.0.22. This confirms that containers are being recreated with fresh network assignments, which is expected behavior after docker compose up --force-recreate. It also confirms that the rebuild cycle is working—new containers are being created—but the nologin issue persists across generations.

The unchanged error text. The PAM error message is identical to earlier failures: "System is booting up. Unprivileged users are not permitted to log in yet." This verbatim repetition is a strong signal that the Dockerfile modification did not take effect as intended. The systemd-user-sessions.service may not be the only source of the nologin file, or the mask command in the Dockerfile may not have executed at the right point in the boot sequence.

The absence of diagnostic output. The message shows only the test script's output, not any intermediate verification steps. The assistant does not, for example, exec into a container to check whether /run/nologin still exists after the rebuild, or whether systemd-user-sessions.service is actually masked. This is understandable—the assistant is operating in a rapid edit-test cycle, and each verification step adds time—but it means the message captures a moment of incomplete diagnosis.

The Input Knowledge Required

To fully understand message 1645, a reader needs several layers of context:

1. Ansible mechanics: Understanding that the connectivity check uses Ansible's ping module over SSH, and that a failed ping blocks all subsequent playbook execution.
2. PAM and systemd internals: Knowing that pam_nologin.so checks for the existence of /run/nologin (or /var/run/nologin) and that this file is created by systemd-user-sessions.service during early boot and removed once the system reaches a multi-user target.
3. Docker + systemd quirks: Recognizing that running systemd inside Docker containers is a known pattern (sometimes called "CI-friendly systemd") but comes with edge cases, including the nologin persistence issue. In a normal boot, systemd removes the nologin file when it transitions to a running state. In a Docker container, this transition may never fully complete, leaving the file in place permanently.
4. The test harness architecture: Knowing that the test environment consists of an Ansible controller container and multiple target containers (kuri-01, kuri-02, s3-fe-01), all running systemd, connected via a Docker network.
5. The preceding debugging history: Understanding that this is not the first attempt to fix the nologin issue—the assistant has already tried manual removal (message 1628) and a Dockerfile fix (message 1630), and has now rebuilt everything from scratch.

The Output Knowledge Created

Despite being a failure, message 1645 creates valuable knowledge:

• The Dockerfile fix is insufficient: Masking systemd-user-sessions.service in the Dockerfile does not prevent the nologin file from being created, or the file persists through some other mechanism.
• The rebuild cycle is verified working: Containers are being recreated with fresh IPs, confirming that --no-cache and --force-recreate are functioning correctly.
• The problem is deterministic: The failure occurs on every container startup, not intermittently, ruling out race conditions or timing issues.
• A different approach is needed: The assistant must either find the correct systemd unit to disable, modify the PAM configuration to skip the nologin check, or remove the file at a later stage in the container startup sequence.

The Thinking Process Visible in the Message

The sleep 5 command is the most telling detail. It reveals a hypothesis being tested: "Maybe the containers just need more time to boot." This is a natural debugging instinct—when a service isn't ready, wait and retry. But the identical error message proves this hypothesis wrong. The assistant is thinking iteratively, applying the scientific method to infrastructure: form a hypothesis, run an experiment, observe the result, refine.

The decision to include the full command in a single line (sleep 5 && cd ... && ./run-tests.sh) rather than separate steps also reveals a workflow preference: the assistant wants atomic, reproducible test invocations. Each test run is a self-contained experiment with a clear command and output, making it easy to compare results across iterations.

Conclusion

Message 1645 is a small but revealing moment in a larger debugging narrative. It captures the precise point where a well-reasoned fix meets stubborn reality, where assumptions are falsified, and where the debugger must decide whether to dig deeper into the same hypothesis or pivot to a new approach. The pam_nologin error is, in the grand scheme of the FGW deployment pipeline, a minor infrastructure hiccup. But the process of diagnosing and resolving it—the iterative cycle of hypothesis, experiment, and refinement—is the essence of infrastructure engineering. This message is not just a failure report; it is a data point in a learning process, a snapshot of a system being understood one experiment at a time.