The Moment of Pivot: Debugging pam_nologin in a Systemd Container Test Harness

In the middle of an intense debugging session for a distributed Filecoin Gateway (FGW) cluster deployment pipeline, a single message from the assistant captures a pivotal moment of technical realization and pragmatic decision-making. The message is brief — barely a paragraph of reasoning followed by a file read — but it represents the culmination of a multi-step debugging chain and a conscious shift in strategy. This article examines that message in depth, unpacking the reasoning, assumptions, and knowledge boundaries that define it.

The Message

Here is the subject message in full, as it appears in the conversation:

[assistant] The service file is gone but the nologin file is still created. There might be something else creating it. Let me just update the setup script to wait for the system to be fully booted: [read] /home/theuser/gw/ansible/test/setup.sh <file> 00059| echo "=== Starting target hosts ===" 00060| docker compose up -d kuri-01 kuri-02 s3-fe-01 00061| 00062| echo "Waiting for SSH to be available..." 00063| for host in kuri-01 kuri-02 s3-fe-01; do 00064| container="ansible-test-$host" 00065| timeout=60 00066| while ! docker exec "$container" nc -z localhost 22 2>/dev/null; do 00067| sleep 1 00068| timeout=$((timeout - 1)) 00069| if [ $timeout -le 0 ]; then 00070| echo "ERROR: SSH on $ho...

The message is a turning point. It begins with a statement of observed fact — "The service file is gone but the nologin file is still created" — followed by a hypothesis: "There might be something else creating it." Then comes the decision: "Let me just update the setup script to wait for the system to be fully booted." This is not a random change; it is a deliberate strategic pivot born from the exhaustion of a narrower line of investigation.

The Debugging Chain That Led Here

To understand why this message matters, one must understand the context. The assistant has been building and debugging a Docker-based test harness for Ansible deployment scripts. The test harness simulates production Ubuntu 24.04 servers using systemd containers — each container runs the full systemd boot sequence, including SSH server startup. The Ansible controller then connects to these containers via SSH to run playbooks that deploy Kuri storage nodes and S3 frontend proxies.

The immediate problem is that SSH connections are failing with a specific error: "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)." This error comes from the Pluggable Authentication Modules (PAM) framework, specifically the pam_nologin module. During system boot, Linux systems create a /run/nologin file (or /var/run/nologin, which is often a symlink). When this file exists, pam_nologin denies all non-root login attempts, preventing Ansible from connecting via SSH as the fgw user.

The assistant's first attempt to fix this was surgical: remove the systemd-user-sessions.service from the Docker image, assuming that service is responsible for creating the nologin file. The assistant edited the Dockerfile (message 1630), rebuilt the images with --no-cache (message 1643), and recreated the containers (message 1644). Yet after all that, the nologin file still appeared (message 1646). When the assistant checked whether the service file existed inside the container, the response was: ls: cannot access '/lib/systemd/system/systemd-user-sessions.service': No such file or directory (message 1647).

This is the critical moment captured in the subject message. The service is gone — the assistant's hypothesis has been experimentally disproven — yet the symptom persists. The nologin file is being created by something else in the systemd boot sequence.## The Reasoning: From Root-Cause Analysis to Workaround

The assistant's reasoning in this message is a textbook example of debugging under uncertainty. The first sentence — "The service file is gone but the nologin file is still created" — establishes the experimental result. The assistant had a theory (systemd-user-sessions.service creates the nologin file), tested it (removed the service, rebuilt the image), and observed the outcome (nologin still appears). The second sentence — "There might be something else creating it" — acknowledges the failure of the initial hypothesis and opens the door to a broader investigation.

But here is where the message becomes interesting. Instead of diving deeper into what "something else" might be — instead of tracing the systemd boot sequence, reading PAM configuration files, or checking which systemd services or tmpfiles.d snippets create /run/nologin — the assistant makes a conscious decision to change approach: "Let me just update the setup script to wait for the system to be fully booted."

This is a workaround, not a root-cause fix. The assistant is choosing to let the system finish booting (at which point the nologin file is removed by the same mechanism that creates it) rather than preventing the nologin file from being created in the first place. The existing setup script already has a loop that waits for SSH port 22 to be available using nc -z localhost 22. The assistant's plan is to extend this wait loop — or perhaps add an additional check — to ensure the system is fully booted before Ansible tries to connect.

Assumptions and Their Validity

The assistant makes several assumptions in this message, some explicit and some implicit.

The first assumption is that the nologin file is temporary and will be removed automatically once the system finishes booting. This is generally correct for standard systemd-based Linux systems: the systemd-user-sessions service (or equivalent boot-time logic) creates /run/nologin early in the boot process to prevent logins during startup, and removes it once the system reaches a multi-user target. However, the assistant has already removed the systemd-user-sessions.service file, which means the normal mechanism for removing the nologin file may also be absent. If the nologin file is created by a different mechanism (such as a tmpfiles.d configuration or a different systemd service) and the removal mechanism is gone, the file could persist indefinitely. The assistant does not verify that the nologin file actually gets removed after boot — this is a gap in the reasoning.

The second assumption is that waiting for SSH to be available is sufficient to guarantee the system is fully booted. In practice, SSH can become available before the boot sequence is complete. The nc -z localhost 22 check only confirms that the SSH daemon is listening on port 22, not that PAM is ready to accept logins. The assistant's existing wait loop could succeed while pam_nologin is still blocking non-root access. This is precisely why the tests are failing: the wait loop exits when SSH starts listening, but the system is still booting and the nologin file still exists.

The third assumption is that the setup script is the right place to fix this. The assistant could have pursued other approaches: adding a rm -f /run/nologin command to the container's startup sequence, modifying the PAM configuration to disable pam_nologin, or creating a systemd service that removes the nologin file after boot. Instead, the assistant chooses to modify the test harness's orchestration logic. This is a reasonable choice for a test environment — it keeps the production-like behavior of the containers intact while working around it in the test runner — but it does mean the test harness is now less faithful to production behavior, since it introduces an explicit wait that production deployments would not need.

Input Knowledge Required

To understand this message, the reader needs several pieces of background knowledge. First, familiarity with Linux PAM and the pam_nologin module: understanding that /run/nologin blocks non-root SSH logins during system boot. Second, knowledge of systemd's boot sequence and service lifecycle: knowing that systemd creates and removes the nologin file as part of its startup protocol. Third, understanding of Docker's behavior with systemd containers: that running systemd inside Docker requires special configuration and that the boot sequence happens asynchronously after the container starts. Fourth, familiarity with Ansible's connection mechanism: that Ansible connects via SSH as a non-root user (in this case, fgw) and that PAM restrictions apply to these connections. Fifth, knowledge of the test harness architecture: that setup.sh orchestrates container startup, binary installation, and Ansible controller setup before run-tests.sh executes the playbooks.

Output Knowledge Created

This message creates several pieces of output knowledge. First, it documents that removing systemd-user-sessions.service is insufficient to prevent nologin file creation — there is another mechanism at work. Second, it establishes a design decision: the test harness will use a wait-for-boot-completion strategy rather than modifying container internals. Third, it implicitly defines a boundary between the test infrastructure (which should accommodate boot delays) and the container images (which should remain as close to production as possible). Fourth, it creates a record of the debugging process for future reference — the assistant is effectively saying "we tried X, it didn't work, here's why, now we're trying Y."

The Thinking Process

The thinking process visible in this message is one of pragmatic triage. The assistant has been iterating rapidly: remove service, rebuild, recreate, test, observe failure, check service existence, confirm service is gone, observe nologin still exists. At each step, the assistant is narrowing the problem space. The message captures the moment when the assistant realizes the problem is deeper than expected and makes a conscious decision to stop digging and switch to a workaround.

This is a common pattern in infrastructure debugging. The assistant could spend hours tracing the systemd boot sequence to find the exact mechanism creating the nologin file. But the goal is not to understand systemd internals — the goal is to get the test harness working. The assistant correctly identifies that the workaround (waiting for full boot) is simpler, faster, and less risky than continuing the root-cause analysis. The "might be" in "There might be something else creating it" is a deliberate acknowledgment of uncertainty, paired with a decision not to resolve that uncertainty immediately.

The file read that follows the reasoning is equally telling. The assistant reads setup.sh not to understand the problem, but to identify the exact location where the wait loop needs to be modified. The read is targeted and purposeful — the assistant already knows the structure of the script and just needs to see the current code to plan the edit.

Mistakes and Incorrect Assumptions

The most significant mistake in this message is the assumption that the existing wait-for-SSH loop is adequate. The assistant has been using nc -z localhost 22 to check SSH availability, but this check passes before the system is fully booted. The nologin file can persist after SSH starts listening, leading to the exact failure pattern the assistant is seeing. The fix — extending the wait loop — needs to check for something more reliable than port availability, such as the absence of /run/nologin or the ability to actually establish an SSH session.

A second potential mistake is not verifying that the nologin file is eventually removed. The assistant assumes it will be, but the removal mechanism may have been inadvertently disabled when the systemd-user-sessions.service was removed. If the same mechanism that creates the file also removes it, and the assistant has removed that mechanism, the file could persist indefinitely. The assistant does not check whether the nologin file disappears after a reasonable wait time.

A third issue is the lack of a systematic approach to identifying the nologin file's creator. The assistant could use systemd-analyze or check /usr/lib/tmpfiles.d/ for configurations that create /run/nologin. Instead, the assistant jumps directly to a workaround. While this is pragmatically defensible, it means the root cause remains unknown and could cause issues in other contexts (such as production deployments that use the same container images).

Conclusion

This message, though brief, captures the essence of infrastructure debugging: a cycle of hypothesis, experiment, observation, and strategic pivot. The assistant's decision to move from root-cause analysis to a pragmatic workaround reflects a mature understanding of when to dig deeper and when to work around. The message also reveals the importance of understanding the full stack — from PAM configuration to systemd boot sequences to Docker container behavior to Ansible connection mechanics — and the difficulty of debugging at the boundaries between these layers. The nologin file sits at exactly such a boundary: created by systemd, checked by PAM, blocking Ansible, running inside Docker. Each layer has its own logic, and the interaction between them creates failure modes that no single layer's documentation fully explains.