The PAM_Nologin Fix: Tracing a Systemd Quirk Through an Ansible Test Harness

Message: [assistant] The issue is the systemd-user-sessions.service creates the nologin file. Let me add a line to disable that service: [edit] /home/theuser/gw/ansible/test/docker/Dockerfile.target Edit applied successfully.

At first glance, this message appears trivial: a single line of reasoning followed by a one-line edit to a Dockerfile. But this brief moment represents the culmination of a focused debugging chain that had been blocking an entire Ansible deployment test pipeline. The assistant had been iteratively fixing a suite of deployment scripts for the Filecoin Gateway (FGW) cluster — fixing environment file syntax, correcting log level formats, excluding dotfiles from wallet directories, and separating database migrations — only to hit a wall when the test harness itself refused to connect to its target hosts. The error message was cryptic: "System is booting up. Unprivileged users are not permitted to log in yet." This article unpacks how the assistant traced that error to its root cause in systemd's boot-time session management and why the fix — disabling a single systemd service in the Docker image — was the correct architectural decision.

The Symptom: A Test Harness That Cannot Connect

The debugging chain began at message 1626, when the assistant ran the test suite for the first time after a series of fixes. The test harness, a Docker Compose-based environment simulating production Ubuntu servers with SSH access, failed at the very first step: a connectivity check using Ansible's ping module. The error read:

"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."

This is a standard Linux Pluggable Authentication Modules (PAM) behavior. During system boot, the pam_nologin module checks for the existence of /run/nologin or /var/run/nologin. If either file exists, it blocks all non-root logins with the message above. The file is created by systemd early in the boot sequence and is normally removed once the system reaches a multi-user target. In a production server, this is a useful safety mechanism — it prevents users from logging in before the system is fully initialized. But inside a Docker container running systemd as its init system, the boot sequence never quite completes in the same way, leaving the nologin file in place indefinitely.

The Debugging Trail: From Symptom to Root Cause

The assistant's first response was pragmatic but naive: wait and retry. At message 1627, a sleep 10 was inserted before re-running the tests. This is a common instinct — system boot takes time, so perhaps the container just needed a few more seconds. But the retry produced the identical error. The nologin file was not going away on its own.

At message 1628, the assistant shifted from waiting to investigating. A shell command checked for the existence of the nologin files inside the running containers:

docker compose exec -T kuri-01 ls -la /var/run/nologin /run/nologin

Both files existed. The assistant then removed them with rm -f across all three target containers (kuri-01, kuri-02, s3-fe-01). This was a tactical fix — it would allow the current test run to proceed — but it was not sustainable. Every time the containers restarted, the nologin files would reappear.

At message 1629, the assistant considered two approaches: "Let me update the Dockerfile to remove it or the setup script to wait/remove it." The choice between these two options — a permanent fix in the Docker image versus a procedural workaround in the setup script — would determine whether the test harness was robust or fragile.

The Root Cause: systemd-user-sessions.service

Message 1630 is where the assistant makes the critical leap from symptom to root cause. Instead of just removing the file or adding a wait loop, the assistant identifies the specific systemd unit responsible: systemd-user-sessions.service. This service is part of systemd's boot sequence and is responsible for creating and later removing the nologin file as part of the user session lifecycle. In a normal boot, it creates the file early (preventing premature logins) and removes it once the system is ready. In a Docker container where the full boot sequence may not complete cleanly, the file persists.

The assistant's reasoning is concise but demonstrates a deep understanding of Linux boot internals: "The issue is the systemd-user-sessions.service creates the nologin file." This is not a guess — it reflects knowledge of how PAM, systemd, and SSH authentication interact. The pam_nologin module checks for the file; systemd-user-sessions.service manages the file's lifecycle; and in a containerized environment, the service may never reach the point where it removes the file.

The Fix: A Permanent Solution in the Dockerfile

The chosen fix was to add a single line to Dockerfile.target, the Docker image definition for the test target hosts:

RUN systemctl disable systemd-user-sessions.service

This is a clean, permanent solution. By disabling the service at image build time, every container launched from this image will start without the nologin mechanism. No setup script workaround needed, no manual file removal after container start, no race conditions between boot completion and SSH availability. The fix addresses the root cause rather than the symptom.

The decision to fix the Dockerfile rather than the setup script is architecturally significant. A setup-script fix would have been fragile — it would depend on timing (removing the file after the container starts but before Ansible tries to connect), would need to run on every container start, and would introduce a maintenance burden. A Dockerfile fix is baked into the image, requires no runtime coordination, and is automatically applied to all containers from that image. This is the difference between a workaround and a solution.

Assumptions and Knowledge Required

To understand and implement this fix, the assistant needed several pieces of knowledge:

PAM architecture: Understanding that pam_nologin blocks SSH logins when /run/nologin exists, and that this is a deliberate security mechanism, not a random error.
Systemd boot sequence: Knowing that systemd-user-sessions.service is the unit responsible for managing the nologin file, and that it can be disabled without breaking other functionality.
Docker with systemd: Recognizing that running systemd inside a Docker container (as opposed to a simpler init like supervisord) introduces boot-sequence quirks that don't occur on real hardware or virtual machines.
Ansible connection mechanics: Understanding that Ansible connects via SSH, which goes through PAM, which checks nologin — so the failure was at the transport layer, not in the playbook logic.
Dockerfile build-time vs runtime: Knowing that systemctl disable in a Dockerfile RUN command persists in the image and affects all containers started from it.

Mistakes and Incorrect Assumptions

The initial mistake was assuming the problem was transient. The sleep 10 retry at message 1627 reflected an assumption that the container was simply slow to boot and would eventually remove the nologin file on its own. This assumption was wrong because systemd inside Docker containers often does not complete its boot sequence in the same way as on real hardware — the nologin file is created but never cleaned up.

A more subtle incorrect assumption was that the fix could be applied at the setup script level. The assistant briefly considered "update the setup script to wait/remove it" before settling on the Dockerfile approach. The setup-script approach would have worked but would have been fragile and would have required careful ordering of operations (remove nologin before Ansible connects, but after SSH is running). The Dockerfile fix is superior precisely because it eliminates the timing dependency.

Broader Significance

This single message, for all its brevity, represents a pattern that recurs throughout infrastructure engineering: a cryptic error at one layer (SSH connection failure) turns out to be caused by a mechanism at a completely different layer (systemd boot sequence inside Docker). The assistant's ability to trace the error from "SSH connection refused" through PAM to systemd to a specific service unit is a model of systematic debugging.

The fix also illustrates an important principle in test infrastructure design: test environments should be as close to production as necessary, but no closer. Running systemd inside Docker containers for Ansible testing is a reasonable choice — it simulates the service management layer that production uses. But the nologin mechanism, while important in production to prevent premature logins during boot, is counterproductive in a test environment where containers are expected to be immediately accessible after startup. Disabling it is not a hack; it's an appropriate adaptation of the test environment to its purpose.

Conclusion

Message 1630 is a study in minimalism. In just two sentences and one edit command, the assistant identifies a root cause, proposes a fix, and applies it. But behind that minimalism is a chain of reasoning that spans Linux authentication, systemd internals, Docker container behavior, and Ansible transport mechanics. The fix — disabling systemd-user-sessions.service in the Docker image — is permanent, clean, and architecturally sound. It transforms the test harness from one that fails on every fresh container start to one that works reliably, allowing the broader Ansible deployment pipeline to proceed to the next stages of validation.