The Persistent Nologin: A Debugging Microcosm in Infrastructure Automation

The Message

Still failing. Let me check if the nologin file exists:

>

``bash cd /home/theuser/gw/ansible/test/docker && docker compose exec -T kuri-01 ls -la /run/nologin /var/run/nologin 2>&1 -rw-r--r-- 1 root root 142 Jan 31 17:58 /run/nologin -rw-r--r-- 1 root root 142 Jan 31 17:58 /var/run/nologin ``

At first glance, this appears to be a trivial exchange: an engineer checks whether a file exists on a container, finds it does, and moves on. But within the arc of a complex infrastructure debugging session, this single message represents a pivotal moment of hypothesis falsification — the point where a carefully constructed theory about why Ansible cannot SSH into Docker containers collapses under the weight of empirical evidence.

The Context: Ansible Meets systemd in a Docker Test Harness

The broader session involves building and debugging an Ansible-based deployment pipeline for the Filecoin Gateway (FGW), a horizontally scalable S3 storage system. The assistant has constructed a Docker-based test harness that simulates production Ubuntu servers using systemd-enabled containers. These containers run SSH, and Ansible is supposed to connect to them, deploy binaries, configure services, and verify health.

The problem manifests as a frustratingly consistent failure: every time the assistant runs the connectivity test, Ansible reports:

"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."

This error comes from pam_nologin, a Pluggable Authentication Module that prevents non-root SSH logins while the system is booting. In a systemd-managed Linux system, the presence of /run/nologin or /var/run/nologin triggers this behavior. The file is typically created during early boot and removed when boot completes. In a Docker container running systemd as PID 1, the boot sequence is simulated but can get stuck in a transitional state, leaving the nologin file in place indefinitely.

The Hypothesis and the Fix Attempt

The assistant's initial hypothesis was that the systemd-user-sessions.service was responsible for creating the nologin file. This is a reasonable assumption: the systemd-user-sessions.service is documented to manage user session tracking and interacts with PAM configuration. The assistant modified the Dockerfile to disable this service:

RUN systemctl disable systemd-user-sessions.service

Then, to ensure the fix took effect, the assistant force-rebuilt the Docker images with --no-cache and recreated the containers with --force-recreate. This should have produced containers where the service was disabled from the moment of boot.

The Moment of Truth

Message 1646 captures the exact moment the assistant tests this hypothesis. The command is precise and diagnostic:

docker compose exec -T kuri-01 ls -la /run/nologin /var/run/nologin

The output is devastatingly clear: both /run/nologin and /var/run/nologin exist. Their timestamps — Jan 31 17:58 — are critical. The containers were rebuilt and recreated after the Dockerfile modification. If the fix had worked, these files would either not exist or would have been removed during boot. Their presence with a post-rebuild timestamp proves that the systemd-user-sessions.service was not the culprit.

Assumptions Under the Microscope

This message reveals several assumptions that were baked into the debugging approach:

Assumption 1: systemd-user-sessions.service creates nologin. This was the core hypothesis. The evidence disproves it. In reality, the nologin file is created by systemd-logind.service or by the systemd-user-sessions unit in a way that isn't prevented by simply disabling the service. The assistant later discovers that the service file doesn't even exist in the container (ls: cannot access '/lib/systemd/system/systemd-user-sessions.service': No such file or directory), confirming that the entire line of inquiry was a red herring.

Assumption 2: A --no-cache rebuild guarantees fresh state. The assistant assumed that rebuilding with --no-cache and recreating containers would produce a clean slate. While this is true for the image layers, it doesn't account for runtime state created by systemd during boot. The nologin file is created dynamically at boot time, not baked into the image.

Assumption 3: The fix was correctly applied. The assistant edited the Dockerfile to disable the service, but the edit may have targeted the wrong service or used an incorrect mechanism. The fact that the service file doesn't exist suggests the Docker image's systemd package set may not include systemd-user-sessions at all, making the systemctl disable command a no-op.

The Thinking Process Visible in the Message

The message's opening words — "Still failing" — reveal the assistant's emotional and cognitive state. There is a weariness here, a recognition that yet another attempt has not yielded the desired result. But rather than expressing frustration, the assistant immediately pivots to diagnosis: "Let me check if the nologin file exists."

This is textbook scientific debugging. The sequence is:

1. Observe failure: The connectivity test still fails with the pam_nologin error.
2. Form hypothesis: The nologin file is present, confirming the root cause.
3. Implement fix: Disable systemd-user-sessions.service in the Dockerfile.
4. Apply fix: Rebuild images and recreate containers.
5. Test hypothesis: Run the test again.
6. Observe continued failure: The test still fails.
7. Gather diagnostic data: Check if the nologin file still exists.
8. Compare with prediction: The file exists, so the hypothesis is wrong. The assistant doesn't jump to conclusions or try another random fix. Instead, it goes back to first principles: verify that the condition you thought you fixed is actually fixed. This is the discipline that separates effective debugging from cargo-cult problem solving.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

• Linux boot process and PAM: Understanding that pam_nologin.so checks for /run/nologin and /var/run/nologin to prevent logins during boot.
• systemd architecture: Knowing that systemd creates these files as part of its boot sequence and removes them when the system reaches a certain target (typically multi-user.target or graphical.target).
• Docker with systemd: Understanding that running systemd inside a Docker container is a known pattern for simulating full Linux environments, but it comes with quirks — the boot sequence may not complete properly, leaving the system in a perpetual "booting" state.
• Ansible connection mechanics: Knowing that Ansible connects via SSH and that SSHd checks PAM configuration, including pam_nologin, before allowing login.
• File timestamps as evidence: Recognizing that the Jan 31 17:58 timestamp proves the file was created after the container was rebuilt, not inherited from a previous state.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

1. The nologin file persists despite the fix. This is the primary datum. It redirects the debugging effort away from systemd-user-sessions.service and toward other mechanisms.
2. The file is being created dynamically at boot. The timestamp matches the container's boot time, not the image build time.
3. The hypothesis is falsified. The assistant now knows that disabling systemd-user-sessions.service is insufficient or irrelevant.
4. A new debugging direction is needed. The assistant must find the actual source of the nologin file — which turns out to be a different systemd mechanism entirely.

The Broader Significance

What makes this message worth studying is not its content — two lines of bash output — but what it represents in the arc of infrastructure debugging. It is a moment of intellectual honesty: the assistant had invested time and effort in a particular theory, had modified code, rebuilt images, and re-run tests. The natural human tendency would be to rationalize the continued failure, to look for reasons why the fix should have worked but was thwarted by some external factor. Instead, the assistant accepts the evidence and moves on.

This is the essence of operational maturity in software engineering. Every hypothesis must be testable. Every test must produce clear evidence. Every piece of evidence must be allowed to falsify the hypothesis, no matter how much effort was invested in it. The nologin file doesn't care about the engineer's intentions. It simply exists — or doesn't — and the engineer must respond accordingly.

The message also illustrates the importance of diagnostic commands that produce unambiguous evidence. ls -la on a specific file path returns either a file listing or a "No such file or directory" error. There is no ambiguity, no parsing required, no interpretation needed. The assistant chose the right tool for the job: a command that would produce a clear boolean answer to the question "Does the nologin file still exist?"

In the messages that follow, the assistant pivots to a new approach: manually removing the nologin files after container startup and then running the tests. This workaround succeeds, and the deployment pipeline passes all stages. But the deeper lesson — that systemd's boot sequence in Docker containers is unreliable and that the nologin mechanism must be actively managed — becomes part of the team's operational knowledge, encoded in the setup script's wait-and-clean logic.

Conclusion

Message 1646 is a small but perfect specimen of infrastructure debugging in the wild. It captures a hypothesis being tested, evidence being gathered, and a theory being discarded. It demonstrates that the most important skill in systems engineering is not knowing the right answer, but knowing how to ask the right question — and having the discipline to accept the answer, even when it means your work was in vain. The nologin file remains, but the engineer's understanding has advanced, and that is the only progress that matters.