The Nologin That Wouldn't Go Away: Debugging Systemd's Boot-Time SSH Blockade in Docker Containers

In the middle of an intense debugging session for an Ansible-based deployment pipeline for Filecoin Gateway (FGW) clusters, a single short message captures a pivotal moment of empirical investigation. The message, indexed at position 1647 in the conversation, reads:

[assistant] The systemd-user-sessions.service removal didn't prevent nologin. The nologin is probably created by PAM or the boot process. Let me try a different approach - check if the service exists: [bash] cd /home/theuser/gw/ansible/test/docker && docker compose exec -T kuri-01 ls -la /lib/systemd/system/systemd-user-sessions.service 2>&1 ls: cannot access '/lib/systemd/system/systemd-user-sessions.service': No such file or directory

This message is deceptively simple—a mere two sentences and a shell command—but it represents the culmination of a chain of reasoning, a failed hypothesis, and a crucial pivot in strategy. To understand its significance, we must unpack the context, the assumptions that led to this moment, and the knowledge it produced.

The Context: Ansible Deployment Meets Systemd Containers

The assistant had been building and debugging a Docker-based test harness for Ansible playbooks that deploy FGW clusters. The test environment used Ubuntu 24.04 containers running systemd as their init system, simulating production servers. The Ansible controller needed SSH access to these containers to execute playbooks.

A persistent problem had emerged: SSH connections were being rejected with the message "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)." This is a standard Linux security mechanism—during early boot, the pam_nologin module checks for the existence of /run/nologin (or /var/run/nologin, which is typically a symlink) and refuses login attempts from non-root users if the file exists. In a normal system, this file is created by systemd-user-sessions.service during early boot and removed once boot completes.

The assistant had previously identified this issue in message 1629, noting "The nologin file exists due to systemd boot." The chosen fix was to disable the systemd-user-sessions.service in the Dockerfile, assuming this would prevent the nologin file from being created in the first place. This was a reasonable hypothesis: if the service that creates the file is disabled, the file should not appear.

The Hypothesis and Its Failure

The Dockerfile edit (message 1630) added a line to mask or disable the service. After rebuilding the Docker images with --no-cache and recreating the containers (messages 1643–1645), the assistant ran the tests again. The result was unchanged: SSH connections were still blocked by the nologin mechanism.

Message 1647 is the moment of reckoning. The assistant checks whether the service file even exists inside the container: ls -la /lib/systemd/system/systemd-user-sessions.service. The answer is "No such file or directory." This is a stunning result. The service file doesn't exist at all—meaning the Docker image never included it, or the previous Dockerfile change had inadvertently removed it. Yet the nologin file persists.

This single ls command disproves the core assumption. The nologin file is not being created by systemd-user-sessions.service (which doesn't exist), so disabling that service was never going to fix the problem. The assistant's hypothesis was wrong.

The Thinking Process: Empirical Debugging in Action

What makes this message so instructive is the clarity of the reasoning process it reveals. The assistant follows a textbook debugging cycle:

  1. Observe failure: SSH connections are rejected due to pam_nologin.
  2. Form hypothesis: The nologin file is created by systemd-user-sessions.service.
  3. Implement fix: Disable the service in the Dockerfile.
  4. Test: Rebuild images, recreate containers, run tests.
  5. Observe failure persists: The fix didn't work.
  6. Refine hypothesis: Maybe the service wasn't actually disabled, or maybe something else creates the file.
  7. Gather evidence: Check if the service file exists.
  8. Interpret evidence: The service file doesn't exist, so the hypothesis was fundamentally wrong.
  9. Pivot: "The nologin is probably created by PAM or the boot process." This is not a random guess—it's a reasoned shift based on evidence. If systemd-user-sessions.service doesn't exist, then something else must be creating /run/nologin. The assistant correctly identifies two other possibilities: PAM modules (which can create the file independently) or the general boot process (systemd itself may create the file during early initialization before any services run).

Assumptions and Their Consequences

Several assumptions underpin this debugging episode:

Assumption 1: systemd-user-sessions.service is the sole creator of /run/nologin. This is a common belief, but it's not entirely accurate. While that service does manage the nologin file in many distributions, systemd's boot process itself can create /run/nologin very early, before any services are started. The file is part of the "userspace boot sequencing" mechanism. In some configurations, the file is created by systemd's PID 1 directly, not by a service unit.

Assumption 2: Disabling the service in the Dockerfile would be effective. The assistant added a line to disable the service, but the Docker build process may have been more complex. The Dockerfile uses a multi-stage build or a target-based bake system. The service file might never have been included in the base image (Ubuntu 24.04 minimal), or it might be part of a package that wasn't installed. The ls command reveals that the service file was never there to begin with.

Assumption 3: The Docker image rebuild was complete. The assistant used --no-cache to force a full rebuild, but Docker layer caching can be subtle. However, the ls result confirms the service file is absent, so the rebuild did include the Dockerfile changes. The issue is that the changes targeted a service that didn't exist.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. The nologin mechanism is not controlled by systemd-user-sessions.service alone. The service file doesn't even exist in the container, yet the nologin file is created. This means the root cause is deeper—likely in systemd's PID 1 or the kernel's PAM integration.
  2. A different approach is needed. The assistant immediately pivots from "disable the service" to considering PAM or boot process as the source. This leads to the next steps: either waiting for boot to complete, removing the nologin file after container startup, or modifying the PAM configuration.
  3. The Dockerfile change was ineffective but not harmful. The service didn't exist, so the disable command was a no-op. The real fix would need to address the boot process itself.

The Broader Significance

Message 1647 is a masterclass in honest debugging. The assistant doesn't double down on the failed hypothesis or try to rationalize why the fix should have worked. Instead, it immediately gathers evidence, accepts the disconfirmation, and pivots. The subsequent messages show the assistant moving to a more robust solution: modifying the setup script to wait for boot to complete or to remove the nologin file after the containers start.

In the larger arc of the session, this moment represents a transition from "fix the root cause" (prevent the file from being created) to "handle the symptom" (remove the file or wait for boot). While not architecturally elegant, it's pragmatically correct for a test harness. The lesson is profound: sometimes the most effective debugging move is not to fix the root cause but to accept its existence and work around it.

This single message, in its brevity, encapsulates the essence of disciplined systems debugging: form a hypothesis, test it empirically, accept the evidence even when it disproves your assumption, and adapt. The nologin file remained, but the understanding of the system deepened—and that understanding is what ultimately leads to a working solution.