The Nologin Red Herring: A Case Study in Debugging Assumptions During Ansible Test Harness Development

The Message

[assistant] Still booting issues. The problem is the systemd pam_nologin - there's a file blocking logins during boot. Let me check and remove that:
[bash] cd /home/theuser/gw/ansible/test/docker && for host in kuri-01 kuri-02 s3-fe-01; do
    echo "Checking $host..."
    docker exec "ansible-test-$host" rm -f /run/nologin /etc/nologin 2>/dev/null || true
    docker exec "ansible-test-$host" ls -la /run/nologin /etc/nologin 2>&1 || echo "No nologin file"
done 2>&1
Checking kuri-01...
ls: cannot access '/run/nologin': No such file or directory
ls: cannot access '/etc/nologin': No such file or directory
No nologin file
Checking kuri-02...
ls: cannot access '/run/nologin': No such file or directory
ls: cannot access '/etc/nologin': No such file or directory
No nologin file
Checking s3-fe-01...
ls: cannot access '/run/nologin': No such file or directory
ls: cannot access '/etc/nologin': No such file or directory
No nologin file

Context: Building an Ansible Deployment Test Harness

This message sits within a larger effort to create automated deployment scripts for a horizontally scalable S3 storage architecture called the Filecoin Gateway (FGW) cluster. The project involved multiple components: Kuri storage nodes that manage data, stateless S3 frontend proxies that route requests, and a YugabyteDB backend for metadata. The assistant had already built the core application binaries, debugged the cluster infrastructure, and optimized performance. Now, the focus had shifted to operationalizing the deployment through Ansible playbooks.

The assistant had just committed a comprehensive set of Ansible roles and playbooks—seven roles spanning common setup, wallet distribution, YugabyteDB initialization, Kuri node deployment, and S3 frontend configuration, orchestrated through five playbooks. To validate these playbooks without touching production infrastructure, the assistant created a Docker-based test harness: a Docker Compose environment with a YugabyteDB container, three Ubuntu 24.04 target hosts (two Kuri nodes and one S3 frontend) running systemd with SSH enabled, and an Ansible controller container. The test harness included setup, run, and cleanup scripts to automate the validation workflow.

The first test run had revealed several issues: the YugabyteDB health check needed the correct hostname, the controller container lacked essential tools like psql, cqlsh, and sshpass, the test inventory was missing group_vars for the Kuri and S3 frontend groups, and read-only volume mounts caused file permission problems. Each of these had been addressed iteratively, and the assistant was now working through the remaining blockers to get a clean test pass.

What the Error Actually Said

The immediate trigger for message 1547 was the output of the previous test run (message 1546), which showed:

=== Test 1: Connectivity Check ===
[ERROR]: Task failed: Action failed: The module interpreter '/usr/bin/python3' was not found.
...
yb-test | FAILED

This error came from Ansible's ping module, which requires Python on the target host to execute its modules. The test inventory at this point included a host called yb-test representing the YugabyteDB container. However, the YugabyteDB container runs a minimal YB-specific image that does not include Python. When Ansible attempted to connect to this host via SSH and execute its ping module, it failed because there was no Python interpreter available.

The critical detail in this error is the hostname: yb-test. This was the YugabyteDB host, not one of the Kuri or S3 frontend targets. The assistant had previously attempted to remove the Yugabyte host from the inventory (message 1543), but the change had not taken effect—possibly because the inventory files were copied to the controller container before the edit, or because the test script still referenced the old inventory structure.

The Reasoning: A Chain of Assumptions

The assistant's response reveals a fascinating reasoning process. Rather than reading the error message literally—"Python 3 not found on yb-test"—the assistant interpreted the failure through the lens of earlier debugging experience. In the immediately preceding test runs, the connectivity check had failed because SSH connections were being refused. The assistant had verified that systemd was running on all target hosts (message 1544) and confirmed that the targets were in a "running" state. Yet the test was still failing.

The assistant's reasoning went: "Still booting issues. The problem is the systemd pam_nologin—there's a file blocking logins during boot." This is a sophisticated diagnosis that draws on deep knowledge of Linux boot sequences. The pam_nologin module is a Pluggable Authentication Module (PAM) that prevents non-root users from logging in when a file like /run/nologin or /etc/nologin exists. Systemd creates /run/nologin during early boot to prevent logins before the system is fully initialized, and removes it once boot completes. If a system is still booting, this file would be present and could cause SSH authentication to fail even if the SSH daemon is running.

This was a reasonable hypothesis given the assistant's recent experience: the SSH connection had been refused moments earlier, and now the error had changed to a Python-not-found error. The assistant may have reasoned that the SSH connection was succeeding at the transport level (TCP connection established) but failing at the authentication level (PAM denying login), which would manifest differently depending on how Ansible reports the failure. The "module interpreter not found" error could be a secondary symptom: if PAM blocks the user's login shell, Ansible might still establish an SSH connection but fail to execute the remote Python interpreter.

The Investigation and Its Result

The assistant executed a bash command across all three target hosts to check for and remove the nologin files. The command used docker exec to bypass SSH entirely and run directly inside each container, which would work regardless of PAM configuration. The output was unambiguous: no nologin files existed on any of the three target hosts. The ls commands returned "No such file or directory" for both /run/nologin and /etc/nologin on all three containers.

This negative result is valuable. It disproves the nologin hypothesis and forces a re-examination of the actual error. The assistant now knows that the problem is not a boot-time login block. The next step—visible in the following messages—was to test SSH connectivity manually using sshpass, which succeeded immediately (message 1548), confirming that SSH was working correctly. The subsequent test run (message 1549) still failed with the same Python error, but now the assistant correctly identified the issue: the test inventory still included the YugabyteDB host, which lacked Python. The fix was to exclude the yugabyte group from the connectivity test (message 1550), after which all three target hosts responded successfully (message 1551).

The Incorrect Assumption

The central incorrect assumption in this message is that the Python interpreter error was caused by a boot-time login block. In reality, the error was straightforward: Ansible could not find Python on the target host because the target host was the YugabyteDB container, which does not include Python. The assistant's prior experience with SSH connection failures created a mental model that biased the diagnosis toward boot/login issues rather than inventory configuration problems.

This kind of mistake is extremely common in debugging, especially under time pressure and with multiple interacting systems. The assistant had been fighting a series of infrastructure issues—Docker networking, volume mounts, missing packages, health check configurations—and each fix had revealed another problem. The cognitive load of tracking multiple concurrent issues made it easy to misread a familiar-looking error. The pam_nologin hypothesis was creative and technically well-grounded; it was simply the wrong diagnosis for this particular symptom.

Input Knowledge Required

To understand this message fully, one needs knowledge of several systems and concepts. First, Ansible's module execution model: Ansible pushes Python scripts to target hosts and executes them via SSH, so the target must have a Python interpreter. Second, systemd's boot sequence and the PAM nologin mechanism: systemd creates /run/nologin during early boot stages to prevent user logins until the system is fully initialized. Third, Docker's docker exec command, which bypasses SSH and PAM entirely to execute commands directly inside a container. Fourth, the test harness architecture: the assistant knew that three target containers existed (kuri-01, kuri-02, s3-fe-01) and that the YugabyteDB container (yb-test) was a separate system not intended for Ansible management. Fifth, the SSH authentication flow: understanding that PAM modules can reject authentication even when the TCP connection and SSH daemon are functioning.

Output Knowledge Created

This message produced several valuable outputs. The most immediate was the negative evidence: confirming that no nologin files existed on any target host, ruling out one class of problem. The command also served as a diagnostic probe—by using docker exec to bypass SSH, the assistant verified that the containers themselves were healthy and that the issue was specific to the SSH/PAM layer or the Ansible inventory configuration. The loop structure of the command (iterating over three hosts with consistent output) established a repeatable diagnostic pattern that could be extended to other investigations.

More broadly, this message contributed to the debugging process by eliminating a plausible hypothesis. In any systematic debugging effort, disproving wrong theories is as important as confirming right ones. The assistant's willingness to test the hypothesis quickly—rather than continuing to speculate—demonstrates good debugging discipline. The three-line bash loop is a model of efficient hypothesis testing: formulate a theory, design a minimal experiment, execute it, and interpret the results.

The Thinking Process

The assistant's reasoning, visible in the brief preamble "Still booting issues. The problem is the systemd pam_nologin," reveals a compressed diagnostic chain. The assistant connected the earlier SSH refusal errors to the current Python interpreter error through a model of progressive boot stages: first the SSH daemon starts but PAM blocks logins (connection refused or authentication failure), then the nologin file is removed and logins succeed but Python is not yet available (if the filesystem is still being set up), and finally the system is fully booted. This model is plausible for a freshly booted system, but it did not account for the fact that the failing host was the YugabyteDB container, which was never intended to be an Ansible target.

The use of docker exec rather than SSH-based probing is a clever methodological choice. By bypassing the very layer being investigated (SSH/PAM), the assistant could isolate whether the problem was at the container level or the SSH level. The consistent "no nologin file" output across all three hosts provided clean, unambiguous data. The assistant then pivoted to a direct SSH test using sshpass in the next message, completing the diagnostic loop: container is healthy, SSH works, so the problem must be in the Ansible inventory or test script configuration.

Lessons for Debugging

This message illustrates several important debugging principles. First, error messages should be read literally before being interpreted through context. The error explicitly named yb-test as the failing host and stated that Python 3 was not found. Reading the error literally would have immediately pointed to the inventory configuration rather than boot sequencing. Second, when multiple issues occur in sequence, each new error should be evaluated independently rather than assumed to be a continuation of the previous problem. The SSH connection refused error (message 1541) and the Python interpreter error (message 1546) were separate issues with separate causes, even though they appeared in the same test step. Third, the fastest way to test a hypothesis is with a minimal, targeted command—the three-line bash loop in this message is a textbook example of efficient hypothesis testing.

The message also demonstrates the value of negative results. The "No nologin file" output across all three hosts was not a fix—it did not resolve the test failure—but it was essential information that narrowed the search space. Without this diagnostic step, the assistant might have continued investigating PAM configuration or added nologin file removal to the Docker image build, introducing unnecessary complexity. Instead, the clean negative result forced a re-examination of the actual error and led to the correct diagnosis in the following messages.

In the end, the fix was simple: exclude the YugabyteDB host from the Ansible connectivity test. But arriving at that fix required ruling out the more complex nologin hypothesis. This message captures the moment of investigation, the testing of a plausible but incorrect theory, and the generation of evidence that redirected the debugging effort toward the real solution.