A Read-Only Database and the Perils of Presumed Root Access

In message [msg 1532] of a sprawling opencode coding session, the assistant issues a single bash command over SSH:

ssh 10.1.2.104 "sudo systemctl stop vast-manager; sleep 1; sqlite3 /var/lib/vast-manager/state.db 'UPDATE host_perf SET bench_rate = 61.36 WHERE machine_id = 15136;'; sudo systemctl start vast-manager"
Error: stepping, attempt to write a readonly database (8)

The command is straightforward in intent: stop the vast-manager service, wait a moment, patch a single row in a SQLite database, then restart the service. But the execution fails with a cryptic SQLite error — "attempt to write a readonly database" — and the error code 8. This short message, barely a line of code and its output, encapsulates a surprisingly deep debugging episode that spans permission boundaries, SQLite concurrency semantics, and the subtle ways that privilege escalation chains can break in practice.

The Broader Context: A Benchmark Data Integrity Crisis

To understand why this message exists, one must understand the crisis that preceded it. The assistant had built a distributed GPU benchmarking platform on vast.ai, a marketplace for renting cloud GPU instances. Each rented instance runs a benchmark to measure its proofs-per-hour rate, and the results are stored in a host_perf SQLite table on a central management server. This data drives critical decisions: which machines to keep, which to destroy, and what minimum performance thresholds to set.

In the messages leading up to [msg 1532], a subtle data integrity bug had been discovered. The host_perf table used an INSERT ... ON CONFLICT ... DO UPDATE pattern that unconditionally overwrote existing records with new results. This became a problem when two benchmark instances ran simultaneously on the same physical machine (machine_id 15136, a server in Norway with an RTX 4090). The first instance completed with an impressive 61.4 proofs/hour. The second, suffering from resource contention because it shared GPU and memory bandwidth with the first, managed only 46.2 proofs/hour. Because the second result arrived later, it overwrote the first, and the host_perf table now showed the lower value — a misleading record that would cause the system to underestimate the machine's true capability.

The assistant had already addressed the root cause by modifying the Go source code in [msg 1527], changing the ON CONFLICT clause to use bench_rate = MAX(bench_rate, ?) so that only improvements would be recorded. The binary was rebuilt and deployed in [msg 1529]. But the existing record in the database still held the wrong value. This message is the attempt to surgically correct that historical record.

The Three Attempts and a Persistent Error

The message in [msg 1532] is actually the third attempt to fix this single database row. In [msg 1530], the assistant tried a direct SQLite command without stopping the service first:

ssh 10.1.2.104 "sqlite3 /var/lib/vast-manager/state.db \"UPDATE host_perf SET bench_rate = 61.36 WHERE machine_id = 15136; SELECT * FROM host_perf ORDER BY bench_rate DESC;\""
Error: stepping, attempt to write a readonly database (8)

The assistant correctly diagnosed this as a lock issue: "The DB is locked by vast-manager." In [msg 1531], the approach was refined to stop the service first:

ssh 10.1.2.104 "sudo systemctl stop vast-manager && sqlite3 /var/lib/vast-manager/state.db \"UPDATE host_perf SET bench_rate = 61.36 WHERE machine_id = 15136;\" && sudo systemctl start vast-manager && sleep 1 && sqlite3 /var/lib/vast-manager/state.db \"SELECT * FROM host_perf ORDER BY bench_rate DESC;\" 2>/dev/null || ssh 10.1.2.104 'curl -s http://127.0.0.1:1235/api/host-perf'"
Error: stepping, attempt to write a readonly database (8)
Host key verification failed.

Same error. The assistant's message in [msg 1532] is the third attempt, now using semicolons instead of && for command chaining and adding an explicit sleep 1:

ssh 10.1.2.104 "sudo systemctl stop vast-manager; sleep 1; sqlite3 /var/lib/vast-manager/state.db 'UPDATE host_perf SET bench_rate = 61.36 WHERE machine_id = 15136;'; sudo systemctl start vast-manager"
Error: stepping, attempt to write a readonly database (8)

Still the same error. The persistence of the "readonly database" message despite stopping the service is the key mystery.

Root Cause: The Missing Sudo

The critical detail is hiding in the command structure. The sudo keyword in the command only applies to systemctl stop vast-manager. The subsequent sqlite3 command runs without sudo. In a chain of commands separated by semicolons or &&, privilege escalation does not carry forward — each command inherits the privileges of the SSH session, not the privileges of the previous command.

The database file at /var/lib/vast-manager/state.db was created by the vast-manager service, which likely runs as root or a dedicated service user. Its file permissions almost certainly restrict write access to the owner (root). When sqlite3 is invoked by the regular SSH user, it opens the database file and finds it readable but not writable. SQLite's error code 8, "attempt to write a readonly database," is the library's way of saying: "I can read this file, but I cannot write to it, and you just tried to execute an UPDATE statement."

The assistant had assumed that stopping the service with sudo was the critical prerequisite, and that once the service was stopped, the database would be accessible. This was a reasonable assumption — in many configurations, SQLite databases in /var/lib/ are world-writable or group-writable. But this particular deployment had stricter permissions, and the assumption proved incorrect.

What This Message Teaches

This message is a masterclass in the subtlety of Unix privilege models. The error message — "readonly database" — is misleading because it sounds like a filesystem-level problem (a read-only mount, or a file with 0444 permissions). But the real issue is a mismatch between the user running the query and the file's ownership. SQLite reports "readonly" when it opens a file successfully for reading but the file descriptor doesn't have write permission, which is exactly what happens when a non-root user opens a root-owned file.

The message also illustrates a common pattern in debugging: when an error persists across multiple attempts, it's time to question the assumptions underlying all of them. The assistant tried three variations — different command chaining, different timing, different quoting — but never varied the one thing that mattered: the privilege level of the sqlite3 invocation. The fix would have been trivial: sudo sqlite3 ... instead of sqlite3 ....

Knowledge Required and Created

To fully understand this message, the reader needs knowledge of: SQLite's error codes and their meanings; the Unix privilege model and how sudo scoping works across command chains; the systemd service lifecycle and how systemctl stop interacts with running processes; and the specific schema and purpose of the host_perf table in the vast-manager application.

What this message creates is a concrete demonstration of a permission boundary failure in a distributed systems context. It shows how a seemingly administrative task — updating a database record — can fail in unexpected ways when the privilege model is not fully considered. It also documents a specific operational pattern: when a service-managed process owns a SQLite database, any direct database manipulation must run with the same privileges as the service itself, even after the service is stopped.

The broader lesson is about the importance of holistic privilege management. In distributed systems, where commands are composed across SSH sessions, shell pipelines, and service managers, it is easy to assume that a single sudo at the beginning of a chain covers everything that follows. This message proves otherwise, and the proof comes in the form of a single, stubborn error code that refused to go away until the underlying assumption was examined.