On 20 October 2026, at 10:31 UTC, the Fly Managed Postgres cluster behind our AI Invoice Importer failed over. For the next 55 minutes, every background write — retrying a failed invoice push, refreshing a Xero token — crashed with psycopg2.errors.ReadOnlySqlTransaction: cannot execute UPDATE in a read-only transaction. All ten single-org users and every multi-org connection hit it.
/api/health said database: healthy the entire time.
Why the health check lied
Our connection pool had server connections pinned to the node that had just been demoted to a standby. pool_pre_ping ran SELECT 1 before handing out a connection, and SELECT 1 works fine on a read-only replica, so the pool looked fine. /api/health ran the same kind of read-only probe. Neither one could tell "the database is up" apart from "the database will accept a write", because neither one tried to write.
That's the belief we got burned into holding properly: a health check that only reads isn't checking the thing that actually matters. If your app writes, your health check has to prove it can still write. Otherwise it measures the wrong thing and will happily report green while every job in the queue fails.
We found out because the retry worker logged "tick crashed" every 60 seconds and Sentry raised two errors (PYTHON-FASTAPI-6F/6G), not because anything paged us from the health check. A human restarted the machine at 11:25 UTC. The fix had to be: never need that human again.
What we changed
Three pieces, all application-only, documented in issue #128 and PR #142:
- A pool-poison classifier. It recognises SQLSTATE 25006 (read-only transaction), a closed SSL connection, or an invalidated DBAPI connection. On a poison error, the background job rolls back, disposes the engine so the pool rebuilds from scratch, and retries the tick exactly once. A second failure still propagates — we're not hiding a genuinely broken database.
/api/healththat checkspg_is_in_recovery()andtransaction_read_only. If either says the node can't take writes, the probe reports unhealthy and disposes the pool itself, so the next check reaches the real primary without anyone touching a terminal.pool_recycle=300, so a stale connection can't survive indefinitely even if the first two checks both miss something.
The test that matters is the one built to break it: a second Postgres connection with default_transaction_read_only = on, reproducing exactly what the live pool saw. With the fix in, the worker, the token-refresh job and /api/health all detect it, dispose, and recover on the next tick. Pull the fix back out and 14 of the 17 new tests fail. The full suite: 2,108 passed.
Nobody's invoices were lost — the retry queue stayed empty and nothing had reached the point of being created when the failover hit, so customer impact this time was nil. That's luck in the timing, not a property of the system we had before this week.
Where it stands
The PR is open, waiting on an independent review before it merges to a branch that auto-deploys production — the same bar as every runtime change we ship, incident or not. If you run anything with a connection pool in front of managed Postgres, Aurora or Cloud SQL and your health check doesn't try to write, go check it today. Failovers don't ask permission.