The cluster survived the hardware failure. The backups existed. Everything that was supposed to be robust, was. And yet — when the dust settled and I started checking on services, the shared Postgres cluster had promoted the wrong instance, and a dozen apps were staring at a database with one fifty-sixth of the expected data.
This is the story of what happened after ocp-cp02 died, and what it taught me about automatic failover, WAL archives as a blast radius, and why Postgres refusing to do something is almost always the right call.
What was running on that node #
Three control-plane nodes, one worker, Rook-Ceph for storage, CloudNativePG for the shared Postgres that backs nearly everything: Authentik, Gitea, n8n, Coder, Postiz, Airflow, NetBox, Temporal, Gatus, Umami. Over a dozen applications sharing one postgres-cluster with three instances. It’s the single most connected piece of infrastructure in the whole setup, which made what happened next cascade exactly as predictably as it sounds.
The cascade that nobody scheduled #
When cp02 died, the immediate damage was to whatever pods were running there. But the second-order effect was what really bit: Garage — the S3-compatible object store used as CNPG’s WAL archive target — had its primary instance on cp02. Kubernetes, correctly, refused to release the Garage pod’s RBD volume from a node it couldn’t confirm was actually dead. The pod sat in Terminating forever.
Garage down → barman-cloud-wal-archive failing on every WAL segment → CNPG treating the WAL archiving failures as a signal that something was deeply wrong → automatic failover.
So far, standard HA behavior. The failure is detected, a new primary is elected, life goes on.
Except life didn’t quite go on.
Promoted: the instance with 1/56th of the data #
CNPG had three instances: postgres-cluster-1, postgres-cluster-3, postgres-cluster-4. During the chaos of the crash, whichever instance responded at the right moment got promoted — cluster-4 became the new primary.
I noticed something was wrong when apps that should have been healthy started erroring. I checked pg_controldata across instances:
postgres-cluster-1: timeline=6, NextXID=0:1300457 (~1.3M transactions)
postgres-cluster-4: timeline=7, NextXID=0:23660 (~23K transactions)Same Database system identifier. The lineage is real — these started as the same cluster. But cluster-4 had roughly 1/56th the transaction history of cluster-1. Databases like Gitea, n8n, Coder, Postiz, Temporal, NetBox, and Airflow didn’t exist on cluster-4 at all. Even the databases that were there — Authentik, Gatus, Umami — were stale.
Why this can happen: CNPG’s automatic failover picks based on availability, not on comparing WAL position across instances that are themselves unreachable at decision time. If the instance with the actual up-to-date data (cluster-1) is down or unreachable at the moment of failover, and a stale instance (cluster-4) is the only one that responds, CNPG has no choice. It can only promote from what it can see. This isn’t a CNPG bug — it’s an inherent limitation of failover-under-partition. The system made the only decision it could make with the information it had.
The error that told me everything #
Once I spotted the transaction-count mismatch, I wanted to point things back at cluster-1. The first sign that things were more interesting than a simple status correction: Postgres itself refused to start as a replica of cluster-4:
FATAL: requested timeline 7 is not a child of this server's history
DETAIL: Latest checkpoint in file "pg_control" is at 17/DA009AC0 on
timeline 6, but in the history of the requested timeline, the server
forked off from that timeline at 1/6A001E48.Translation: cluster-4’s new timeline (7) branched off from a point much earlier than where cluster-1 had already been. Postgres was correctly refusing to let a more advanced node follow a less advanced one, because doing so would silently discard data. This is Postgres’s own built-in safety net. It is not failing — it is working exactly as designed. Every time you see this error, Postgres is protecting your data. Trust it and dig into why instead of trying to force past it.
Getting CNPG to accept the correction #
This part required working through several approaches that didn’t quite land before finding one that did. Each failed attempt is worth noting because the failure mode is informative:
Direct status patch (kubectl patch cluster ... --subresource=status -p '{"status":{"targetPrimary":"postgres-cluster-1"}}'): worked for about a second. The CNPG operator’s reconcile loop silently reverted it back to cluster-4, because from the operator’s perspective, cluster-4 was healthy and it had no reason to distrust it.
cnpg.io/fencedInstances annotation: CNPG’s built-in pause-in-place mechanism. Fencing the current primary got further — the operator logged “the current primary instance is fenced, we won’t trigger a switchover” — but it also refused to auto-elect a replacement, leaving the cluster in a stalled state. Fencing is designed for planned maintenance, not for disqualifying a wrong winner.
kubectl cnpg promote: The actual supported plugin command (install via brew install kubectl-cnpg). Reported success, but cluster-1 still couldn’t come up for reasons explained below.
Scaling .spec.instances down to 1: CNPG kept whichever pod was the recorded primary, not the one with the best data. Scaling doesn’t re-evaluate which instance should be primary.
What actually worked: Fence both non-target instances (stops their postgres processes entirely, removes any ambiguity for the operator) + patch both .status.targetPrimary and .status.currentPrimary to cluster-1 + delete the cluster-1 pod. That last step is the one that’s easy to miss: CNPG bakes the replica/primary decision into the pod at creation time. A live pod won’t re-evaluate a status change without a restart. Delete the pod, it comes back as a primary, things proceed.
Except they didn’t — yet.
The poisoned WAL archive #
Even after all of the above, cluster-1 kept hitting the exact same FATAL: requested timeline 7 error, byte-for-byte identical, on every restart. That repetition is the tell that something external and stateful was involved, not the live cluster state.
When cluster-4 got wrongly promoted, it wrote its own timeline history file (00000007.history) into the shared S3 WAL archive that all instances read from for archive-based recovery. Postgres, by design, always tries to recover toward the latest available timeline in the archive. So cluster-1 kept rediscovering the bad timeline-7 marker in Garage and trying to follow it — no matter what the live cluster status said. The live state was correct; the archive told a different story.
Fix: list the archive, identify every object that belonged to the bogus lineage, delete only those.
aws --endpoint-url http://garage.backup-s3.svc.cluster.local:3900 \
s3 ls s3://cnpg-backup/postgres-cluster/wals/ --recursiveReading a Postgres WAL archive: filenames follow the pattern <timeline><log>/<timeline><log><segment>. The bogus objects were:
wals/00000007.history— cluster-4’s timeline declarationwals/0000000700000001/<segment>— WAL segments cluster-4 wrote on timeline 7wals/0000000100000001/<segment>— early timeline 1 artifacts from cluster-4’s initialization
Seventeen objects total. The real data — every wals/0000000600000017/* object — was left completely untouched. Deleting the seventeen bogus objects and restarting cluster-1 produced a clean recovery on timeline 6. The cluster came up, all databases present, transaction counts correct.
What Ceph did (which was almost nothing) #
One thing worth noting specifically because it’s easy to overlook: Rook-Ceph stayed boring. cp02 hosted two of the six OSDs. The main replicated pool (size=3) had zero availability impact because the other four OSDs on the remaining three nodes covered it. Reads and writes kept working throughout. The only storage that went dark was ai-models-pool — intentionally configured at size=1 because the data it holds (model weights) is large but re-downloadable, not something that needs triple replication. That’s a deliberate and documented trade-off, not a failure.
The blast radius from one dead node was: Postgres chaos (as above), ~130 pods stuck Terminating across nearly every namespace (resolved quickly once the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint was applied, which tells the CSI controller the node is confirmed gone and force-releases RWO volumes), and about 30 hours of degraded 2-of-3 control-plane quorum. Significant, but not catastrophic. The cluster kept running.
The backup that changed the risk calculus #
Before touching anything destructive in the recovery process, I opened Kasten and checked the most recent backup.
Full namespace backup, ~24 minutes before the crash.
I didn’t need it. But knowing it was there changed every decision that followed. Without that backup, every step of the CNPG manipulation — fencing instances, patching status, deleting pods, removing WAL archive objects — carries the weight of what if I make this worse? With the backup, the question becomes what’s the worst-case rollback point if something goes wrong? Twenty-four minutes. That’s a very different conversation.
If you’re running stateful workloads without scheduled backups: the argument isn’t “you’ll definitely need a restore.” It’s that the backup removes friction from every recovery decision. Confident, methodical troubleshooting is faster than cautious, hesitant troubleshooting.
What to do differently next time #
After any automatic failover, before trusting the result: run pg_controldata across every instance and compare timeline + NextXID. Don’t assume the promoted instance is the one with the most data. CNPG’s status.currentPrimary tells you who won; it doesn’t tell you if the winner was correct.
A shared WAL archive is a shared blast radius. If any instance gets wrongly promoted and writes to the shared archive, it can poison recovery for every other instance using that archive. The fix is surgical — identify exactly which archive objects belong to the bogus lineage and delete only those — but you need to understand WAL archive layout to do it safely.
Postgres refusals are data protection, not failure. Every time Postgres refused to start in this recovery — the timeline fork error, the history-mismatch error — it was correctly protecting data from being silently discarded. Trust the refusal. Understand why before acting. The system was right every time it said no.
kubectl cnpg > raw patches for HA operations. But even the plugin can’t fix a poisoned WAL archive. Know when the problem has moved up a layer from “cluster state” to “storage backend.”
The hardware investigation that preceded this — why cp02 died in the first place — is a separate post. Short version: dead CMOS battery, BIOS factory reset, legacy boot on a UEFI-only RHCOS install, new parts on order.