Ripping out Ceph: an overnight ODF to Longhorn migration

ODF was costing me 62 pods, about 900 millicores and 20 GiB of RAM to serve roughly 13 GiB of actual data. On a four-node homelab that is an absurd trade. So I pulled it out entirely and replaced it with Longhorn 1.12.0.

The short version: it worked, nothing was lost, and every service came back. The long version is a catalogue of things that fail silently — configuration that applies cleanly and simply does nothing. Those are the interesting ones, so that is most of what follows.

The constraint that shaped everything

#

One requirement drove the design: lose any single node with no impact. That meant one dedicated disk per node, four nodes, three replicas with hard anti-affinity so no two replicas of a volume ever share a node.

I had five SSDs available — four 466 GiB and one 233 GiB. The temptation was to hand Longhorn all five. I deliberately did not. The odd 233 GiB disk lives on cp01, and adding it would have let two replicas of the same volume land on the same physical node while Longhorn still reported everything healthy. It became a separate fast-local StorageClass via the Local Storage Operator instead. One disk per node keeps the guarantee unambiguous.

Ceph does not let you leave gradually

#

My first instinct was to drain one OSD at a time, feeding disks to Longhorn as they freed up. The safety review killed that idea, correctly.

Every Ceph pool here was size: 3 with a host failure domain, and three of the four 466 GiB disks were the only OSD on their node. Removing one is fine. Removing a second drops you to two OSD-bearing hosts — below the minimum for 3× replication — and Ceph sits in permanent PG_DEGRADED no matter how much free space remains.

There is no safe half-migrated state. Ceph had to be emptied completely and destroyed in one go. That reframed the whole job: copy everything off first, verify it, then demolish.

So: 31 PVCs copied to NFS, byte counts and file counts verified on both sides, plus an independent pg_dumpall of the shared Postgres. Only then did the StorageCluster get deleted.

Silent failure #1: the backup target that wasn’t

#

Longhorn 1.12.0 moved the backup target out of defaultSettings.backupTarget into a top-level defaultBackupStore: block, where it materialises as a BackupTarget CR.

My original values used the old key. Helm rendered it without complaint. The install succeeded. And there was simply no backup target — no error, no warning, nothing.

I only caught it because I grepped the rendered chart instead of trusting that the install had done what I asked:

helm template longhorn longhorn/longhorn --version 1.12.0 -f values.yaml \
  | grep -i backup-target

Empty output. That habit — render and inspect rather than install and hope — caught two more problems the same night.

Silent failure #2: the setting that quietly guaranteed data loss

#

This is the one that would have actually hurt.

I set replicaZoneSoftAntiAffinity: false, reasoning that “hard anti-affinity” was what I wanted. What that flag actually does is make zone anti-affinity hard — and this is a single-zone homelab with no zone labels on any node.

The result: Longhorn could only ever place one replica per volume. Every volume sat degraded with two unscheduled replicas, while all four nodes reported Ready and Schedulable with 322 GiB free each. The volume condition said ReplicaSchedulingFailure and nothing anywhere explained why.

Node-level anti-affinity (replicaSoftAntiAffinity: false) is what delivers single-node-failure tolerance. The zone flag must be true in a single-zone cluster.

Had I accepted “install succeeded, pods are running” as done, I would have shipped single-replica volumes with zero redundancy — the exact opposite of the requirement. The only reason I found it is that I explicitly tested replica placement:

oc get replicas.longhorn.io -n longhorn-system -o json \
  | jq -r '.items[] | select(.spec.volumeName=="'"$VOL"'") | .spec.nodeID'

Three distinct node names, or it is not doing what you think.

Silent failure #3: thin provisioning versus the scheduler

#

Then volumes started going FAULTED with Multi-Attach errors.

Longhorn schedules on requested PVC size, not actual usage. My PVCs requested 513 GiB total; at three replicas that is 1,539 GiB of scheduling demand against roughly 1,304 GiB usable. Actual data on disk: about 13 GiB. The volumes are thin, but the scheduler does not care.

At the default storageOverProvisioningPercentage: 100, Longhorn refuses to schedule. Raising it to 500 fixed it instantly. Worth knowing before you inherit a pile of over-generous PVC requests, which most of us have.

Racing my own teardown

#

A failed first install left a broken Helm release. I uninstalled and reinstalled — and the managers started failing with the server could not find the requested resource (get nodes.longhorn.io).

The CRD count was drifting downward: 23, then 21, then fewer. I was patching individual CRDs back while something removed them.

That something was my own helm uninstall. Longhorn ships its CRDs as chart templates rather than in crds/, so uninstall deletes them — asynchronously. My reinstall was recreating them while the previous teardown was still working through the list.

The fix was to stop being clever: uninstall with --no-hooks, delete every Longhorn CRD explicitly, delete the namespace (which needed finalizer clearing on two stuck CRDs and a raw /finalize API call), wait until there were genuinely zero CRDs and no namespace, and only then install. It worked first time.

CNPG lied to me about the cause

#

The worst debugging detour of the night. With storage rebuilt, CloudNativePG refused to start:

phase=Cluster is unrecoverable and needs manual intervention
reason=One or more instances were previously created, but no
       PersistentVolumeClaims (PVCs) exist.

The PVCs were plainly there and Bound. I spent real time adding CNPG’s ownership labels and annotations to them, patching status, deleting the hibernation annotation — all chasing that message.

The actual cause was in the Cluster spec:

spec:
  storage:
    storageClass: ocs-storagecluster-ceph-rbd   # deleted hours earlier

That was it. The “no PVCs exist” message was noise. The lesson I keep relearning: when an operator’s status message and the observable state disagree, read the operator’s reconcile log, not its status field. The log said what it actually wanted:

refusing to create the primary instance while the latest generated serial is not zero

Two smaller landmines came with it. Orphaned CNPG webhooks with failurePolicy: Fail still pointed at NooBaa’s bundled CNPG operator — deleted along with ODF — which blocks every CNPG operation cluster-wide. And the Cluster CR’s status had latched, needing delete --cascade=orphan plus a re-apply. That was safe only because I checked first that the PVCs had no ownerReferences and the PVs were Retain; otherwise deleting the CR would have garbage-collected the data I had just spent hours restoring.

Ownership is the real restore footgun

#

Copying bytes is the easy part. Making an application accept them is not.

Every restored volume that failed did so the same way:

chmod /var/lib/postgresql/data/pgdata: operation not permitted

Wrong file ownership. And the correct UID differs per application:

I assumed the OpenShift namespace UID range everywhere, which is wrong for any image that pins its own user. Read spec.template.spec.securityContext.runAsUser (and the container-level override) first; fall back to the namespace annotation only when it is unset.

The compounding mistake was mine. My restore script contained:

chown "$u:$g" "$p" 2>/dev/null || true

That || true swallowed every failure. The copy reported success, verified byte-for-byte, and the application crashlooped hours later. If a step matters, let it fail loudly.

Where a root pod could not chown at all, the trick was to run the restore pod as the target UID so tar creates files owned correctly and no chown is needed.

Other things that bit

#

StatefulSet volumeClaimTemplates are immutable. AAP’s Postgres still referenced the deleted cephfs class and could not be patched. The fix was to update the AAP CR, then delete sts --cascade=orphan so the operator recreated it with the right template and adopted the already-populated PVC. MariaDB’s spec.storage.storageClassName has the same immutability — it still names the dead class, harmless until that PVC is ever recreated.

OLM reverts oc scale --replicas=0 on operator Deployments within minutes. AAP’s seven operators kept resurrecting and recreating PVCs against a StorageClass that no longer existed. To actually stop an OLM-managed operator you must remove its Subscription and CSV.

MCO ignores SIGTERM mid-update. During the iscsid rollout, the machine-config daemon logged Got SIGTERM, but actively updating and kept running — my pod deletes did nothing. It needs --grace-period=0 --force. Worse, I restarted the machine-config controller mid-drain, which wiped its in-memory drain state and stranded the node forever on the desiredDrain/lastAppliedDrain handshake. Do not restart the controller while a node is draining.

The one that is still broken

#

Longhorn’s NFS backup target does not work here, and it is not a configuration mistake.

Longhorn’s NFS backupstore mounts with a hardcoded -t nfs4. Passing ?nfsOptions=nfsvers=3 is silently overridden by the mount type. My NAS serves these paths over NFSv3 only — every working kubelet PV mount on the nodes shows vers=3, and a plain test pod mounts and writes that exact directory without complaint.

So the path, permissions and directory are all correct; only NFSv4 access is missing. That is a change on the NAS, not the cluster. Options are to enable NFSv4 on the export, or point Longhorn’s backups at the Garage S3 instance already running in-cluster. For now Kasten still backs up to NFS independently, so nothing is unprotected — but I would rather say plainly that it is unfinished than quietly leave it looking configured.

Where it landed

#

Four nodes Ready, zero degraded cluster operators. Postgres back with all 18 databases (Authentik alone has 215 tables), MariaDB with its 8, Authentik answering 200. Twenty-seven Longhorn volumes healthy, each with three replicas on three distinct nodes. ODF removed down to zero CRDs, zero CSVs, zero pods.

Resource cost of the storage layer went from 62 pods and ~20 GiB of RAM to a handful of Longhorn pods.

The through-line, if there is one: almost everything that went wrong applied cleanly and did nothing. The backup target, the anti-affinity flag, the ownership replay, the over-provisioning limit — every one of them was accepted without error and silently failed to do the thing I asked. The only defence is to verify the outcome rather than the command. Render the chart and read it. Query the replica placement. Open the database and count the tables.

“It applied successfully” is not the same as “it works”, and storage is a bad place to learn the difference.