When your object store eats its own metadata: moving Loki/Mimir/Tempo to plain NFS

It started, as these things do, with a crash loop. Loki on the new OpenShift homelab cluster was restarting every couple of minutes, and the logs were a wall of red:

caller=flush.go:178 ... msg="failed to flush" err="failed to flush chunks:
  store put chunk: NoSuchBucket: Bucket not found: loki-chunks"
caller=compactor.go:477 ... msg="error asking ring for who should run the
  compactor" err="at least 1 healthy replica required, could only find 0"

Two different errors, one underlying mess. This is the story of chasing it down to the object store — Garage — and then deciding the object store wasn’t worth keeping at all.

The trail: NoSuchBucket

#

Loki, Mimir and Tempo were all configured to use Garage (an S3-compatible store) as their backend, with buckets loki-chunks, mimir-blocks, tempo-blocks and friends. The NoSuchBucket error meant exactly what it said: the bucket Loki wanted was gone. A quick garage bucket list confirmed it — only tempo-blocks survived; everything else had vanished.

Buckets don’t usually evaporate. So why?

# garage statefulset.yaml — the smoking gun
volumes:
- name: meta
  emptyDir: {}        # <-- garage's LMDB metadata, on ephemeral storage

Garage keeps two things on disk: data (the actual object blobs) and metadata (an LMDB database of buckets, access keys and the cluster layout). The data dir was on a persistent NFS PVC. The metadata dir was on an emptyDir. Every time the Garage pod restarted — a node drain, an OOM, anything — the LMDB was wiped, and with it every bucket, key and grant. The data blobs were still on NFS, but orphaned, because nothing remembered they existed.

So the timeline was: Garage pod restarted ~an hour earlier → metadata gone → buckets gone → Loki’s flushes started 404ing → flush backlog saturated the pod → liveness probe timed out → kill → restart → repeat. The “unhealthy ring” error was a symptom: a process too busy failing to flush to heartbeat its own ring.

The second bug: split-brain

#

While I was in there, a second problem surfaced. Garage ran two replicas, but:

rpc_public_addr = "0.0.0.0:3901"   # every node advertises 0.0.0.0
replication_factor = 1

With rpc_public_addr hardcoded to 0.0.0.0, each Garage node advertised itself as 0.0.0.0 and the two could never actually peer. garage status from node 1 listed node 0 as “never seen”. With replication_factor = 1, every object lived on exactly one node — so any S3 request the Kubernetes Service load-balanced to the wrong node failed with a quorum 503. Half the writes were doomed by a coin flip. The two pods were also mounting the same RWO PVC for their data dir, which is its own kind of wrong.

I stabilised it first the boring way — recreated the buckets by hand, scaled Garage to the single node Loki was actually talking to, moved meta onto the persistent PVC via a subPath so it would survive restarts, and re-initialised the layout/key/buckets. Loki went green. Crisis over.

The pivot: do we even need an object store?

#

Here’s the thing about a homelab: all of this — Garage data, Garage metadata, every app’s PVC — ultimately lives on one NFS server. I’d been about to rebuild Garage as a proper 3-node replication_factor=3 cluster for high availability, until I said the quiet part out loud: three replicas writing three copies to the same NFS box buys you availability, not durability. If that one server dies, all three copies die together. You’re paying 3× write amplification to a single spindle for an availability win on a telemetry store that already tolerates a few seconds of unavailability.

And Garage had just demonstrated it was a source of outages, not a buffer against them. The failure mode that actually bit us — metadata on emptyDir — had nothing to do with node count.

So the decision: take Garage out of the data path entirely. Loki, Mimir and Tempo can all write straight to a filesystem. Point them at the NFS telemetry store and skip the S3 indirection.

The new layout

#

Everything now lives under one NFS export, organised by app:

10.20.30.7:/var/nfs/shared/telemetry/grafana/
├── loki/    (chunks, index, wal, compactor)
├── mimir/   (blocks, tsdb, ruler-store, alertmanager-store)
└── tempo/   (blocks, wal, generator-wal)

Each app gets a PV pointing at /var/nfs/shared/telemetry, mounted at its data dir with subPath: grafana/<app>. The config change per app:

• Loki — object_store: filesystem. While here, I bumped the ingestion rate limit (the default 4 MB/s couldn’t keep up with cluster-wide pod logs) and max_line_size to 1 MB, because the AAP automation-controller cheerfully emits single 386 KB log lines.
• Mimir — backend: filesystem for blocks, ruler and alertmanager storage; rings stay in-memory. Raised max_global_series_per_user 1M → 2M, since OpenShift platform + user-workload monitoring blew past 1M.
• Tempo — backend: local.

One replica, on purpose

#

This is the trade-off worth being explicit about. Filesystem and local backends are single-writer. Tempo’s local backend flatly does not support multiple replicas — two compactors on one directory will corrupt blocks. Loki and Mimir can technically treat a shared NFS dir like an object-store bucket with a memberlist ring, but it’s off the supported path and the compactor can race.

So all three run a single replica with a Recreate rollout strategy — never two pods on the same directory at once. Durability comes from NFS; the apps buffer in their WAL and retry across the brief gap when a pod reschedules. For a homelab telemetry stack, that’s the right amount of engineering.

The gotcha that cost twenty minutes

#

Swapping each app onto a new PV meant changing the PVC’s volumeName — which is immutable on a bound claim. Fine, delete and recreate the PVC. Except kubectl apply -k immediately scales the Deployment back to 1, and the new pod grabs the still-terminating PVC, re-adds the pvc-protection finalizer, and now you have a pod happily running on the old storage while the PVC hangs in Terminating forever.

The fix is patience and ordering:

scale Deployment to 0  →  wait for ALL pods gone
                       →  delete the PVC
                       →  poll until it's actually gone
                       →  THEN apply -k (fresh PVC binds the new PV)

Don’t apply while the old PVC is terminating. Learned that twice before it stuck.

Where it landed

#

All three are 1/1 Ready, writing to telemetry/grafana/{loki,mimir,tempo}, with zero S3 or quorum errors in the logs. The Mimir series-limit drops are gone. Garage is no longer in the picture — one fewer distributed system to babysit, and the failure mode that started all this (metadata on emptyDir) is structurally impossible now, because there’s no metadata database to lose.

Sometimes the most reliable object store is the one you delete.