Secondhand parts, real lessons: diagnosing a dead control-plane node

The homelab motto: if something can break, this is the right place for it to break. Better here than in production, where the stakes are someone else’s business and 3am pages land on people who didn’t design the system. This is the story of a control-plane node that exercised that principle fully — and what a dead CMOS battery, a dining-room workbench, and twenty RAM modules taught me about secondhand hardware.

The node went dark

#

Saturday evening, July 4th. ocp-cp02 sent its last heartbeat to the OpenShift cluster at 8:26 PM EDT. No warning, no graceful drain, no eviction sequence — the kubelet just stopped posting.

From the cluster’s perspective, the damage was immediate:

• etcd degraded: 2 of 3 members healthy, quorum maintained by exactly one member of margin
• Multiple cluster operators reporting degraded
• Pods stuck Terminating across namespaces — Kubernetes deliberately refuses to force-release volumes from a node it can’t confirm is gone
• A Gitea RBD PVC stuck in Multi-Attach error, blocking the pod from rescheduling elsewhere

The cluster kept running. OpenShift with three control-plane nodes tolerates one CP failure for exactly this reason. Degraded is not down. But two healthy control planes is no margin at all — and a cascading failure from there would be a much worse day.

IPMI: the out-of-band lifeline

#

The server is a Supermicro X10DRL-i. IPMI runs on a dedicated management NIC, completely independent of the OS and data plane. Even with RHCOS fully dead, the BMC answered at 192.168.1.225.

First query: the SEL (System Event Log). The log between June 4th and July 5th contained almost nothing — a handful of fan speed events from a single startup sequence a month prior. No hardware events during or before the crash. Over a month of completely quiet sensors right up to the moment the OS disappeared.

The sensor snapshot confirmed the hardware was physically fine:

CPU1 Temp         :  38°C  ✓
CPU2 Temp         :  40°C  ✓
12V               : 11.87V ✓
VDIMMAB/CD/EF/GH  :  1.20V ✓ (all four banks)
VmemABVRM Temp    :  all ok ✓
P1-DIMMB1 Temp    :  No Reading
P2-DIMMF1 Temp    :  No Reading

All voltage rails nominal. All temperatures in range. The two No Reading DIMM sensors are worth noting — they indicate slots the BMC couldn’t read. That could mean empty slots (normal) or something else.

The crash left no hardware fingerprint in IPMI. Whatever killed the OS did it in software, invisibly to the out-of-band management plane.

The BIOS boot mode trap

#

After power-cycling via IPMI, the node came back with IPMI confirming it was powered on — but SSH returned no route to host. Not “connection refused” (which would mean the OS was up but SSH wasn’t listening). “No route to host” means the kernel’s network stack was completely absent. The OS had not started.

The eventual diagnosis: the CMOS battery had died.

When a server with a dead CMOS battery power-cycles, it loses all non-volatile BIOS settings and resets to factory defaults. On a Supermicro X10DRL-i, the factory default is legacy BIOS boot mode. RHCOS on this node was installed in UEFI mode. Legacy boot mode cannot see a GPT disk with an EFI system partition. The NVMe was physically fine, spinning up correctly — it was just invisible to a BIOS configured to look for a legacy MBR.

Result: BIOS searched for a boot device, found nothing it could use, and halted. The OS never started. SSH had nothing to respond to.

The six simultaneous Processor Configuration Errors that appeared in the SEL the following day looked alarming at first glance — six CPU errors at once is not a good sign. But the timestamps told the story: these appeared 21 hours after the crash, during recovery attempts via repeated power cycles. They were BIOS confusion during a crash-loop recovery, not the underlying cause. A clean cold boot after the CMOS battery was replaced produced a completely empty SEL. The CPUs were fine.

Into the dining room

#

I pulled the server from the rack and moved it to my workstation. Which is my dining room table. Homelab logistics are what they are — the rack is not always where you need it to be.

With direct console access to the BIOS POST screen, the picture became clearer. The POST screen showed something IPMI could never capture: “Memory signal is too marginal — P2-DIMMF1”. A physical diagnostic message that the out-of-band management plane has no visibility into.

The physical investigation became systematic:

• Disconnected all PCIe devices
• Removed all DIMMs, cleaned contacts with isopropyl, reseated
• Re-seated CPU2 in its socket
• Replaced the dead CMOS battery
• Corrected BIOS to UEFI boot mode

The node POSTed cleanly and RHCOS booted. Problem solved — except for a pattern that outlasted the battery fix.

Twenty modules and the same result

#

After any hardware change, the server refused to POST. Every single time. Change a DIMM → POST failure → reset BIOS → boots fine. Change it back → POST failure → reset BIOS → boots fine. The node only boots correctly from a fresh BIOS state. Any hardware change that causes BIOS to compare current hardware against its cached training data produces a POST halt.

I tested the RAM. I have a stockpile of secondhand DDR4 ECC RDIMMs — over twenty modules from multiple vendors, different capacities, different speeds. The behaviour was identical across every single one. If a bad DIMM were the cause, somewhere in twenty modules I would have found a working one.

RAM eliminated.

The remaining suspects are two:

The CPU’s Integrated Memory Controller. On Intel Xeon E5 v3/v4, the IMC lives in the CPU die itself, not in the chipset. A degraded IMC can pass a fresh training run — starting from zero state, lenient thresholds — but fail when trying to reproduce a cached training result, which requires the IMC to behave consistently across power cycles. The symptom matches precisely: BIOS reset clears the cache, fresh training succeeds, any subsequent validation against cached results fails.

The motherboard. Specifically, the BIOS flash chip’s training-cache storage region. A flash chip with bad cells in that region writes successfully on fresh training but reads back inconsistently on subsequent boots — producing identical symptoms to a failing IMC.

The X10DRL-i has two CPU sockets. Testing each CPU independently would isolate the cause cleanly. But that’s another pull from the rack, and new parts are already on order.

The secondhand hardware trade-off

#

This server runs entirely on secondhand components. Xeon E5-2600 v4 CPUs cost $15–40 on eBay. The X10DRL-i motherboard runs $50–150. A full replacement set — new CPU and new board — costs less than a single day of equivalent cloud compute.

When you build on secondhand hardware:

• You cannot confirm component provenance
• You do not know what previous workloads did to the silicon
• Seller testing is superficial and often limited to “it POSTed once”
• You accept that something will eventually fail in a way that takes time to diagnose

But “takes time to diagnose” is exactly the point. The troubleshooting process — learning to navigate a POST failure without a display, reading a BMC SEL, understanding what IPMI can and cannot see, keeping a cluster running on reduced quorum while investigating — builds skills that secondhand prices cannot. A $25 CPU that fails teaches you something a $2000 supported enterprise component might never surface in a test environment.

The homelab is the place where cheap components teach expensive skills.

The Kasten backup that flattened the risk

#

Before touching anything destructive, I opened Kasten (Veeam Kasten K10) and checked the most recent backup status.

There was a full namespace backup from approximately 24 minutes before the crash.

I didn’t end up needing a restore. But knowing that backup existed changed every subsequent decision. The internal conversation shifts from “what if I break something during recovery?” to “I can restore to 24 minutes before this started.” That’s not a minor psychological difference — it’s the difference between hesitant, cautious recovery and methodical, confident troubleshooting.

If you’re running stateful workloads in a homelab without scheduled backups: the argument isn’t “you’ll definitely need a restore.” The argument is that having a backup removes the friction that makes every recovery slower and more conservative than it needs to be.

Current state

#

ocp-cp02 is back on the network. RHCOS booted cleanly, kubelet running, all hardware sensors nominal. The cluster recognizes it’s back. It’s carrying two taints:

node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
node.kubernetes.io/unschedulable:NoSchedule

It will stay tainted until the replacement CPU and motherboard arrive. When they do: test CPU1 alone, then CPU2 alone, then conclude. Whatever the root cause, the fix is measured in tens of dollars.

In the meantime, the cluster runs on two healthy control planes and one worker. etcd quorum is maintained. The workloads that were on cp02 have rescheduled. The experiment ran exactly as designed.

With a homelab, if it can break, this is where it breaks — and that is exactly the point.