Blood, Sweat & Solder: A Year Rebuilding My Homelab Into a Private Cloud

One year. Thousands of dollars in and out of the same PayPal account. A handful of cuts on my hands, more sweat than I’d like to admit in a closet that was never meant to hold a server rack — and at least one night where I gave my AI agent a sloppy instruction and watched it confidently undo three days of work in the time it takes to make coffee.

It’s done. The rebuild is complete.

The finished article: a full private cloud living in a closet.

What’s in that cabinet now is not a pile of hobby gear. It’s a 3-node OpenShift control plane plus a GPU worker, backed by Ceph for storage, running a GitOps-driven platform with real observability and a private, self-hosted LLM — 72 cores / 144 threads and 768 GB of ECC memory all told. It is, for all intents and purposes, a miniature version of the exact systems I want to help run at scale.

Here’s how it came together, and what it cost me to get here.

Giving enterprise hardware a second life
#

This didn’t start with a shopping cart. It started with patience and a lot of lost eBay auctions. I sold old gear to fund new gear, won some auctions, lost more, and gave a small fleet of retired enterprise servers a second home:

2× Supermicro X10DRL-i — dual Xeon E5-2630L v4, 256 GB DDR4 ECC each
2× Supermicro X10SDV-16C-TLN SoC boards — 128 GB DDR4 ECC each
4× 500 GB SSDs — one per node, pooled into Ceph
1× NVIDIA A2 16 GB accelerator for the AI workloads

Every one of these was opened, cleaned, re-pasted, and rebuilt by hand.

The platform
#

On top of that hardware sits the part that actually matters: a platform that behaves like one. OpenShift on RHEL CoreOS, Ceph/ODF for block, file and object storage, ArgoCD driving everything from Git, and a full VictoriaMetrics + Grafana observability stack watching all of it. On the AI side: vLLM serving the model, with LiteLLM, OpenWebUI, Qdrant and CloudNativePG rounding out a private AI platform that never sends a token to anyone else’s cloud.

If it moves, it’s on a dashboard. Node thermals sit right next to etcd latency.

The cooling system I couldn’t buy, so I built
#

Here’s the part I’m proudest of.

A sealed rack in a closet gets hot, fast. I went looking for a controller that could do two simple things: drive the fans off actual temperature and humidity, and hand me metrics I could graph next to the rest of my cluster. Nothing off the shelf did both. So I built it.

Cutting shrouds by hand. There was no CAD file — just a receipt, scissors, and stubbornness.

The brain is a Raspberry Pi Zero running a small Python service. It reads a pair of AM2302 / SHT30 temperature-and-humidity sensors, runs the fan curves, and drives a bank of PWM fans mounted on a custom 2U hex-mesh panel I fabricated to fit the rack. Power comes off a 12 V→5 V step-down converter, the sensor runs are made from thermostat wire, and yes — I sat there placing pull-up resistors by hand.

Four fans, a Pi, a breadboard, and every screwdriver bit I own.

The controller itself — a Pi Zero, wired to the fan bank and the sensors.

One of the temperature/humidity sensors that closes the loop.

The Python service doesn’t just spin fans — it exposes Prometheus metrics, so the rack’s thermals land on the same Grafana dashboards as everything else. When the closet warms up, I can see it.

The completed unit before it went into the rack.

Installed. The payoff for a lot of late nights.

The blood part isn’t a figure of speech, by the way.

Sweaty hands, sharp panels. The tax you pay for building it yourself.

The AI chapter — the wonder and the cliff
#

The A2 earns its slot. Serving a local model with vLLM, I benchmarked how latency scales with context length, and the shape of the curve tells its own story:

| Prompt size | Total tokens | Elapsed | | — | — :| — :| | 8k | 8,041 | 0.19s | | 16k | 16,041 | 0.17s | | 24k | 24,041 | 0.22s | | 32k | 32,041 | 0.30s | | 40k | 40,041 | 0.41s | | 48k | 48,041 | 88.08s | | 56k | 56,041 | 23.34s | | 64k | 64,041 | 26.17s | | 96k | 96,041 | 36.70s | | 112k | 112,041 | 41.89s |

Butter-smooth to 40k, a brutal cliff at 48k when it blows past the KV-cache budget and starts paging, then a new, higher baseline once it settles. That single row — 48k → 88 seconds — taught me more about GPU memory, batching and serving limits than any blog post could. That’s the ups and downs of AI in one table.

What a year of this actually taught me
#

The hardware was the easy part. The lessons were the point:

Ceph taught me patience. OSD recoveries, noout flags, mon quorum, watching HEALTH_WARN crawl back to clean at 24 MiB/s. You do not rush storage.
Kubernetes taught me humility. Every clever shortcut had a receipt attached, usually payable at 2 a.m.
AI taught me both. The wonder of a model running entirely on hardware I own — and the day it wiped days of work because I gave it a bad instruction. The human stays in the loop. Guardrails aren’t optional. That was the tears.
And quietly, running all of it end to end taught me leadership: how to prioritize when everything is on fire, how to communicate a failure, and when to stop and think instead of typing.

Why I’m sharing this
#

I didn’t wait for a ticket to learn any of this. I bought the hardware, ran the cables, wrote the Python, broke it, fixed it, and shipped it — because that’s how I learn best: hands-on, end to end, until it actually works.

What’s sitting in my closet is a private cloud: Ceph-backed storage, GitOps, first-class observability, a home-grown control loop, and a private AI platform. It’s the same shape as the large-scale infrastructure I want to help build and keep running — just smaller, and paid for with my own hands.

One year, one rack, a lot of lessons — and a cooling system that didn’t exist until I made it.

Giving enterprise hardware a second life #

The platform #

The cooling system I couldn’t buy, so I built #

The AI chapter — the wonder and the cliff #

What a year of this actually taught me #

Why I’m sharing this #