Building a Stateful Human-in-the-Loop Slack Approval Gateway for Homelab AI Agents

I have a growing pile of AI agents that want to do things to the cluster — apply a manifest, run a migration, delete a PVC. I am not yet brave enough to let any of them act unsupervised. So I built a gate: a small Go service that takes a structured “I want to do X” request from an agent, posts it to a Slack channel as a rich approval card, and waits for a human to type approve (or push back) in the thread before it tells the agent to proceed.

This is the story of slack-approval-app v0.1.0, why it is async rather than the blocking thing I first wrote, and the handful of sharp edges I hit getting it live on the OpenShift homelab cluster.

What it does

#

The contract is deliberately small. An agent does:

POST /v1/approval
Authorization: Bearer <token>

{
  "agent_name": "HomeLabOps_TestAgent",
  "channel_id": "C0BANS7QR1A",
  "callback_url": "http://.../decision",
  "task": {
    "what":   "Deploy slack-approval-app v0.1.0 to the cluster",
    "why":    "Validates the new async callback architecture end-to-end",
    "weight": "New Deployment added, no existing resources affected",
    "snippet": "apiVersion: apps/v1\nkind: Deployment\n...",
    "snippet_filename": "deployment.yaml"
  }
}

The service replies immediately with 202 Accepted and a request_id. A background goroutine then posts a Block Kit card to Slack, attaches the snippet as a file in the thread, and starts polling for a reply. When the operator answers, the service POSTs the decision (approved, redirected, or expired) to the agent’s callback_url. Done.

The architecture evolution: blocking → async

#

The first cut (v0.0.x) was the obvious thing: the HTTP handler posted to Slack, then blocked inside the request polling the thread until a human responded, and only then wrote the HTTP response. It worked beautifully on my laptop with curl. It is also completely wrong for anything living behind an ingress controller.

A human approval can take minutes, hours, or “after my shower.” An HTTP request that stays open that long is going to die, and not gracefully:

• Ingress/router timeouts. OpenShift’s HAProxy router has a default server timeout measured in seconds (30s-ish). The connection gets reset long before a human looks at Slack. The agent sees a 504, not a decision.
• Connection accounting. Every pending approval is a held TCP connection and a parked goroutine on a fixed WriteTimeout. A dozen agents waiting on humans is a dozen sockets the router and the pod are both babysitting for no reason.
• No durability story. If the request dies mid-wait, the decision is lost even if the human did eventually click approve.

v0.1.0 inverts it. The HTTP call is now a fire-and-forget registration: return 202 in milliseconds, do the slow part out-of-band, and call the agent back when there’s news. The HTTP server’s WriteTimeout can stay short (20s) because no handler ever waits on a human. This is the same shape as a webhook/callback API, and it’s the only shape that survives an ingress.

Why three fields: What / Why / Weight

#

The task payload forces three required fields, and I think this is the most important design decision in the whole thing. A human reviewer skimming Slack on their phone needs exactly three things to make a fast, correct call:

• What is about to happen (the action).
• Why (the justification — is this a sane response to a real problem?).
• Weight / Impact (blast radius — “no impact” vs “pvc-foo would be deleted”).

The optional snippet is the receipts: the actual manifest or command, uploaded as a file in the thread so it doesn’t bloat the card but is one tap away. An agent that can’t articulate what/why/weight has no business getting an approval, so the service rejects the request with a 400 if any are missing. Structured context turns “approve this?” into a decision instead of a guess.

Thread-scoped polling done right

#

The polling loop reads thread replies with the slack-go GetConversationReplies call, and the detail that matters is the Oldest cursor:

replies, _, _, err := e.slack.GetConversationReplies(&slack.GetConversationRepliesParameters{
    ChannelID: entry.channelID,
    Timestamp: entry.threadTS, // the parent message = the thread
    Oldest:    lastSeen,       // only messages after the last one we processed
    Limit:     20,
})

Two things fall out of this:

• Thread-scoped. Polling is anchored on the parent message timestamp, so the loop only ever sees replies to this specific approval. Ten concurrent approvals in the same channel don’t cross-talk.
• No reprocessing / no duplicate callbacks. lastSeen advances past every message we’ve already handled, and we skip the parent and any BotID messages (the card itself, our own acks). Without the Oldest cursor you re-read the whole thread every tick and risk firing the callback twice. With it, each human reply is processed exactly once, and the worker returns the instant it acts.

approve (case-insensitive) means proceed. Anything else is treated as a redirect — the operator’s text becomes feedback in the callback so the agent can adjust and try again. That redirect path is what makes this a loop and not just a gate.

Callback delivery: retries with a fresh context

#

Telling the agent the answer is the one part that must not be lost, so fireCallback does up to three attempts with linear backoff (2s, 4s) and treats any 5xx as retryable. The subtle bit:

deliverCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)

It uses a fresh context.Background(), deliberately not the request or shutdown context. Why: when the pod gets SIGTERM, I cancel the root context to wake every pending worker so it can fire an expired callback — but if the delivery HTTP call were also bound to that same cancelled context, the callback would be cancelled before it could be sent. The whole point of the graceful shutdown is to notify agents that their pending approvals are dying; the delivery has to outlive the cancellation that triggered it.

Production hardening

#

A few things that separate “works on my laptop” from “I’ll let it run”:

• Timing-safe auth. The bearer token check uses crypto/subtle.ConstantTimeCompare, so it doesn’t leak the token byte-by-byte through response timing. It also refuses to start if the token is still the REPLACE_ME placeholder.
• Non-root. The container runs as USER 1001, which keeps OpenShift’s restricted-v2 SCC happy without any anyuid escape hatch.
• Graceful shutdown that fires expired callbacks. SIGTERM cancels the root context; every pending worker posts “⏰ Request expired” to its thread and fires the callback with status: expired. No agent is ever left waiting on a pod that no longer exists.
• /healthz + /readyz. Liveness is a dumb 200. Readiness returns {"status":"ready","pending_approvals":N} — which doubles as a free gauge of how many humans are currently the bottleneck.
• JSON structured logs. Every line is slog JSON with request_id, agent, and channel_id, so it drops straight into Loki/Vector with no parsing.

Getting it onto OpenShift without a registry

#

I didn’t want to stand up a CI pipeline just to ship one binary, and my workstation is arm64 while the cluster is amd64. The trick I used (and now reuse) is a build init-container: the Go source lives in a ConfigMap, an init container running golang:1.22 compiles it GOOS=linux GOARCH=amd64 into an emptyDir, and the runtime container (almalinux:10-minimal) just executes the resulting binary. No registry, no cross-build dance, the manifests are the artifact. Slower cold start (~a minute to compile), but for a tiny service it’s a great trade.

Config is split across two Secrets, both git-ignored per repo convention: slack-approval-app-slack-token carries the xoxb- OAuth token as SLACK_BOT_TOKEN, and slack-approval-app-config is the entire config.yaml (including the bearer token) mounted over /app/config.yaml. The image ships a bearer_token: "REPLACE_ME" placeholder; mounting the Secret as the config file overrides it cleanly, and the service hard-refuses to boot if it ever sees the placeholder. Two OpenShift Routes (edge TLS) expose it at ai-approvals.apps.example.com.

Two gotchas I walked into

#

file.upload.v2: file size cannot be 0. The very first live test posted a perfect card but no snippet file. slack-go’s UploadFileV2 requires you to set FileSize explicitly even when you pass Content as a string — leave it at zero and the API rejects the upload. One line fixed it:

FileSize: len(req.Task.Snippet),

Rollout endpoint overlap. Updating the source ConfigMap and doing a rollout restart had a brief window where the Service still routed to the terminating pod. A test request landed on the dying pod, which then SIGTERM’d and (correctly!) fired an expired callback for it. Annoying during testing, but it was actually the graceful-shutdown path proving it works. Once the new pod was the sole endpoint, everything was clean.

The live test

#

To validate the whole round-trip I sent a real approval request through the external route while writing this:

curl -sk -X POST https://ai-approvals.apps.example.com/v1/approval \
  -H "Authorization: Bearer <token>" \
  -d '{ "agent_name":"HomeLabOps_TestAgent", "channel_id":"C0BANS7QR1A",
        "callback_url":"http://slack-approval-app.ai-approvals.svc.cluster.local:8080/healthz",
        "task":{ "what":"Deploy slack-approval-app v0.1.0 ...",
                 "why":"Validates the new async callback architecture ...",
                 "weight":"New Deployment added, no existing resources affected",
                 "snippet":"apiVersion: apps/v1\nkind: Deployment\n...",
                 "snippet_filename":"deployment.yaml" } }'
# {"request_id":"db7d8fbaf88cad805b83e91e","status":"pending"}  HTTP 202

Note the callback_url: it points at the in-cluster Service DNS, not the public route. The router terminates TLS with OpenShift’s own ingress cert, which isn’t a public CA — so a self-callback over https://...apps... would fail x509 verification. Staying on svc.cluster.local over plain HTTP both guarantees the 200 and follows the house rule that internal traffic never leaves the cluster network.

202 came back in milliseconds, the Block Kit card and the deployment.yaml attachment rendered in Slack, and /readyz flipped to pending_approvals: 1. Then I went to wait for a human to type approve. Which is, after all, the entire point.