Let me tell you about an experiment I ran, and then walked away from. Not because it failed. Because it taught me something I was not looking for.
What was this experiment about
AI coding agents like Claude and Codex are great at writing code. But the moment you point them at real infrastructure, be it a Kubernetes cluster, a production database, or a live AWS account, things get scary. You do not want the agent’s good behaviour to be the only thing stopping it from deleting something it should not touch.
So I set out to build a harness for this. The idea: give an agent its own identity, not yours, scoped down to exactly what it needs, enforced by the system itself and not by a polite instruction in a prompt. And make the skills that agent uses portable. Write a skill once, run it on Claude Code, Codex, or whatever agent runtime comes next, the same way an Ansible role runs on any server.
I called it AOH, an Agentic Ops Harness. And I set myself one more rule at the start, out loud: it had to be simple enough to run from inside whatever coding agent I was already talking to. No separate tool to learn and no additional CLI options to remember.
What I built
More than I expected, and a lot of it actually worked.
Three different agent runtimes could all run the same skill pack. I proved the safety story on a real cluster, not a mock one. The agent tried to delete a pod, and Kubernetes itself said no. Not because the agent chose to be careful. Because its permissions simply did not allow it. That single moment told me the core idea was sound.
I also built a strict review process around every change. Before anything shipped, a second AI would review the design, specifically trying to find what I had missed. And it did find real things. Once it caught a permission rule that accidentally gave read access to every secret in the cluster. Another time it caught a credential design that would have quietly leaked tokens. That process earned its keep every single time.
By the end there was even a full write-a-skill-and-publish loop. You draft a skill in whatever session you are already in, run one command, and it lands as a real commit or a real pull request on GitHub. I watched this happen against a real repository, with three hundred and sixty seven tests behind it. So this was not a toy. It was real, working software.
Why I am still abandoning it
Here is the part that stings a bit. One day I asked myself the simplest question a builder can ask: can I actually use this?
And the honest answer was six different commands, spread across two modes of the same tool, with flags that meant different things depending on which mode you were in. I had built a harness, and I could not operate my own harness without a cheat sheet next to me.
That is not a missing feature. That is a sign the design was wrong from the start. I had said clearly, on day one, that I wanted zero commands to remember. I actually built that, exactly once, for one single feature, the publish step. For everything else, install, fleet management, version locking, I kept adding a new command instead of wrapping it the same simple way. Each of these decisions was fine on its own. Add them all up, and you get a tool that only its own builder could drive.
So I am abandoning this build. Not the idea behind it, the idea is still good, and I proved enough of it live to trust it. What I am abandoning is this particular attempt. I do not think you fix a tool that has drifted this far from its own starting point by patching it here and there. You go back to that starting point and build toward it directly, this time holding the one rule that mattered most as non-negotiable from day one, not bolted on later for a single feature.
The lessons
Working software and the right software are two different tests.Every phase of this project passed its own review. Tests were green. Features worked. And the whole thing still drifted away from what I said I wanted at the start. Passing your own review is not the same as staying on course.
Ask “can I use this without notes” after every single milestone, not at the end.I should have run that test every week. I ran it once, forty days in, and by then the answer was already no.
Building with AI is fast, and fast cuts both ways.You can go from idea to tested, working, reviewed code in hours. But it is just as easy to build your way away from your original intent as it is to build toward it, because every small step still feels justified, still passes review, still works. Nothing stops you automatically and asks if you are still headed where you meant to go.
Complexity built with AI is easy to add, but surprisingly hard to unwind. Once a system gets complicated enough, with enough small decisions that each made sense in the moment, it becomes genuinely hard to undo. Not impossible, just hard in a way that quietly pushes you toward one more patch instead of a rethink. And when an AI can produce that next patch in minutes, the cost of “just one more fix” drops so low that you stop noticing how many fixes you have already made.
Adversarial review is worth every bit of the extra time. The bugs that mattered most were never caught by a quick look. They showed up only when someone, or something, traced the whole path across files with the specific job of trying to break it.
What a DevOps engineer, SRE, or platform practitioner should take from this
If you are exploring AI agents for operations work, three things worth carrying forward.
One, put the safety boundary in the real system, not in the prompt.
RBAC, scoped credentials, whatever your platform gives you. Never trust an agent’s good behaviour as your only line of defence.
Two, if you are building any kind of tool or workflow around agents, decide your usability bar before you write the first line, and check against it constantly, not once at the end. It is far easier to hold a line from the start than to pull a system back to it later.
And three, do not be afraid to restart. An abandoned build is not proof the idea was wrong. Sometimes it is proof you learned enough to try it again, better. That is exactly what I am doing next, same idea, different shape, and I will tell you how it goes.
The code for this attempt is public, MIT licensed, at agenticdevops/aoh
with docs at agenticdevops.github.io/aoh. If any part of it is useful to you, take it further than I did.