Taking myself out of the loop
Building a reviewer, generator, evaluator harness inside an Azure perimeter, and what broke on the way
I was spending two and a half hours a day pasting production data into Claude and reviewing what came back. It was working. It also meant the bottleneck had become me.
For most of this year I've been building AI harnesses into my day to day engineering work: automating tasks, prototyping, and delivering enterprise applications using context driven development with Claude in the loop.
I was still far faster than working unaided, but I was sitting inside the workflow rather than at the end of it. What I actually wanted was to be the reviewer, not the runner.
Here is what that looked like in practice.
THE PROBLEM
Monitoring a production system. Several times a day I would pull the state off the UI, take the transaction data out of the application, and paste it into Claude. We would look for patterns and anomalies. If something wasn't visible in that data, I would go into Application Insights, pull the logs and traces for the period, and feed those in too. Between us we would identify issues, prioritise them, and fix the high value ones before the next run.
That was dramatically faster than doing it without an assistant. It also cost me two and a half hours a day. With the next phase of delivery about to land, that was not sustainable.
The vision was simple: the harness finds the problem, generates the fix, raises a draft PR, and sends me a Teams message. I either decline it or test it and apply it.
THE APPROACH
My first instinct was a single agent on a daily job. Everything I'd learned over the previous months told me a one shot agent wouldn't produce anything worth my review time.
So I went with a reviewer, generator, evaluator pattern instead.
The reviewer goes first. It reads the collected context, works out what's actually wrong, and prioritises. The generator then takes that diagnosis and writes the fix. The evaluator makes one decision at the end: is this worth my review time. If it isn't, the run ends quietly and nothing reaches me.
The part that made the fixes usable rather than merely plausible was injecting the application's own CLAUDE.md and harness from its repo into the reviewer, generator and evaluator. They aren't working from general knowledge of the language, they're working to the codebase's own conventions and patterns. A fix that ignores the house style is a fix I have to rewrite, which defeats the point.
The evaluator matters for a different reason. A harness that raises a PR every day trains you to ignore it.
The decision that mattered more than I expected was packaging all the collected context into a structured JSONL artefact before any agent touches it. The agents reason over the artefact, not over the application. Which means the harness itself is agnostic to the tech stack underneath, and can be used by other teams and systems.
THE SETUP
The application this framework is monitoring is .NET, hosted behind a private network with no public endpoints on Azure.
My first build ran from GitHub Actions: headless browser against the front end, calls to the domain APIs, App Insights for telemetry. It sort of worked, but on GitHub Enterprise it couldn't reach the internal GraphQL endpoints.
Moving the whole thing into Azure was the right call. A container job, scheduled daily, sitting in the same resource group as the application. From there it can see App Insights, reach the internal APIs and GraphQL endpoints, and drive Playwright against the UI.
The job spins up, does the work, logs to console, and shuts down. The collected data goes with it, and that's fine. If there's nothing worth fixing, nothing happens. If there is, the draft PR carries everything a developer needs to understand the problem and the fix.
The job also pulls the last 24 hours of commits from the repo. In my experience most production problems trace back to the most recent change.
I also log the full agent conversation into the PR. In a regulated environment, being able to show how a proposed code change was reasoned into existence is worth as much as the change itself.
WHAT I'D TAKE FROM IT
Two and a half hours a day has become one or two PR suggestions every other day, and that number is falling as the application gets more stable. Which is the outcome I wanted. The harness working itself quietly out of a job is the point, not a sign it's failing.
Know your boundaries before you design the pipeline. My GitHub Actions attempt didn't fail on the AI, it failed on network topology. Moving inside the perimeter made everything else straightforward.
The same setup runs just as well against dev and UAT, and that's where I think the real saving sits. Catching these issues before they reach production is cheaper than catching them after, and the harness doesn't care which environment it's pointed at. It can also flag code smells in the areas it's already touching.
The first time a draft PR landed with a Teams notification next to it was very exciting indeed. Now there are real genuine code fixes being made, without me being in the room!
Interested to hear from anyone running something similar. What did you find hardest to move out of the loop?