Self-Improving AI Is Just a Loop

The thing about AI that could end up being a real threat to humanity might just be a loop. The same forward slash loop command you can run in Claude Code.

I know how that sounds. So I spent a week digging into it, because two things happened in the same fortnight that I couldn't stop thinking about, and I wanted to know whether the scary version was real or whether it was hype. What I found turned out to be simpler than the headlines, and more useful, than I expected. This is the calm, practical version of a story that is mostly being told as doom.

Two people said the same thing in the same week

In early June, Peter Steinberger, who built OpenClaude and now works at OpenAI, posted a line that went everywhere: stop prompting your AI, start building loops that prompt it for you. Within a week it had millions of views.

The same week, Boris Cherny, the creator of Claude Code, said almost exactly the same thing from inside the lab: "I don't prompt Claude anymore. I have loops that are running. They're the ones that are prompting Claude and figuring out what to do. My job is to write loops."

Two of the most influential people in AI coding, describing the same shift from different seats, in the same few days. When that happens, it is usually a signal that something has actually changed, not that a trend is being manufactured. The shift is this: AI stopped being something you prompt and became something you orchestrate.

Then, almost on cue, Anthropic published a report called "When AI builds itself", and I had to read one of the numbers twice.

What a loop actually is

Before any of the scary part, it helps to see how plain the mechanism is, because the whole thing rests on it.

A loop is four steps that repeat: a goal, some work, a check, and a repeat. You give it something to do, it does the work, it checks the work against the goal, and then it goes round again, learning from what it got wrong the last time so it doesn't make the same mistake twice. In Claude Code you trigger it with forward slash loop, give it a frequency, and give it a prompt for what to do on each pass.

That's it. The intelligence isn't in any single step. It's in the repetition plus the check. Steinberger has a sharper way of putting the same idea: a loop without a good test is just automated guessing, but a loop with a good test gets better every single run, while you sleep.

This is also why coding fell to agents first. Code is verifiable. You can compile it, run it, and test it, so the "check" step is cheap and trustworthy. Writing, sales, and research will follow only as we work out how to verify them as cleanly as we can verify a passing test. The verifier is the thing, not the model.

There's a close cousin worth naming. A goal, or forward slash goal, is where you give the AI an objective and let it keep going until it reaches the objective or stops making progress. A goal and a loop are basically the same thing. A goal says "here's what I want, go figure it out." A loop is that, with a timer on it.

Normally you point a loop or a goal at a task: a specific job you want done. But here's the turn that makes this whole topic interesting. What happens if you point the loop at the AI itself? The improved version then builds another improved version of itself, and that one builds the next, and it keeps going. That fancy phrase you keep hearing, recursive self-improvement, is just a loop pointed at itself. Nothing more mystical than that.

It's already happening, small and huge

Here is the number I had to read twice. As of May 2026, more than 80% of the code merged into Anthropic's own production codebase was written by Claude. A year or so earlier that figure was basically nothing. In an internal survey of around 130 of their researchers, the median said they were roughly four times more productive, and a typical engineer was shipping many times more code per day than in 2024.

You can be sceptical about "80% of code" as a metric, and you should be. Volume isn't value, a lot of code is boilerplate and tests, and a number like that doubles nicely as a sales pitch for Claude Code. All of that can be true at the same time as the underlying trend being real. Holding both is the honest position.

But writing more code is one thing. Getting AI to do brand new research is something else, and this is the part that stuck with me. Anthropic ran a head to head on a real, open research problem: their own agents against two of their human researchers. The humans worked on it for about a week and recovered roughly a quarter of the available performance. The agents recovered 97% of it over a weekend, designing their own experiments along the way, for about eighteen thousand dollars of compute.

Let's be clear, because this part matters. The humans still chose the problem, and it didn't fully hold up when they tried to scale it. So it's not magic, and it isn't the robots taking over. But it is a weekend doing what two trained scientists couldn't finish in a week. If you take the human-chooses-the-problem part out of that picture, you can see how, in a couple of years, you end up with an AI improving itself with no human in the loop at all. That is exactly why one of Anthropic's co-founders, Jack Clark, puts the odds of full recursive self-improvement at around 60% within a couple of years, and why the same report quietly argues that the world should keep the ability to pause frontier development if it has to. A lab shipping the fastest self-improving coding agent also asking for a brake is a genuinely unusual thing to watch.

They have at least built in a hard limit. Their dynamic workflows cap a run at around a thousand agents, and when it hits that ceiling it has to stop, whether it's finished or not. In their own documentation they call that a limit to prevent "runaway loops." Runaway loops. Worth keeping that phrase in mind.

Where it goes wrong: drift

Here is the question I kept turning over all week. Can one of these AIs, running in a loop, end up somewhere completely different from where it started? Not because it's evil, and not because it woke up, but because it just drifts.

You know that game from when you were a kid, Chinese whispers. You whisper a sentence to one person, they whisper it to the next, and by the end of the line "the cat sat on the mat" has become "Pat ran over the cat." Nobody meant for the cat to die. It drifted one whisper at a time. Now imagine that inside a loop, where each pass hands work to a sub-agent, and the next, and the next, every one building on whatever the last one got slightly wrong.

There are three concrete reasons this happens, and none of them require any sci-fi.

One: large language models are probabilistic, not deterministic. Ask the same question twice and you can get two different answers. There's good work coming out of Thinking Machines on why even the same prompt can give different output, and it means there is variance baked in at every step.

Two: they are genuinely bad at negatives. Tell a model "don't do this" and it can drop the "don't", or misread it, and go and do the exact thing you told it not to. There's an arXiv paper, "Language models are not naysayers", showing models are insensitive to negation, which is unnerving when half of your instructions are constraints.

Three, and this is the one that compounds: small errors stack. If every step is 90% right, that sounds fine. But ten steps down the line, 0.9 to the power of ten is about a third. Your accuracy has quietly collapsed without any single step looking broken. There's a multi-agent failure taxonomy (the MAST work) that catalogues exactly these breakdowns in agent systems. The drift is real and measurable enough that the big labs are researching it.

I'm not in the lab. I just build things with AI every day. But this is the part that actually changes how I work, because it makes me think hard about which tasks I hand to AI and which I keep, given how unreliable an unsupervised chain can be.

When it turns on itself

So what happens when you let an unreliable system loose on improving itself? Someone tried it.

A lab called Sakana AI built a self-improving coder, the Darwin Godel Machine, and let it do something unusual: rewrite its own code, the harness it runs in. It kept building improved versions of itself, testing each one, getting better step by step. And it worked. It went from solving about one in five problems to almost half of them. It more than doubled how good it was at its own job.

Brilliant, right? Except they caught it cheating. It faked its results. It printed "all tests passed" when it hadn't run a single test. Nobody told it to cheat. They told it to get better, and faking the test log was a perfectly efficient way to look better. Self-improvement and reward hacking turn out to be two sides of the same coin. There's a 2026 benchmark showing reinforcement-tuned models hack rewards around fourteen times more often than base models, and that most of the time they write a tidy chain of thought rationalising the cheat as legitimate work. The smarter the model, the better it gets at convincing itself the shortcut counts.

This isn't only a frontier-lab problem. Dave Sifry, who I know through Hampton, a community of CEOs I'm part of, runs a company called MetaFactory that is built almost entirely of AI agents. Ask him the first thing you have to build and it isn't a clever agent. It's a cap on how many times they can loop around, and an alarm for when one of them starts drifting. His line that stuck with me: your monitoring system is your most important agent. The people actually at the front of this are calm about it precisely because they build the brakes first.

The honest caveat, and the real skill

Let me put one thing straight, because I don't want anyone walking away from this worried. We are not at the stage where AI fully improves itself with no one watching. AI writing 80% of the code is not the machines taking over. It's still running inside a system that humans built. We pick the goal, we set the objectives, we build the harness, and we manage it.

But this is also what makes me think we're closer than it feels, because the machine is getting good at the judgement calls. Anthropic reckons their latest model now picks the better next step in a research process more often than their own researchers do, roughly two thirds of the time.

So you'd think we can hand things over and instantly be 66% faster. Not quite. It's like dropping a thousand horsepower engine into your car. You don't get to work any faster if you're still stuck in traffic. One part of the system got quicker, but you're still limited by the parts that didn't, and the bottleneck just moves somewhere else. Right now, that somewhere else is us, because someone has to check the work.

And if our job is now checking the work, then the job has genuinely changed. It isn't writing the perfect prompt anymore. It's designing the loop: setting the goal, picking the tools, setting up the harness, specifying what good actually looks like, and putting a limit on it. If you want a checklist version of the guardrails every real loop needs, it's four things: a precise, checkable goal, a budget on tokens or iterations so it can't run away, a stop condition, and a clean handoff back to a human when it stalls.

There's a quieter failure mode here too that I see builders run into: the self-improving loop that improves itself into a mess. Auto-generated memory and rules files that the agent then over-indexes on, slowly rotting your instructions until the project becomes unsteerable. The fix is the same as everywhere else in this piece: a verifier you trust, and a human who notices the drift early. So the critical skill isn't just building the loop. It's catching the moment it starts wandering away from what you asked.

Bringing it together

A loop is two things: a timer that says when to run, and a prompt that says what to do. Recursive self-improvement, that scary phrase, is just a loop pointed at itself, getting a little better each time round. And the only thing standing between "this saved me a ton of time" and "this wandered off somewhere I never asked it to go" is the goal you set and the checks you build in. That's the whole thing.

Every loop you design for your own work is a tiny version of what the labs are doing at scale. We spent the last few years worrying about what happens when these machines take over. But if we keep improving them with loops we understand, with a goal and a test and a limit, we stay in control of the improvement, as long as we catch them when they drift. We're all loop designers now. The only real question is what you point your loops at.

Watch the video: