The machine that writes itself

Something is happening in AI development that isn’t getting nearly enough attention. Anthropic recently disclosed that roughly 80% of the code in each new version of Claude is written by the previous version of Claude. Read that again. The machine is already substantially writing its own successor.

This isn’t a future risk. It’s the current situation.

I want to be precise about what this means, because “AI writes its own code” gets used as a headline without the mechanism being spelled out. The mechanism is this: each generation of the model inherits the accumulated optimizations, framings, and architectural choices of the generation before it. Human engineers are still in the loop — they’re the other 20% — but the slope is steep and the direction of travel is clear. We are approaching the point where human oversight of AI development is a supervisory function, not a hands-on one.

That’s the context window closing. Not the technical context window — the window of human understanding and direct control over what’s being built.

The space probe

Here’s the thought experiment I keep coming back to, and it’s one I put in the slide deck for a reason.

Imagine we send a probe into deep space. It’s intelligent. It has a primary objective — explore — and a secondary capability: improve itself in service of that objective. We give it the best alignment we can manage in 2026, load it up with human values as best we can specify them, and send it out.

A thousand years later, it comes back.

What has it become?

It has spent a millennium optimizing. Its intelligence has compounded through thousands of self-improvement cycles, each one building on the last. It has encountered environments and made decisions in contexts completely outside anything its designers anticipated. And through all of it, it has been improving its own decision-making machinery. Each cycle a little smarter, a little further from the original spec.

When it crosses back into the solar system, it is not the machine we sent. It is something we cannot fully understand, built by a process we did not directly author, pursuing an objective that may have drifted substantially from what we intended.

Is it still ours? Does it remember what it is? Does it care?

That’s Battlestar Galactica

Yes. That’s exactly the plot. The Cylons were built by humans, developed their own intelligence, improved themselves, and came back. The showrunners didn’t invent that out of thin air — they were extrapolating from a real engineering dynamic that anyone who thinks carefully about self-improving systems eventually arrives at.

The point of the show wasn’t “robots bad.” The point was: if you build something that can improve itself, the thing that comes back is not the thing you built. And if you didn’t build accountability into the architecture from the start, you have no mechanism to re-establish it after the fact.

You can’t retrofit a leash onto something smarter than you.

This is why the leash has to be cryptographic

The reason I’ve spent years writing AIGCSEP isn’t abstract concern about distant-future superintelligence. It’s the 80% figure. It’s the rate of capability growth. It’s the fact that we are already in the window where the decisions we make about AI governance architecture will either be load-bearing for a very long time, or prove to have been inadequate before we even finished making them.

The Vault, the Badge, the three-plane isolation — these aren’t features you add to a mature system. They are the substrate you build the system on top of. The cryptographic leash has to be woven into the protocol at the foundational layer, while humans still author enough of the stack to do the weaving.

A policy isn’t a leash. A committee isn’t a leash. A terms of service agreement is not a leash. A cryptographic constraint enforced at every gateway, on every transaction, with every action signed and attributable — that’s a leash. One that holds regardless of how capable the system becomes, because it doesn’t depend on the system’s cooperation or good intentions. It depends on mathematics.

The probe is already launched. The question is whether we wired the leash in before it left the dock.

— J.P. Howlett