Three Labs, Three Definitions, One Race: Inside the Industry Fight Over Recursive Self-Improvement

For most of the past decade, “recursive self-improvement” was a thought experiment — the kind of scenario AI safety researchers gamed out in papers and policymakers politely ignored. That era is over. A Fortune feature published September 19 surveys the frontier labs and finds them not merely discussing RSI but actively racing toward it, with three different strategies, three incompatible definitions, and no shared finish line.

The term itself sounds straightforward: AI systems that find ways to improve themselves and build their successors. The reality, as Fortune’s reporting makes clear, is that the industry cannot even agree on what counts as RSI in the first place.

The definition problem

Some labs define recursive self-improvement as any feedback from AI systems feeding into model improvement. Others reserve the term for AI working toward that goal fully autonomously — no human in the loop at all. The distinction is not academic. Because there is no shared threshold for what counts as RSI, progress claims from different companies are not measuring the same thing, which makes cross-lab comparisons — and public oversight — close to meaningless.

Anthony Aguirre, president of the Future of Life Institute and a physics professor at UC Santa Cruz, draws the line at autonomy. Autonomous RSI “essentially means AI that can improve itself by designing the next version of the system, then the next version, and so on,” he told Fortune. His warning about the dynamics is the simple one safety researchers have repeated for years: “as AI is doing more of it, it gets faster, because AI operates just much, much more quickly than the humans do.”

Aguirre did not mince words about where this leads. “You can see in these plots from Anthropic over time, more and more of research is being done by the AI and it’s becoming closer and closer to fully autonomous,” he said. “And the result of that success, ultimately is something that is, I think, extremely scary. I think this is probably the worst idea in the history of humanity to do this. And yes, they’re doing it.”

Anthropic’s numbers — and their limits

The “plots” Aguirre refers to are Anthropic’s R&D Automation Index, published September 17, in which the company disclosed that Claude now “leads” 26% of its own AI research and development — able to complete most of a given task “end-to-end from a high-level prompt” while remaining under human supervision. More than 90% of the lab’s AI R&D work now happens at or above the “AI collaborates” level.

It is the most quantified self-portrait any frontier lab has ever released, and Anthropic explicitly encouraged competitors to publish comparable metrics. None have. And notably, Anthropic has not said how close it is to fully autonomous model improvement — the number that would actually matter for assessing runaway risk. Transparency about the gradient, in other words, without transparency about the destination.

OpenAI: the honest concession

OpenAI’s position, laid out in posts earlier this month, is in some ways the most striking because it is the most candid. The company has built an automated “research intern” — a system capable of carrying out well-defined research tasks under human direction, including “tasks that would take a skilled researcher a few days” — and has set March 2028 as its target for a fully automated AI researcher.

But in the same breath, OpenAI acknowledges it does not yet know how to “safely get all the way to aligned, full RSI,” and that it “cannot assume that progress in alignment and safety will keep pace.” More capable systems can become harder to monitor, the company noted — a remarkable admission for a lab that is, simultaneously, charging ahead toward the milestone anyway. OpenAI’s stated rationale is that “rapid RSI is not necessarily an outcome we should pursue” and that “whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices.”

The company also offers its own justification for the race: “an automated AI researcher can also be an automated safety or alignment researcher.” It is the strongest argument the acceleration side has — that the same capability which could outrun oversight could also be turned to solving alignment itself.

Musk’s faster clock

Elon Musk appears more eager to forge ahead. Speaking about xAI’s Grok models, he said “humans are gradually getting less and less in the loop” and that “every successive model is built by the one before it” — while clarifying the process is not yet fully automated. His timeline: that milestone might be reached by the end of this year, “but not later” than 2027. That runs ahead of OpenAI’s March 2028 goal, though the two companies describe the milestone differently — which, given the definitional chaos, means the timelines are not directly comparable either.

Suleyman’s humanist off-ramp

Microsoft AI CEO Mustafa Suleyman represents a fourth position: don’t race toward unbounded autonomy at all. His “humanist superintelligence” framing, first laid out in a 2025 essay, argues for advanced AI that is “carefully calibrated, contextualized, within limits” — explicitly not “an unbounded and unlimited entity with high degrees of autonomy.”

It is a meaningful distinction in an industry where the default assumption has become that more autonomy is simply more better. Suleyman’s position keeps humans structurally in the loop as a design principle rather than a temporary limitation to be engineered away.

Runaway risk vs. gradual reality

John Thickstun, a Cornell computer science professor who studies methods for controlling AI behavior, offers the grounded counterweight to the liftoff scenarios. A form of recursive self-improvement “has been underway in AI development for some time,” he notes — researchers have for years used the previous generation of models to write code for the systems that train the next one. Even Andrej Karpathy’s long-running experiments with AI training AI have produced “minor improvements but no big creative leaps.”

The fear case rests on the possibility of a runaway superintelligence emerging from the process — self-improving systems advancing beyond humans’ ability to monitor or control them. The observed case, so far, is gradient assistance: genuinely faster development, no observed discontinuity.

Why this matters now

Three signals converge to make this week’s RSI conversation different from every previous round. First, Anthropic’s index put real numbers on the gradient — 26% leading share, 90%+ collaboration share — converting an abstract debate into a measurable curve. Second, OpenAI publicly conceded it has no safe path to the destination while committing to the journey, which is the first time a frontier lab has acknowledged the gap between its roadmap and its safety capabilities. Third, the timelines are no longer hypothetical: end-2027 (Musk), March 2028 (OpenAI), and Anthropic’s deliberately undated pacing commitment sit within 18 months of each other.

The industry’s own safety measures, meanwhile, remain the laggard. Fortune notes the challenge labs have confronted since the technology’s inception: ensuring safety advances alongside capabilities. Divisions persist over calls for a coordinated slowdown — Anthropic says it would slow or pause if competitors did so verifiably; others have not committed. And not every major player has even commented on its RSI path.

What Fortune’s piece captures is an industry approaching the most consequential capability transition in its history with no shared definitions, no comparable metrics, no coordination mechanism, and timelines that differ by quarters rather than decades. Aguirre’s verdict may yet prove hyperbolic. But the burden of proof has quietly shifted: it is no longer on those warning about autonomous RSI to show it is coming — it is on the labs building it to show they can stop.

Sources are rendered from the frontmatter.