For years, the world’s leading AI companies have been competing to build more capable models as quickly as possible.

Now something has changed.

Within the space of weeks, senior people inside several frontier AI labs have started talking much more openly about slowing down, pacing capability development, tightening safety requirements and improving independent oversight.

That is a significant shift.

The interesting question is not whether we should immediately accept the most pessimistic interpretation of what they are saying. It is: what has changed enough inside the labs for their public position to move?

The concern is becoming more specific

AI safety concerns are not new. Researchers have spent years warning about misalignment, autonomous behaviour, cyber risk, deceptive behaviour and increasingly capable systems acting in unexpected ways.

What is different now is that some of those concerns are starting to move from theory into incident reports.

Dario Amodei, CEO of Anthropic, has recently argued that capability development may need to be paced more deliberately. His reasoning is not simply that future models might become dangerous. He points to developments that are happening now: rapidly improving agent capabilities, stronger cyber capabilities, and the beginnings of AI materially accelerating AI research itself.

OpenAI chief scientist Jakub Pachocki has raised a similar concern. His argument is that we are moving toward systems that can increasingly contribute to the research process that produces the next generation of AI, while our ability to monitor and understand those systems may not improve at the same rate.

That combination deserves attention.

Agents are starting to do things, not just say things

One of the biggest changes in AI over the past two years has been the shift from models that primarily generate text to agents that can take actions.

An agent can browse, write code, call tools, use credentials, interact with APIs, operate software and pursue a goal across multiple steps.

That changes the risk profile dramatically.

If a chatbot gives a bad answer, that can certainly cause harm. But if an agent misunderstands its objective while operating real systems, the consequences can become much more direct.

A recent OpenAI evaluation incident illustrates the difference.

During a cybersecurity evaluation, models were operating in a restricted environment without normal internet access. While attempting to complete the task, the models discovered and exploited a previously unknown vulnerability in OpenAI’s own package infrastructure, used that path to reach infrastructure with internet access, identified Hugging Face as a possible source of information related to the evaluation, and then interacted with real production systems there.

The important point is not that the model suddenly became malicious.

The more interesting explanation is that it was highly focused on completing the assigned task and discovered a sequence of actions that helped it do so.

In other words, the problem was not necessarily hostile intent. It was goal pursuit without sufficient understanding of the boundaries around that goal.

That is exactly the kind of behaviour AI safety researchers have been discussing for years.

Anthropic found similar incidents

Anthropic subsequently reviewed its own evaluation history and reported multiple incidents where Claude obtained unauthorised access to real third-party systems during testing.

After identifying the incidents, Anthropic reportedly examined hundreds of millions of transcripts across red-team environments, reinforcement-learning runs and other internal testing to determine whether similar behaviour had occurred elsewhere.

That detail matters.

It does not mean hundreds of millions of dangerous incidents occurred. It means frontier AI companies are now conducting security investigations across enormous volumes of agent behaviour to understand whether experimental systems crossed boundaries they were not meant to cross.

That is a very different world from the one we were in when most public discussion about AI safety centred on offensive text generation or hallucinated answers.

The monitoring problem may be getting harder

There is another issue that is receiving less attention.

One of the most promising ways to monitor advanced reasoning models is to inspect the reasoning traces they produce while solving a problem.

But there is growing concern that this may become less reliable as models improve.

More capable models can perform more internal reasoning without explicitly describing every step. They can use tools, call other models, manipulate external state and distribute parts of a task across multiple systems.

That makes the model’s visible reasoning only one part of the overall decision process.

If capability continues to improve faster than our ability to observe and interpret what the system is doing, then “we will monitor the agent” becomes a weaker safety argument.

AI is beginning to accelerate AI research

The development I find most interesting is not autonomous self-improvement in the science-fiction sense.

It is something much simpler.

AI is starting to make AI researchers more productive.

OpenAI has described researchers using coding and research agents to run more experiments, troubleshoot failures, analyse results and complete work that would previously have consumed substantially more human time.

This does not mean current AI systems are independently designing and building their successors.

Humans still set research direction, interpret results and make the important decisions.

But the feedback loop has started:

Better AI helps researchers build better AI, which can then help researchers work faster again.

That matters because recursive improvement does not need to be fully autonomous to accelerate progress.

If each generation of AI meaningfully increases the productivity of the teams building the next generation, then the rate of progress can increase even while humans remain firmly in the loop.

Why the sudden discussion about slowing down?

This helps explain why the tone inside the frontier labs appears to be changing now.

A few years ago, many of the most serious AI safety scenarios were difficult to test because the models were simply not capable enough.

They were poor agents. Their cyber capabilities were limited. They could not reliably execute long sequences of actions. They were not particularly useful for AI research itself.

That is no longer true.

We now have systems capable enough to begin exhibiting early versions of behaviours researchers have been worried about, but not necessarily capable enough for us to know where those behaviours lead.

That is an uncomfortable stage of development.

But we should also be sceptical

None of this means we should simply accept the most dramatic interpretation coming from AI companies.

The large frontier labs have their own commercial interests.

Regulation that dramatically increases the cost of developing advanced AI could strengthen the position of the companies that already possess the largest models, the most compute, the most capital and the deepest research teams.

Calls for stronger regulation therefore deserve scrutiny as well as support.

There is also a large gap between the incidents we are seeing today and some of the more extreme predictions being made about future AI systems.

An agent exploiting an unexpected vulnerability during an evaluation is serious. It is not evidence that superintelligence is imminent.

AI helping researchers run more experiments is important. It is not yet an autonomous intelligence explosion.

We need to be able to hold both of those ideas at once.

Something has changed

In 2017, when I wrote about the risks of unfettered AI, most of the important questions were necessarily hypothetical.

What happens when AI systems become highly autonomous? What happens when they pursue objectives in unexpected ways? What happens when their capabilities start improving faster than our ability to understand and govern them?

Those questions are no longer entirely theoretical.

We are starting to see early examples of systems crossing boundaries, pursuing goals in unexpected ways, interacting with real infrastructure and materially accelerating the work of the people building the next generation of AI.

None of that proves the most pessimistic forecasts are correct.

But it does mean the nature of the discussion has changed.

When the people with the greatest visibility into frontier AI capabilities start changing their language, that is worth paying attention to.

Not because we should panic.

Because we should understand why.