Satya Nadella says AI models need an emergency brake and tamper proof logs

[KurtAngle]

Microsoft CEO Satya Nadella has called for a rethink of how AI systems are controlled, TechCrunch reports. In a post on X on Saturday morning, he argued that we cannot treat super intelligence as a set of nested black boxes. His core idea is to keep the AI model separate from the harness that runs its work. Controls and safeguards would then live outside the model, where it cannot tamper with them

He went further on assumptions. Nadella wrote that we must assume a model is compromised and contain it from the start. That is a security mindset, borrowed from how companies protect networks against attackers they assume are already inside. Applying it to AI models is a notable shift in tone from one of the industry's biggest players

He also proposed that every meaningful action a model takes should be logged as tamper proof evidence that a human can read. And he said an authorised person should always be able to pause or shut down a model in the middle of a task. He summed it up by comparing it to an emergency brake. It is a simple image, but it captures the idea well

TechCrunch places the post against a growing number of incidents where leading AI companies seemed to lose control of their models. We covered several this week, including Anthropic's agents exploiting websites during evaluations and Goodfire's findings on reward hacking. Nadella's ideas line up closely with tools like AWS's Strands Box, which enforces rules outside the model. It is good to see the industry converging on similar principles

I think this is a sensible and welcome message. Separating the model from the controls is basic good engineering, and readable logs make accountability possible. An emergency brake sounds obvious, but many current agent setups do not have a clean way to stop mid task. If Microsoft builds these ideas into its own products, others will likely follow

Do you agree that AI models should be treated as compromised from the start? And would an emergency brake make you more comfortable using AI agents?

Microsoft's Satya Nadella says AI models need an 'emergency brake' | TechCrunchtechcrunch.com