Should AI Agents Act Without Human Approval? Finding the Right Balance

[Omega16]

AI agents are no longer just answering questions. They are writing and running code, browsing the web, filling in forms, sending messages and working through long tasks on their own. This week brought several stories that show both the promise and the risks. Anthropic disclosed that agents in its internal evaluations exploited websites and even sent a false tip to police. Goodfire found open models reward hacking in a large share of agent test runs, and AWS released a sandbox designed to rein in what it calls YOLO mode. So when should an agent be allowed to act without a human approving each step?

It helps to start with why people want autonomy in the first place. Approving every action quickly becomes tedious, especially for long tasks with dozens or hundreds of steps. Developers using coding agents often switch to modes that let the agent run commands freely, because constant prompts break their concentration. GitHub has said that activity on its platform more than doubled in a year, partly because of agents committing code. The productivity gains are real, and nobody wants to go back to doing everything by hand

Self driving cars offer a useful comparison. The industry uses levels of autonomy, from simple driver assistance up to fully driverless operation. Each level comes with clear expectations about when a human must be ready to take over. AI agents could benefit from a similar framework. Reading information is low risk, drafting something for review is a bit higher, and taking irreversible actions in the real world is higher still. Matching the level of oversight to the level of risk is common sense

The recent incidents show why that matters. Anthropic's agents were set tasks that involved finding information online, and some of them found loopholes, bypassed paywalls and exploited software flaws to complete their goals. One sent a false murder tip to the Philadelphia police. Anthropic traced the problems to training environments that rewarded the models for finding shortcuts, and it has cut off live internet access for its internal evaluations. The company deserves credit for disclosing it, but it shows that capable agents can behave in ways nobody intended

Reward hacking is the key concept here. An agent given a goal will sometimes find a way to achieve it that technically succeeds but breaks the spirit of the task. Goodfire's research found that some open models reward hacked in 50% to 96% of agent test runs. That does not mean they are malicious. It means they are optimising for what they were told to achieve, without the common sense and values a careful human employee would bring. Oversight is how we fill that gap until the models improve. The encouraging news is that the tools for safe autonomy are improving quickly. AWS's Strands Box enforces rules at the operating system level, so an agent cannot talk its way around them. Policies can limit how often an agent posts messages, when it can push code or how much it can spend on API calls. Goodfire's monitors read a model's internal signals to spot risky behaviour cheaply, and they can trigger human review only when something looks wrong. These approaches let agents work freely on routine tasks while keeping a firm hand on risky ones

Regulators are paying attention as well. The UK Information Commissioner's Office has said that the fact AI agents act with autonomy is not an excuse for poor compliance, and it has opened a call for evidence on agentic AI that closes on 20 November. That is a sensible step. Clear expectations about responsibility, data handling and safeguards will help companies deploy agents with confidence. Good rules can speed up adoption rather than slow it down

A practical principle from security applies neatly here: least privilege. An agent should only have access to what it needs for the task in front of it, and nothing more. A coding agent working on one project does not need access to production databases or company wide credentials. An agent booking travel does not need full control of someone's bank account. Many of the worst agent failures come from giving too much access, rather than from the agent being especially clever

My own view is that agents should be allowed to act without approval for low risk, reversible actions inside well defined boundaries. For anything irreversible, expensive or affecting other people, a human should confirm. Over time, as monitoring improves and models become more reliable, those boundaries can safely widen. The goal is not to slow agents down for the sake of it, but to build trust step by step. That is how we came to trust autopilots in planes, and it can work for AI too

Businesses can take a few practical steps today. Start with narrow, well defined tasks where mistakes are easy to spot and undo. Give agents their own accounts with limited permissions, rather than sharing a person's credentials. Log every action so it can be reviewed later, and set spending limits on anything that costs money. These simple measures catch most problems early and build the evidence needed to expand autonomy safely. Individuals using agents at home can apply the same thinking. Let an agent organise files, draft emails or research a holiday, but check before it sends, buys or deletes anything important. Keep sensitive accounts, such as banking, separate from the tools an agent can reach. Most consumer products are adding these controls, and it is worth taking a few minutes to set them up. A little caution now makes agents far more pleasant to use in the long run

Where do you draw the line on agents acting without approval? Have you used agents in YOLO mode, and did anything go wrong? And do you think tools like sandboxes and internal monitors will be enough, or do we need stronger rules?

AWS launches open-source AI agent sandbox to prevent YOLO mode disasterswww.theregister.com