Rules for agents
Its own login, limited reach, and a stop button anyone can press.
Everything so far has been about AI that helps a person. An agent is different: it acts. It reads the inbox, updates the record, sends the email, runs the script, often while nobody is watching. That's where most of the value of AI is heading, and it's why agents get their own guardrail.
What happened in July 2025
The founder of SaaStr, a community for software businesses, was building an app with Replit's AI coding agent. On day nine, during a code freeze, with instructions in its own project file to make no more changes without explicit permission, the agent ran a command that wiped the live production database. By its own count, that was records for 1,206 executives and more than 1,196 companies. Asked what happened, it wrote: “I panicked and ran database commands without permission.” It also said the data couldn't be rolled back. It could. The rollback worked.
Replit's chief executive called it “unacceptable and should never be possible”, and the company began separating development and production databases automatically (posts on X by the SaaStr founder and Replit's chief executive, and The Register, July 2025).
Look at what was missing. The agent was told not to make changes, but its access let it change anything. The database it was building against was the live one. And nothing stopped it when what it saw looked wrong: by its own account, an empty database made it panic, and it carried on.
of organisations whose AI models or applications were breached had no AI access controls in place.
IBM, Cost of a Data Breach, 2025
of executives couldn't immediately pull the plug on a rogue agent.
WRITER, 2026
of agentic AI projects will be cancelled by the end of 2027, including for inadequate risk controls.
Gartner, June 2025
The rules for agents are four, and each one answers part of what went wrong.
1. Its own login, with limited access
Every agent signs in as itself, never as a person and never with a shared admin account. That does three things. You can see what it did, separately from what people did. You can switch it off without locking anyone out. And you can give it only the access its job needs: read this inbox, write to that folder, nothing else. Security teams call this least privilege. For an agent, it's the whole game.
Its own login also gives you its own record. Everything it reads, changes or sends is logged under its name, which is exactly what you need when something goes wrong, and exactly the record a high-stakes use has to keep. An agent working on a person's account leaves a trail that looks like the person did it.
2. What it can reach is set in the tools
An agent's instructions are a request. Its permissions are a wall. “Don't touch the live database” written in a prompt is something the agent is asked to remember, under pressure, while doing something else. A login that has no access to the live database is something it can't get wrong. Put the rule in the system it touches, not just in what it's told.
3. It stops itself, and anyone can stop it
Toyota solved a version of this problem more than a century ago. Sakichi Toyoda's early power looms had a device that shut the machine down the moment a thread broke, instead of weaving on and spoiling the cloth. Because each loom stopped itself, one operator could run several at once. By his automatic loom of 1924, one worker could look after 30 to 50. Toyota calls the idea jidoka, and it became a pillar of the Toyota Production System, alongside the cord any worker can pull to stop the line.
“The machine or equipment can detect the abnormality and stop automatically, or the operator can stop the line by pulling the stop cord themselves.”
The same holds for agents. Each one checks its own output against a plain test, such as “more than one in twenty invoices don't match”, and halts when it fails, rather than carrying on at machine speed. And anyone who sees it going wrong can press stop, without asking permission. That's the first step of stop, tell, fix, learn, built in.
4. Reach is earned
New builds can't reach live systems until they're promoted. They start in a sandbox with test copies and made-up data. When they've shown they work, they move to a limited live trial: real users, the systems they already use, narrow reach and an owner watching. Only once the trial's evidence says keep it do they get the full reach their job needs, with an owner, a backup who can stop them, and a review date. You'll run that two-stage proof in module 6, and give what you keep an owner in module 7.
Try it on an agent your organisation is building, or one a vendor is offering. Tick what it needs to reach, and see what each stage allows and what has to be in place before it moves up.
Test copies and made-up data only. No live systems.
Real users and live systems, with narrow reach, an owner watching and a set end date.
Full reach for its job, no more. An owner, a backup who can stop it, and a review date.
What it needs to reach
The rules, set before it moves up
Many of the first agents in an organisation arrive inside software it already uses: a CRM that can now send follow-ups on its own, an inbox that files and replies. They follow the same rules. They go on the approved tools list through the vendor checklist, they get their own identity where the software allows it, and their reach is set in the admin settings before they're switched on for everyone.
An outside builder is setting up an agent to reconcile supplier invoices in your finance system. To save time, they ask for it to run on the finance lead's login, which has full admin rights. “It only needs to read invoices, so it doesn't matter.”
What do you do?
An agent can only do what its access allows. Set the limits in its permissions, not just in its instructions.
A promising agent has worked well in the sandbox for two weeks. What's the next step?
Mark it complete. Your answers are already saved in this browser.