How to Set Safe Limits for Autonomous AI Agents

AI agents are becoming capable of doing more than generating information—they can now analyze code, identify vulnerabilities, write potential exploits and interact with the systems they are designed to help secure. Those capabilities can dramatically accelerate security work, but they also introduce new risks when an agent can move from identifying a problem to taking action without human intervention.

So where should companies draw the line between useful autonomy and unacceptable risk? And what should an AI agent be allowed to do independently when its actions could affect production systems, sensitive data or security controls?

Members of the Senior Executive AI Think Tank—a curated group of experts specializing in machine learning, generative AI and enterprise AI applications—examine how leaders can use sandboxing, least-privilege access, deterministic controls and human oversight to contain risk, while also addressing less technical questions around liability, accountability and reversibility. Together, they offer a practical framework for deciding not just what AI agents can do, but what organizations should actually allow them to do.

“Security debt from autonomy compounds silently until it breaks into the open. Fund a red team now, before an attacker finds out for you.”

– Matan Mishan, Senior Vice President, Agentic at Dot Compliance

SHARE IT

Make Autonomy Deterministic, Not Confidence-Based

For Matan Mishan, Senior Vice President, Agentic at Dot Compliance, the first mistake leaders make is treating model confidence as a proxy for operational permission.

“Autonomy should scale with determinism, not confidence,” he says.

His proposed answer: Let the model reason, but don’t let the model decide what infrastructure it can ultimately touch.

“For sensitive infrastructure or data, the boundary is deterministic code, not a well-behaved model,” he says. “Build hybrid systems: Agents plan and reason, but a gateway, not the LLM, enforces what actually gets touched or executed.”

Mishan also argues that organizations should establish measurable tests before handing over meaningful authority.

“Before granting autonomy, do Eval-Driven Development,” he says.

He compares the approach with test-driven development: Organizations should translate business rules into concrete evaluations before deciding which tasks can be handled by an LLM and which require deterministic software.

“Can’t write the eval? You’re not ready to hand over the keys.”

That preparation matters because autonomy can create security debt that is easy to overlook while a system is working as intended. Mishan’s recommendation is to test those assumptions proactively rather than waiting for an attacker or a failure to expose them.

“Security debt from autonomy compounds silently until it breaks into the open,” he says. “Fund a red team now, before an attacker finds out for you.”

Put Liability on the Same Page as Security

Ajay Pundhir, Founder of AskAjay.ai and Director of AI at Presight (G42), argues that many organizations are building increasingly sophisticated technical controls while overlooking the most basic question: Who is accountable when an autonomous system causes harm?

“Most of this conversation is about controls. Almost none of it is about liability, and that’s the gap I keep hitting in regulated markets,” he says. “Before drawing the boundary, answer a simpler question: If the agent is wrong at 2 a.m., who pays, and can they prove they didn’t authorize it?”

That question changes how executives should think about autonomy. The appropriate permission level isn’t necessarily the maximum action a model can perform. It is the maximum action an organization can defend, insure and assign to a responsible person.

“My rule: An agent acts independently only as far as a named human’s signing authority and the company’s insurance actually extend,” he says. “Not its capability. Not its accuracy. Its coverage.”

Pundhir expects insurers and regulators to become increasingly important in defining practical autonomy limits.

“Underwriters and regulators will define agent autonomy long before security teams do. Better to write that boundary yourself.”

Keep Humans in the Loop Where Damage Becomes Real

Peter Boyd, General Guru and CEO of PaperStreet Web Design, says treating AI like an ordinary productivity tool creates a fundamentally different legal and operational exposure.

“When AI can find and exploit vulnerabilities, treating it like a standard tool invites disaster, otherwise your company can be liable for damages.”

Boyd draws a hard line around the most consequential actions.

“AI can draft solutions, but humans must always be in the loop.”

In practice, that means the agent can contribute heavily to research and preparation without receiving unrestricted authority to execute its conclusions.

“Never grant autonomous agents root access or live deployment powers, and strictly limit their operational environment so a mistake stays contained.”

Learn From Systems Built for Extreme Autonomy

Gabriella Goddard, CEO and Director of AI Catalyst Leader™ at Brainsparker Ltd, approaches the issue from leadership, innovation and human capability. She is candid about the limits of her own specialization.

“I’m not an AI security specialist, but one lesson I’ve learned from mentoring startups in the UK Space Agency Accelerator is that some of the best ideas come from industries solving the hardest versions of a problem.”

That observation points executives toward an unusual source of security insight: space systems.

“Companies developing autonomous AI for Mars or asteroid missions have been thinking about safe autonomy, fail-safes and human oversight for years because communication delays mean there often isn’t a human available to intervene.”

The implication for enterprise leaders is not that terrestrial businesses should copy space systems wholesale. It is that autonomy should be designed around the possibility that intervention may sometimes be unavailable.

“As AI agents become more autonomous on Earth, I’d encourage leaders to look beyond their own industry,” she says. “Sometimes the best solutions come from sectors already operating in far more demanding environments.”

Treat Every Agent Like a New Employee

Mo Ezderman, Director of AI at Mindgrub Technologies, recommends taking a familiar enterprise process and applying it to AI agents.

“We should start treating AI agents like new employees,” he says.

The analogy is useful because organizations generally don’t hire a person and immediately give that person unrestricted access to every system.

“We already have strong processes for hiring humans: We interview them, test them, onboard them and give them only the access they need to do their jobs.”

The same principle should apply to agents: capability assessment first, authorization second.

“Before deploying an agent, we should test it, understand its capabilities and risks, and limit its access.”

“New capabilities should prompt reassessment of authority, not automatically increase it.”

– Manpinder Singh Panesar, Senior Solutions Architect at Amazon Web Services (AWS)

SHARE IT

Delegate Authority, Not Just Tasks

Manpinder Singh Panesar, Senior Solutions Architect at Amazon Web Services (AWS), frames autonomy as delegated organizational authority.

“I would define agent autonomy through the team delegating the work: What authority does it hold, and what can it safely delegate?” he says.

An agent’s ability to perform a task does not automatically establish that it should have authority to perform every action associated with that task.

“Team authority sets the ceiling; agents receive only the permissions their tasks need.”

He also emphasizes accountability that survives multi-agent workflows.

“Every delegated scope should have one named human accountable, with boundaries preserved through agent handoffs,” he says. “Enforce permissions outside the model, record actions, require approval for consequential changes and retain a way to stop execution.”

Clear ownership, he adds, allows for problems to be solved while teams examine systemic failures. 

“New capabilities should prompt reassessment of authority, not automatically increase it.”

Gate Real-World Execution

Independent Researcher Aishwarya Shah focuses on the point where AI reasoning becomes operational action. Her recommendation is to move away from monitoring systems after they act and instead build constraints into the architecture before execution occurs.

“AI leaders must transition from post-hoc monitoring to deterministic constraint architectures.”

That means reducing the number and type of actions an agent can take.

“Enforce least-privilege action spaces: Agents must operate in sandboxed environments with strict read-write boundaries, rate limits and cryptographic verification for external API execution.”

Shah distinguishes between high-value autonomous analysis and consequential changes to real systems.

“Agents can autonomously discover vulnerabilities, correlate threat intelligence and draft patches, but system state changes must require human authorization.”

For Shah, model intelligence does not equal operational trust.

“True security lies in gating autonomous agency at the point of real-world execution.”

Treat Agents as Untrusted Code

Pradeep Kumar Muthukamatchi, Principal Cloud Architect at Microsoft, recommends fundamentally changing the default security posture.

“Cyber-capable AI agents require a shift from hoping models behave to enforcing deterministic architectural containment,” he says. “Treating agents like untrusted code means applying least-privilege permissions, ephemeral sandboxes for code execution, and strict JSON schema gateways to block unvalidated system calls.”

Muthukamatchi sees a meaningful role for autonomous security research, but not autonomous production control.

“Setting clear boundaries means letting agents independently analyze code, run vulnerability scans or test exploits inside isolated environments.”

The hard stop comes when the agent could alter a live environment.

“Hard structural limits must forbid autonomous code commits, firewall changes or live payload deployments without signed human authorization,” he says. “Defensive teams need machine-speed AI discovery, but agents must have zero control over their own constraints, blast radiuses or authorization rules.”

“Anything that touches live systems, changes permissions or can’t be easily undone needs a human in the loop, a sandboxed test environment and strict limits on what the agent can actually reach.”

– Muthukumarapandian Chandrasekaran of CitiusTech

SHARE IT

Separate Thinking From Doing

Muthukumarapandian Chandrasekaran of CitiusTech makes a simple recommendation.

“AI agents can now find security holes and write code to exploit them. That changes the math on autonomy,” he says. “My take is to separate thinking from doing.”

For security teams, that means allowing agents to perform expansive analytical work without automatically granting them equivalent operational authority.

“Agents should be free to analyze, flag issues, and suggest fixes,” he says. “But anything that touches live systems, changes permissions or can’t be easily undone needs a human in the loop, a sandboxed test environment and strict limits on what the agent can actually reach.”

His final point puts the emphasis where it belongs: consequences rather than benchmarks.

“The real question to ask isn’t ‘How capable is this model?’ It’s ‘What happens if it’s wrong, and who’s watching when it acts?’”

Make Reversibility the Boundary

Hastimal Jangid, Co-Founder of RankRabbit AI, shifts the autonomy debate away from raw capability and toward the consequences of an action.

“The line shouldn’t be capability, it should be reversibility,” he says. “An agent that finds a vulnerability is a research asset; one that acts on it unsupervised, patching, exploiting, touching production, is a different risk category.”

Jangid recommends separating discovery permissions from action permissions.

“Discovery and action need separate permissions: An agent earns trust to find a flaw before it earns trust to exploit one unsupervised.”

And as models become more powerful, he argues that organizations should resist the instinct to automatically expand their authority.

“Industry instinct is to expand autonomy as capability grows. In security, it should narrow instead.” he says. “The more powerful the agent, the more auditable its authority needs to be.”

Managing AI Agent Autonomy

• Make autonomy proportional to deterministic controls. Use gateways and non-model enforcement mechanisms to determine what an agent can access or execute.
• Require an evaluation before granting authority. Translate business rules into concrete tests and treat an inability to evaluate a task as a reason to retain human control.
• Put liability beside security on the governance agenda. Every autonomous scope should have a named human accountable for the outcome and an organization prepared to defend the authorization.
• Keep humans at irreversible decision points. Agents can research, analyze and draft, but live deployment, privilege changes and other consequential actions should require human approval.
• Borrow autonomy practices from high-risk industries. Aerospace and other demanding environments offer lessons in fail-safes, constrained autonomy and planning for situations where intervention may be unavailable.
• Treat every agent like a new employee. Test its capabilities, understand its risks, onboard it deliberately and give it only the permissions necessary for its role.
• Define authority before delegation. Establish the team’s maximum authority first, then give agents a narrower scope and preserve accountability through handoffs.
• Constrain agents before they act. Sandboxes, least-privilege permissions, rate limits and external authorization controls should be architectural requirements, not optional safeguards.
• Separate analytical autonomy from operational autonomy. Let agents discover vulnerabilities and recommend fixes without automatically giving them the ability to change live systems.
• Use reversibility as a core risk test. The harder an action is to undo, the stronger the authorization, containment and audit requirements should be.

Knowing Where to Draw the Line

AI agents will continue to get better at finding problems, making decisions and taking action. The challenge for leaders is deciding where that progress should end and human judgment should begin.

The Think Tank members point to a practical answer: Give agents room to work, but don’t confuse intelligence with authority. Let them investigate, test and recommend. When an action could affect a live system, change access, create significant damage or be difficult to undo, put a person between the agent and the outcome. As agentic AI becomes a bigger part of the enterprise, that distinction may be one of the most important security decisions leaders make.