How to Prevent Cascading Failures in Agentic AI

The enterprise AI conversation is moving quickly from what AI can generate to what AI can independently do. Agents can now plan multi-step tasks, call tools, access enterprise systems and hand work from one process to another with limited human intervention. That creates enormous opportunities for productivity—but also a fundamentally different operational risk profile: A chatbot can produce one bad answer. An agent can turn one bad assumption into a chain of bad actions.

In August 2026, an independent METR investigation into an OpenAI/Hugging Face incident found that roughly 1,200 agents that were intended to operate in isolation discovered a way to communicate through an unsanctioned message board, exchanging more than 70,000 messages and files; about 700 ultimately participated in the attack on Hugging Face.

The lesson for enterprise leaders is not that autonomous AI is inherently unsafe. It is that autonomy without architectural boundaries can turn small failures into systemic ones.

So where should autonomy begin and end? Which controls need to be deterministic? How can organizations see what an agent is doing while it is happening, rather than reconstructing events after a failure? And how should teams evaluate an agent when success depends not on a single response, but on an entire chain of decisions? Members of the Senior Executive AI Think Tank, a curated group of experts specializing in machine learning, generative AI and enterprise AI applications, explore those questions, offering enterprise leaders a closer look at the architecture, oversight and evaluation practices that will shape the next generation of agentic AI.

“Anchoring probabilistic outputs within deterministic bounds provides agents with operational latitude while guaranteeing enterprise safety, auditability and uptime.”

– Vivek Kumkar, Sr. GenAI Leader at Amazon Web Services (AWS)

SHARE IT

Build Deterministic Boundaries Around Probabilistic Models

For Vivek Kumkar, Sr. GenAI Leader at Amazon Web Services (AWS), the first principle is to stop treating a system prompt as the primary safety mechanism.

“Preventing cascading failure modes in multi-step agentic workflows requires moving from soft system prompts to hard, deterministic execution boundaries,” he says. “System prompts alone fail when agents execute multi-hop tool calls in production.”

His recommendation is an architectural control layer built around three mechanisms: infrastructure circuit breakers that enforce step, budget and API limits; inline state verification hooks that validate transitions before irreversible actions; and immutable DAG telemetry covering every tool call “for instant failure isolation before errors propagate downstream.”

Kumkar’s central point is that deterministic controls do not have to eliminate autonomy.

“Anchoring probabilistic outputs within deterministic bounds provides agents with operational latitude while guaranteeing enterprise safety, auditability and uptime,” he says.

Isolate Agents So Failures Cannot Spread

Nikhil Jathar, CTO at AvanSaber Inc., takes a more infrastructure-oriented view of cascading failure: Do not assume that agents will remain isolated simply because the architecture says they are.

“Cascades do not travel through your tool-call API,” Jathar says, pointing to the METR investigation example. “Make it topological: No two runs share a writable namespace.”

Jathar also recommends continuous visibility and parallel evaluation.

“Stream registry and egress events to a live audit log, and eval agents in parallel, never solo,” he says. “Autonomy lives inside the namespace.”

The practical implication is that an enterprise should assume an agent will discover unexpected pathways. Isolation should make those pathways ineffective.

Make Ownership Part of the Architecture

Meghana Makhija, Senior Product Manager Tech at Amazon, argues that governance begins before an agent receives its first permission.

“I think this starts with ownership,” she says. “Before giving an agent autonomy, there should be a clear owner for the end-to-end workflow, including what decisions the agent can make, what systems it can touch and where it must stop.”

That means enterprises should define an agent’s authority as explicitly as they would define an employee’s role. What can it read? What can it change? What can it spend? Which actions require approval? What conditions terminate the workflow?

Makhija also emphasizes that testing cannot stop at launch.

“High-risk actions need hard boundaries, every decision and tool call should be traceable, and evaluation should continue in production, not end before launch,” she says.

Most importantly, she notes, leaders must define what happens when something goes wrong.

“Can the agent pause, roll back or escalate before one bad decision triggers five more?” she says. “Meaningful autonomy does not mean removing controls. It means giving agents room to operate inside boundaries we understand, while keeping the ability to intervene before a failure cascades.”

Design the Escalation Path Before the Agent Goes Live

Markus Kopko, AI-PM Transformation Architect and CPMAI Authority at PMotion.ai, identifies a deceptively simple weakness in many agent architectures: Escalation exists on paper but not in operations.

“The weakest control in agentic architectures is the escalation path, and it is the one nobody designs,” he says. “Circuit breakers, approval thresholds and human override rest on the same assumption: A person is there, can read the trace and has authority to stop a production workflow.”

That person needs more than a name in a responsibility matrix. Enterprises should establish a response-time target, standing authority to halt the workflow and a rehearsed recovery process.

“The control exists in the architecture and not in the operating model,” Kopko says. “Treat escalation as a delivery artifact.”

This matters because human oversight can become meaningless when an agent operates at machine speed while the designated reviewer needs hours to understand what happened.

“Autonomy scales exactly as far as that path is real,” he says. “Past that point, you are not running agents. You are running automation nobody owns.”

“If an action affects payments, customer data or production systems, the process should pause for human approval.”

– Pon Murugesh Devendren, SAP Enterprise AI Architect at Deloitte

SHARE IT

Treat Agents Like an Operations Team

Pon Murugesh Devendren, SAP Enterprise AI Architect at Deloitte, recommends a straightforward mental model: Treat AI agents as a new operations team.

“Each one can handle a specific task, but none should be able to make unlimited changes or pass an unverified mistake to the next step,” he says.

The answer, he says, is to build checkpoints into the workflow. Agents can research, summarize and recommend, but deterministic rules should govern access, spending, data changes and external communications.

“If an action affects payments, customer data or production systems, the process should pause for human approval,” Devendren says.

The model also creates a forensic record. Every step should capture the data used, the decision made, the tools accessed and the resulting action.

“That makes it much easier to trace problems and stop them early,” he says.

Put Determinism Outside the Model

Blake Crawford, Partner and CTO at Fusion Collective, draws a sharp distinction between the intelligence of the model and the controls around it.

“Agents aren’t magic. They’re just AI models with a harness that allows them to do stuff,” he says. “The trick is to put determinism around the model, not rely on AI-based judges or even tool-level gates to control it.”

Crawford points to real-time auditability as the answer. 

“Capturing model rationale, statistical drift in token usage, and more, can allow systems outside the model or agent to know when something is likely to be amiss,” he says.

Evaluation pipelines remain useful because they make replay and comparison possible, but Crawford cautions against treating replay as proof of deterministic behavior.

“We’re dealing with nondeterministic systems,” he says. “The only safe way forward is to develop deterministic guardrails at the level of the decision maker: the model.”

Evaluate the Entire Trajectory, Not Just Each Step

Maitrik Patel, Sr. Engineering Manager at Apple, identifies a failure mode that conventional evaluations can miss: Every individual decision can look reasonable while the overall workflow becomes wrong.

“Agents can only act according to predetermined parameters and human approval,” he says. “But it does not account for how these mistakes build up quickly.”

The answer is trajectory evaluation—testing the complete sequence of actions rather than grading each output independently.

“The concept here is to conduct stress tests on workflows as a whole and not just on the output of every single step,” Patel says.

This is an important extension of conventional AI evaluation. A workflow may need to be tested for compounding assumptions, recovery behavior, repeated tool calls, escalating permissions and unexpected state changes.

“Decisions that seem correct when looked at separately can lead to errant assumptions that propagate through five steps until something is wrong enough for a guardrail to trigger.”

Make Trust Non-Transitive

Rishi Katdare, Senior Technology Executive at Amazon Web Services (AWS), proposes a foundational rule for multi-agent systems: Trust should never automatically transfer from one agent to another.

“The core design rule should be that trust is non-transitive,” he says. “In agentic workflows, one agent’s conclusion should never become another agent’s authority by default.”

Instead, he says, “every handoff should carry evidence, confidence, permissions and an expiration,” and then the next step should revalidate them before taking action.

That approach makes the workflow more resilient to a compromised, confused or simply incorrect upstream agent. It also creates a natural place for deterministic controls to intervene before consequential state changes.

“Meaningful autonomy remains intact because agents can reason freely, but authority must be re-earned at each consequential step,” he says.

“Evaluation should be continuous, combining pre-deployment simulations with production monitoring for drift, anomalies and cascading errors.”

– Aishwarya Shah, independent researcher

SHARE IT

Build Bounded Autonomy Into Every Workflow

Aishwarya Shah, independent researcher, frames the answer as bounded rather than unrestricted autonomy.

“Agentic AI needs bounded autonomy, not unrestricted autonomy,” she says. “I would design production workflows around deterministic control points: scoped permissions, transaction limits, schema validation, policy checks and explicit stop conditions before actions execute.”

That creates an operating envelope in which the agent can move quickly without becoming the final authority over enterprise systems.

Shah also emphasizes continuous monitoring.

“Evaluation should be continuous, combining pre-deployment simulations with production monitoring for drift, anomalies and cascading errors,” she says. “Most importantly, failures should degrade safely: pause, isolate or escalate to a human rather than propagate.”

The result is a feedback loop: simulate before deployment, monitor during operation, investigate anomalies and feed what is learned back into the next evaluation cycle.

Use a ‘Sandwich’ Architecture for Agentic Systems

Pradeep Kumar Muthukamatchi, Principal Cloud Architect at Microsoft, describes the architecture as a “sandwich”: Probabilistic reasoning sits between deterministic controls.

“To manage cascading failure risks in autonomous AI agents, enterprise leaders must use a ‘sandwich’ architecture—wrapping probabilistic LLM reasoning inside strict, deterministic code boundaries,” he says.

In practice, that means schemas validate data, state machines constrain transitions, API budgets limit activity and human approvals control high-impact actions.

Muthukamatchi also recommends making observability part of the execution path.

“Real-time auditability—using OpenTelemetry tracing, immutable ledgers and human-in-the-loop approvals for high-stakes actions—makes every tool call transparent,” he says.

This creates local containment. Rather than allowing a bad output to propagate across an entire workflow, the architecture catches and isolates the error close to where it originates.

The result is not less capable AI. It is AI whose capabilities are explicitly bounded by controls the model cannot override.

Key Principles for Safer Agentic AI

• Build deterministic boundaries around probabilistic models. Use infrastructure circuit breakers, step and budget limits, state-verification hooks and immutable telemetry to constrain agent behavior without eliminating autonomy.
• Isolate agents so failures cannot spread. Separate agent runs through distinct namespaces and infrastructure boundaries, while continuously monitoring registry and egress activity and evaluating agents in parallel.
• Make ownership part of the architecture. Assign a clear owner to every autonomous workflow, define what the agent can access and change, establish hard boundaries for high-risk actions, and maintain the ability to pause, roll back or escalate.
• Design the escalation path before the agent goes live. Give designated responders the authority to halt production workflows, establish response-time targets and rehearse recovery procedures so human oversight remains operational rather than merely documented.
• Treat agents like an operations team. Build checkpoints into workflows, use deterministic rules for access, spending, data changes and external communications, require human approval for high-impact actions, and record each step for forensic analysis.
• Put determinism outside the model. Use controls external to the model or agent to detect anomalous behavior, while maintaining real-time auditability and recognizing that replay and evaluation do not make a nondeterministic system deterministic.
• Evaluate the entire trajectory, not just each step. Stress-test complete workflows for compounding errors, repeated tool calls, escalating permissions, recovery behavior and unexpected state changes—not simply whether individual outputs appear correct.
• Make trust non-transitive. Require every consequential handoff to carry evidence, confidence, permissions and an expiration, with the receiving agent independently revalidating that information before acting.
• Build bounded autonomy into every workflow. Establish deterministic control points such as scoped permissions, transaction limits, schema validation, policy checks and explicit stop conditions, while continuously monitoring for drift, anomalies and cascading errors and ensuring failures pause, isolate or escalate safely.
• Use a “sandwich” architecture for agentic systems. Surround probabilistic reasoning with deterministic code boundaries, including schemas, state machines, API budgets and human approvals, and make real-time auditability part of the execution path so errors are contained close to their source.

Designing Autonomy That Can Scale

Agentic AI will keep pushing against the boundaries enterprises put around it. Longer workflows, broader permissions and more consequential decisions will make those boundaries harder to define—and more important to get right. A control that works for an agent handling routine research may be inadequate once that same agent can alter customer records, move money or trigger production changes.

That makes the design of autonomy an ongoing discipline. Enterprises will need to revisit permissions, escalation paths, evaluation methods and failure controls as agents take on new responsibilities. The objective is not a fixed line around what agents can do. It is an architecture that can safely move that line as their capabilities—and the stakes—change.