From Chatbots to Agents: Why Observability and Safety Matter

Summary

What happened

Moderna co-founder Noubar Afeyan sparked debate by likening the fast rollout of AI agents to the thalidomide tragedy, urging mandatory safety checks first. At the same time, EU regulators and cybersecurity groups are updating rules to address these "agentic" systems.

Why it matters now

We are moving from generative AI to agentic AI. Existing compliance setups were built to handle text outputs and bias, not tools that can call APIs, query databases, and make decisions without oversight.

Who is most affected

Enterprise CISOs, compliance teams, and developers working on integrations, along with companies rolling out customer or internal autonomous agents.

The under-reported angle

The shift from simple access control to intent observability. Auditors now want more than proof an agent had permission; they need to see the reasoning and goal behind each action.

Deep Dive

Governing artificial intelligence is running into a basic problem: the move from generating responses to taking real actions. For the last couple of years, rules focused on what LLMs output—copyright issues, bias, jailbreaks. Now that agents can chain steps, trigger APIs, and handle tasks independently, those frameworks feel outdated. The conversation is turning toward system-level accountability instead of just watching the model itself. It is no longer mainly about what an AI said, but about what it did, why it was permitted, and where liability lands.

This point came into focus when Noubar Afeyan compared the quick release of AI agents to the thalidomide case. The comparison draws from how medicine requires strict safety checks before anything reaches patients, and it challenges the tech habit of moving fast. A chatbot making something up is one kind of problem. An agent with financial access or system rights acting outside its bounds is something far more serious. Regulators appear to be taking notice and leaning toward required safety reviews before agents enter wider use.

For an enterprise CISO, agentic AI creates risks that current tools do not cover well. Static permissions fall short. If an agent removes important data, logs may only confirm it had the credentials. New standards push for both execution observability—what the agent actually performed—and intent observability, meaning the context and objective behind the step. Without a record of the agent's internal logic, meeting requirements like SOC 2 or ISO 27001 becomes difficult.

On the legal side, rules are already tightening. Article 14 of the EU AI Act calls for strong human oversight on high-risk systems. For agents, keeping a person in the loop is more involved than it sounds. It means having the ability to watch operations in real time, avoid over-reliance on automation, and step in or stop the process when needed. In practice, deploying an agent requires a reliable way to shut it down if things go off track.

Because of this, legal and security teams are stepping up preparation. Some firms now recommend tabletop exercises that simulate a rogue agent crossing its limits. As shadow AI gives way to shadow agents, the next bottleneck may not be GPUs but the gateways, permission systems, and monitoring needed to keep these tools in check.

Stakeholders & Impact

About the analysis

This review draws from current legal guidance, academic models on AI governance, and compliance data. It is meant for CTOs, CISOs, and enterprise leaders handling the shift from LLM tests to live agent deployments.

i10x Perspective

The push around AI agent rules points to a bigger change in how intelligence systems will be designed and sold in the coming years. The focus is moving beyond raw compute power toward building auditable, controllable environments for models to run in. As liability for autonomous actions becomes clearer, a supporting market around observability and control tooling is likely to grow—enterprises will want systems they can manage within legal bounds.