The Fast Company Impact Council is an invitation-only membership community of leaders, experts, executives, and entrepreneurs who share their insights with our audience. Members pay annual dues for access to peer learning, thought leadership opportunities, events and more.
It’s been five years since the intense early days of the COVID-19 pandemic and the first round of lockdowns that mandated work-from-home for companies around the world. Among the debate at the time: concerns about how younger workers and new recruits would cope without access to experienced colleagues and mentors.
Doomed to impersonal video conferencing in converted bedrooms, these youngsters couldn’t hope to gain the confidence and deep experience of their predecessors. They would make their mistakes out of sight, and fail to learn.
Now imagine those new workers and interns are digital, not human. Since OpenAI launched ChatGPT at the end of 2022, it’s not unusual to see generative AI systems referred to as interns, coworkers or even colleagues.
In that case, it’s tempting to see their offspring, AI agents, as more experienced employees. Using the “brain” of a large language model, agents are given a specific purpose and granted access to an organization’s software tools and data in order to autonomously fulfil their task.
For many enterprises, the question is not whether they should adopt agentic AI, but how quickly and how widely. Gartner forecasts that, agentic AI will address and resolve 80% of regular customer service issues with no human intervention by 2029, and this will result in a 30% reduction in operational costs. With stats like that, other business functions will surely follow—and fast.
Chain of thought
Big-name tech companies such as Salesforce are going all-in on an agentic future and AI companions are already a common feature in business tools such as Zoom and Slack. AI rivals are reaching agreement at an unprecedented pace on new technology protocols that allow the integration of AI models with all types of business tools and applications.
In this new era, the digital workers are being handed the keys to the enterprise. What can possibly go wrong? Potentially, quite a lot. All the major models are fallible and flawed. As Anthropic, maker of the popular Claude family of AI models, explains in a new research paper:
“Language models like Claude aren’t programmed directly by humans—instead, they’re trained on large amounts of data. During that training process, they learn their own strategies to solve problems. These strategies are encoded in the billions of computations a model performs for every word it writes. They arrive inscrutable to us, the model’s developers. This means that we don’t understand how models do most of the things they do.” [Italics added for emphasis.]
Anthropic’s own research shows Claude being tricked into naming the ingredients for a bomb, though stopping short of giving instructions on how to make one. Separate Anthropic-backed research found that more advanced reasoning models, which show the chain of thought they use to reach their conclusions, “don’t always say what they think.”
Without the ability to rely on chain of thought, “there may be safely-relevant factors affecting model behavior” that remain hidden, the researchers concluded. (The researchers evaluated the faithfulness of two state-of-the-art reasoning models, Claude 3.7 Sonnet and DeepSeek-R1.)
Connecting AI models to business tools, via agents, raises the safety stakes. An agent that has access to an email system can be exploited as a useful tool for attacker intent on phishing. Access to database systems can be levered to extract valuable data from an organization. Even instances of accidental misuse can have significant consequences in terms of disruption, cost, and reputational damage to an organization.
An adult in the room
In the absence of the ability to predict or drive the behavior, these new digital colleagues—like their human counterparts—need chaperones to provide guidance and feedback. It’s important there is at least one “adult” in the room to constantly monitor these (not very streetwise) interns, intervening in real time when they may be sent on a fool’s errand, tricked into handing over their wallet, or encouraged to say or do something offensive or illegal.
We know from experience that attempting to rapidly introduce new technology across an enterprise can be a recipe for chaos. Someone, somewhere—and likely many people—will find themselves in the headlines looking silly, at best. At worst, they may lose valuable intellectual property and suffer serious financial and reputational loss.
The best solution for an agentic workforce is agentic oversight—using powerful, customized agents to simulate real-world scenarios and probe AI for weaknesses. Continuous, automated “red teaming” of these new technologies, at speed, can give enterprises the confidence they need before they send their armies of new interns and employees out to do real jobs.
This agentic warfare approach offers the greatest chance of implementing enterprise AI for its intended purposes. After all, you wouldn’t give an unvetted new employee completely unhindered and unsupervised access to your business systems, would you?
Donnchadh Casey is CEO of CalypsoAI.
