By Illitch Real
The AI agent delusion
Nothing captures the current state of AI better than a confession I heard last week at an insights conference. A senior researcher from a major FMCG company stood before an audience of professionals and said, with refreshing honesty: “Look, we don’t really understand AI, we are not an AI company, but we love it.”
That moment of intellectual humility was worth more than a thousand PowerPoint slides on “transformative AI strategies.” Here was someone who understood a principle many C-suite executives seem to have forgotten: acknowledging ignorance is the beginning of wisdom, not the end of credibility.
And yet, in the same week, I watched CEOs from leading tech companies demonstrate AI agents that could help you shop for groceries and design
When AI can’t multiply
I recently published a simple test: I asked several frontier AI models to perform basic multiplication. They failed, until prompted to double-check. This isn’t about arithmetic. It’s about trust. When leading AI business solutions can’t do what a three-dollar calculator can, you don’t have a co-pilot. You have a performance.
Paul Weiss, one of the world’s most prestigious law firms, spent 18 months piloting an AI tool to streamline legal research. The verdict? Verifying the system’s output took so much effort that, in their own words, it “makes any efficiency gains difficult to measure.”
Think about that. If AI can’t reliably tell you what a judge wrote in a published opinion, why are we betting the farm on AI agents managing complex business processes?
We ought to be demanding. Being directionally correct is not good enough.
The astrology of AI Strategy
“When leadership doesn’t understand the technology they’re implementing, every decision compounds the risk.”
Here’s an uncomfortable truth most consultants won’t tell their clients: many C-suite executives are making AI strategy decisions based on the equivalent of astrology. A 2024 IBM study of over 3,000 CEOs revealed a stunning disconnect. Sixty-nine percent believe their organizations are ready for generative AI, but only 29 percent of their executive teams feel they have the expertise to implement it.
A Gartner analysis found that over 80 percent of CEOs believe AI will be the most transformative technology for their industry. But when asked, only 44 percent of CIOs and 46 percent of CISOs said they have the AI knowledge needed to deliver on that vision. This is a competence cascade failure waiting to happen. When leadership doesn’t understand the technology they’re implemen-ting, every decision compounds the risk.
The solution is what I call cognitive skin in the game. Executives need to get their hands dirty with the actual tools, understand their limitations viscerally, and learn to distinguish AI mythology from operational reality.
An Accenture study published in the Ivey Business Journal explains that companies with AI-savvy leadership, where CEOs actively shape AI strategy, are twice as likely to understand how AI creates value. These leaders don’t just read about AI. They experiment with it, fail with it, and develop intuition about when it works and when it doesn’t.
The great professional services awakening
Something interesting is happening across professional services. Over the past year, many firms have launched internal AI initiatives that turned out to be little more than elaborate scaffolding built around prompt engineering and data lakes. Now, many of those same firms are quietly admitting defeat.
The dirty secret of the AI consulting boom is this: most firms build AI-adjacent products without really understanding AI itself. They optimized for demos, not deployment. For impressiveness, not reliability.
Some argue we’re being too cautious. That we should move fast and embrace AI adoption. They point to early internet sceptics as a cautionary tale. But there’s a crucial difference. The internet promised to connect us, not think for us.
When AI agents promise to handle complex reasoning but can’t reliably multiply, healthy scepticism isn’t pessimism. It’s prudence.
A final thought on buzzwords
“AI agent” has joined “synergy” and “disruption” in the pantheon of corporate buzzwords.
The firms that will thrive in the AI era won’t be the boldest or the fastest to chase new model releases. They’ll be the ones with a cognitive immune system, able to resist hype, filter signal from noise, and build on their own DNA.
The first major business disasters from unchecked AI implementations are coming. The headlines will be brutal. And the executives who greenlit systems they didn’t understand will be left explaining to shareholders, regulators, and employees how their “transformation” turned promise into liability.
AI won’t replace judgment. But it will expose the lack of it. So don’t build agents to do what you haven’t mastered. Build the understanding first, through hands-on practice, critical thought, and strategic patience. Not with hype. Not with fear.
