Are you a little weary of hearing about all the things that AI will do? Not yet? How about all the claims that your job is going to be taken over by AI (along with everyone else’s on the planet)? How about the recent layoffs and the assertion that they are due to AI?

In my May column, I wrote about the survey of C-suite executives showing how much pressure they were under to prove that they were getting AI to work (which really means cut headcount), and how out of touch these promises were with reality. I strongly believe that employers wanted to do layoffs anyway to keep investors happy, and are hoping the cuts will somehow be offset by introducing AI.

For a year now, I have been searching for actual evidence of cases where AI has been introduced and how it has changed jobs. I’ve written before about Ricoh’s effort to transform some simple paper processing tasks. What they found initially was that using AI—and by that, we mean large language models like ChatGPT—to simply take over tasks was three times more expensive than having their employees do the work. After a lot of time and investment, they found ways to make it work with AI, but it was very time-consuming, very expensive and it barely cut headcount.

A lot more data could be handled by the same employees—productivity eventually went up by a lot—but employees still needed to chase down all the issues AI could not handle. It was an important improvement, but it was a very big undertaking, and headcount did not fall, although productivity was ultimately up a lot.

Where is AI working?

I got my chance to dig deeper with a partnership Wharton has with Accenture, where the professional services firm found us companies that had introduced AI. I got to interview them in depth to ask this question: What did the work look like before introducing AI, what did it look like after and what was required to make that happen? We extended this effort in a meeting with 30 companies talking about their efforts in late October.

You can listen to the podcast account of four cases, and you might well be surprised.

Here are the main conclusions:

1. Off-the-shelf AI did not do the trick—anywhere

This was in contrast to what the C-suite executives said in their survey. The fact that we can play around with the free and cheap versions of LLMs like ChatGPT, Gemini, Copilot and Claude doesn’t mean they can handle real business problems. A simple task like sorting documents gets complicated when it has to be done a million times. The business model of these AI companies is not to provide free service.  As they burn through money, don’t bet on that lasting.

2. Most of the successful cases were using machine learning, not LLMs

Machine learning was hot before the pandemic. It’s statistics for engineers, used to make predictions based on analyzing lots and lots of data:  When will this machine break, what image of people fits the approved list (pattern recognition), when will customers be interested in asking for loans? It requires building algorithms from your data, and it’s really expensive to do.

3. The biggest challenges were all around management, not AI

This is the big one, and there was universal agreement. Especially in trying to introduce LLMs into white collar work (what people mean when they say “agents”), virtually all the work looks like old-fashioned HR, starting with “job analysis.” First, we have to figure out the workflow in any project or task, then what each employee does in that project, every task they handle. Then, we have to figure out whether an “agent” can be introduced into any task. We need the help of employees to figure that out. If an agent can be introduced, then we need the employees to “train” it: Is it giving us the right answers? If not, what should be different?

Once we get agents on these tasks, then we have to jump into job redesign: “Ten percent of this job is now done by agents, 20% of that one.” Does it make sense to reorganize the individual jobs to improve the flow?

4. Headcount did not fall

This was the big surprise to the pundits: Productivity goes up, quality goes up and we can do more with the same workforce. But what the C-suite executives and the investors behind them were hoping for—cheap and quick headcount reductions—simply does not happen.

My strong sense is that much of the frenzy around AI and, particularly, the claims that AI is behind layoffs all rest on the wish—which we can now call a myth—that AI is the miracle drug that cuts jobs, the cost investors dislike the most. It will be a hard and unpleasant bone to swallow that using AI is expensive, hard work that does pay off, but over the long run.

Will investors and boards figure that out before executives get fired? And will that happen before we cut jobs and then can’t figure out how to get the work done?