An AI pilot is a small, time-boxed trial of an AI tool on one real job, run to find out whether it pays before you spend more on it. AI Money Check is our new series on what AI actually does to a small business's money, and its first rule is the one this post is about: find out what a number measured before you let it make a decision for you.
I've had AI pilots fail on me... and paid for the privilege. Ha ha! Anyone else? I once paid $6,500 to set up an AI call agent, plus $200 a month after that, and the company behind it filed for bankruptcy in September. Then on October 24, 2026 I ran a pilot of my own that didn't make the cut either: an overnight run on xAI's Grok Imagine, a tool that makes images and short video clips, to rebuild two of our short videos, with a $12 cap set before it started. It used about $2.13, and when I watched them, the videos still looked like graphics from about five years ago. Same verdict, very different bill.
Where does the "95% of AI projects fail" number come from?
It comes from "The GenAI Divide: State of AI in Business 2025", a report from MIT's Project NANDA by Aditya Challapally, Chris Pease, Ramesh Raskar and Pradyumna Chari, dated July 2025 and labeled preliminary findings. Its summary says "95% of organizations are getting zero return" on generative AI, the kind of AI that writes text, pictures or code, like ChatGPT or Claude, despite $30 to $40 billion of enterprise spending.
Then the number traveled. On August 18, 2025, Fortune ran it as "MIT report: 95% of generative AI pilots at companies are failing," and the same day The Register's page title read "Generative AI does nothing for 95 percent of companies." Organizations became pilots, pilots became companies, and by now you'll hear it as "95% of AI projects fail."
What did the MIT report actually measure?
The report drew on three things gathered between January and June 2025: a review of over 300 publicly disclosed AI initiatives, structured interviews with representatives from 52 organizations, and 153 survey responses from senior leaders at four industry conferences. Its appendix defines success as "deployment beyond pilot phase with measurable KPIs," a KPI being the one number a business watches to know whether something worked, with the return "measured 6 months post-pilot."
So "failed" meant a big company's pilot hadn't reached full use with measured results six months on. It didn't mean the tool broke, and the report doesn't show the arithmetic that produced the 95%. The authors are upfront about the limits: their figures are "directionally accurate based on individual interviews rather than official company reporting," and they list "selection bias possible in organizations willing to participate in AI research." Fortune's article described the inputs as 150 interviews and a survey of 350 employees, which doesn't match the July 2025 preliminary report we read.
Two findings in the same July 2025 report matter more to a small business than the headline. Tools bought from or built with a specialist reached full use "~67% of the time, compared to ~33% for internally built tools," though the authors say the gap "may reflect organizational capabilities." And workers at over 90% of the companies surveyed used personal AI tools for work, while only 40% of the companies had bought an official subscription.
What do the solid numbers say about businesses my size?
The U.S. Census Bureau's Business Trends and Outlook Survey asks businesses every two weeks whether they used AI. Its May 26, 2026 story reports that 19.8% of U.S. firms used AI in the two weeks ending May 3, 2026, and 32% of firms with 100 to 249 employees and 37% of firms with at least 250. Use among firms with fewer than 20 employees didn't change significantly between December 2025 and May 2026.
The Federal Reserve's April 3, 2026 note, "Monitoring AI Adoption in the US Economy" by Jeffrey S. Allen, puts firm-level adoption at about 18% at the end of 2025, while about 41% of the workforce reports using generative AI for work. The note says the gap is mostly about what each survey counts, every firm equally or every worker. And Google's 2025 DORA report, published September 23, 2025 from a survey of nearly 5,000 technology professionals, found 90% use AI at work and over 80% say it made them more productive, while only 24% trust it "a great deal" or "a lot."
The long and short of it is that most small firms haven't adopted AI on paper, a lot of their people already use it anyway, and the people who use it most still check its work. That's a picture of early, uneven use, not of near-total failure.
What does a failed AI pilot cost a 5 to 200 person business?
Whatever you let it cost. My call agent cost $6,500 up front plus $200 a month, and you can read why it didn't match the demo. My video pilot cost about $2.13. Both failed. The difference wasn't the AI... it was the size of the bet and how soon I judged it.
The upside is just as countable. Our flagship production system took one job its users used to do by hand from 90 to 120 minutes down to about 45 seconds. Here's the arithmetic for your shop, with your own numbers in place of mine: if someone does a 90-minute job like that five times a week, that's about 7.5 hours a week, and at $35 an hour that's about $260 a week, or roughly $13,650 over 52 weeks. A med spa owner with about 15 staff told me about 8 hours per person per week go to the work around missed enquiries, so doing nothing has a price too.
How do you keep a failed AI pilot cheap?
Set four things before you start. A spending cap, in dollars, that you'd be fine losing. One number to judge it by, like hours saved, calls answered or leads followed up the same day. A date to judge it on, a week or two out, not six months. And a copy of the setup and its instructions you keep, so a vendor going away doesn't take your work with it.
That's the shape of our First Fix: one automation, live in a week, for $500 flat. Yes, that's us selling something, and the four rules work just as well without us. Done right, a small pilot saves you time, keeps your money where you can see it, makes your people more efficient, tells you what to build next and keeps you out of long contracts... all of it.
Where does this approach not work?
A one-week pilot is the wrong test for work that pays off slowly. Getting found in AI answers or on Google takes months, and a seven-day check will call it a failure. MIT's own authors say six months "may be insufficient" for complex projects.
It doesn't work on a job you can't count. If you don't know how long something takes today, you can't tell whether a tool made it faster, so measure the before first.
It doesn't work on rare jobs. A task that happens twice a year will never pay back the setup, so don't automate it.
My two pilots are a ledger, not a study, and they don't prove anything about your business. And if nobody on your team will own the tool after week one, spend the money somewhere else.
What's one job in your business you'd be willing to test for a week, with a cap you set?
If you're not sure which one, the Bottleneck Finder™ is our free diagnostic, and it will point you to the job worth testing first, or tell you to build nothing yet. No worries either way.