← Blog When it does not work
Why do AI projects fail?
The demo worked. The meeting went well. Six months later the system is still in staging and nobody wants to be the person who says it should be switched off. That path has repeated causes, and none of them is the model.
Key takeaways
- MIT measures that 95% of AI pilots return nothing you can put a number on, across 52 executive interviews, 153 surveys and 300 public deployments. The industry problem is not adopting AI: it is sustaining it.
- RAND cites estimates that more than 80% of AI projects fail, twice the rate of technology projects without AI, and puts a non-technical cause first: stakeholders misunderstand, or miscommunicate, what problem needs solving.
- The most frequent cause is not technical: nobody defined what a correct answer was before starting. Without that criterion, "it works" is an opinion and the discussion never closes.
- The second is cost per query discovered too late. A pilot at a hundred queries a day costs nothing; the same system at a hundred thousand changes the conversation with finance.
These numbers are not ours
They come from the market, each with a study behind it, and they explain why most implementations disappoint.
95%of AI pilots return nothing measurable — MIT, 2025
80%+of AI projects fail, by estimates RAND cites, 2024
2×the failure rate of a technology project without AI — RAND, 2024
Seven ways to lose the project
- Nobody defined what a correct answer is. Without a set of real questions and their expected answers, every review is a matter of taste. The project does not die: it stalls in an endless round of tweaks.
- Cost per query showed up at the end. The pilot cost pennies. In production, with full context and ten times the volume, the monthly model bill exceeds whatever headcount it saved.
- Retrieval was bad and the model got blamed. If the passage that mattered never reached the prompt, no model can answer well. It is the most common failure and the easiest to mistake for hallucination.
- The system never learned to say it does not know. An assistant that always answers with the same confidence means the first serious error destroys months of accumulated trust.
- There was no owner after handover. The vendor delivered, the internal team had other priorities, and the system aged on its own. The document base changed and retrieval degraded with nobody watching.
- A process nobody had organised got automated. AI on top of a chaotic process produces chaos faster. If the rules are not clear to a person, the model will not infer them.
- A demo got bought instead of a system. Building the demo is a weekend. Sustaining it takes error handling, spend limits, traces, permissions and somebody on call. That is what is almost never in the quote.
The model is almost never the problem
When a system answers badly, the reflex is to swap models. Sometimes it helps; it usually masks the symptom. In most cases that reach a diagnostic, the model was receiving incomplete, contradictory or plainly wrong information, and answered with what it had.
Separating the two is cheap and takes an afternoon. Take thirty real questions, check by hand whether the correct passage appeared in what was retrieved, and the failure point becomes obvious. If retrieval misses, the work is in chunking, reranking and hybrid search. If retrieval hits and the answer is still wrong, then it is the prompt or the model.
Skipping that step is expensive: teams have switched model providers three times chasing a problem that lived in how the documents were split.
Six questions that reveal whether a project is on track
- Is there a test set of questions with their correct answers? How many?
- What is retrieval recall@k today, and what was it three months ago?
- What does one query cost, and what would it cost at ten times current volume?
- What does the system do when it cannot find the answer? Does it say so, or improvise?
- Who reviews, monthly, the questions the assistant could not answer?
- How long does a prompt change take to reach production, and who approves it?
What we do
- Project rescue The system is already in production and answers badly. We measure it and determine what to fix.
- AI consulting Architecture, evaluation criteria and cost per query before writing code.
- RAG systems Chunking, reranking and hybrid search, evaluated with recall@k and NDCG.
- AI chatbots and assistants With the evaluation set run before it ever reaches production.
Frequently asked questions
What percentage of AI projects fail?
MIT measures that 95% of AI pilots produce no measurable return ("The GenAI Divide", 2025), and RAND cites estimates that more than 80% of AI projects fail, twice the rate of technology projects without AI (2024). What is most useful in RAND is not the headline but the causes: the first is misunderstanding, or miscommunicating, what problem AI is meant to solve. The bottleneck sits between pilot and operation, not at initial adoption.
What is the most common reason a chatbot answers badly?
Retrieval, not the model. If the passage holding the answer never reached the prompt, the model answers with what it has and that reads as a hallucination. Before swapping models, review thirty real questions by hand and check whether the correct content was among what was retrieved.
Can a failed AI project be rescued?
In most cases yes, and it costs less than starting over. What gets salvaged is usually the integration, the permissions and the domain knowledge, which is where the time went. What gets rebuilt is usually document chunking, the retrieval strategy and error handling. The prior diagnostic is what separates a rescue from a rewrite in disguise.
How long does it take to move a pilot into production?
Four to eight weeks when the pilot was built well, and that work is almost never about the model: it is the evaluation set, per-user permissions, error handling, cost control, traces and an operating plan. When the pilot was built without any of those pieces, the route to production looks more like building it again with what the first attempt taught.
How do I stop it happening again?
Define the acceptance criterion before starting. Thirty to fifty real questions with their correct answers, written by someone who knows the business, and two target numbers: how often the system finds the information that mattered, and how often the answer stays inside that information. With that, any change can be evaluated in hours instead of argued about in meetings.
Written by
Quarl engineering team
We are an AI engineering team in Medellín. We build and run systems that reach production — automations, assistants and agents — and for three years we ran an assistant for two million users across ten countries.
See how we work →Want to apply this in your company?
In fifteen minutes we tell you whether it is worth it, free and with no pitch. The proposal arrives within 48 hours.