Every consulting engagement in the past two years has had an AI conversation. Sometimes the client comes in wanting to implement AI for a specific purpose that’s well-defined and appropriate. More often, there’s organizational pressure — from the board, from a competitor announcement, from fear of being left behind — to “do something with AI” without a clear picture of what problem it solves.
The most valuable thing an AI consultant can do for clients in this environment isn’t to help them implement AI. It’s to help them evaluate honestly whether AI is the right approach — and when it isn’t, to recommend what actually is.
The Substitution Test
The first question I ask when evaluating any AI use case: could a non-AI solution solve this problem effectively? Specifically: could a well-designed rule engine, a better database query, a cleaner UI, or a simple automation script accomplish what the AI is supposed to do?
If the answer is yes, the AI implementation is probably the wrong choice. Not because AI is bad, but because simpler solutions are easier to build, easier to debug, cheaper to operate, and more predictable in their behavior. The explainability of a rule engine — you can point to the exact condition that produced the output — is a feature that AI systems lack and that matters in regulated industries, audit-heavy environments, and anywhere that “the model said so” isn’t an acceptable justification.
The use cases where this test most commonly reveals that AI is wrong:
Document routing based on content type. “We want AI to read incoming emails and route them to the right department.” In practice, the routing logic is usually based on a set of detectable keywords and patterns. A well-maintained rule engine routes correctly 95% of the time. An LLM routes correctly 95% of the time but costs more, adds latency, occasionally produces confidently wrong results, and is harder to audit when routing goes wrong.
Data validation and formatting. “We want AI to clean and normalize our customer data.” The data quality issues are usually systematic — inconsistent date formats, missing required fields, non-standard address abbreviations. A validation pipeline that checks for and corrects known patterns is faster, cheaper, and doesn’t hallucinate.
FAQ and knowledge base responses. “We want a chatbot to answer customer questions.” If the question space is bounded and the answers are deterministic — product specifications, return policies, shipping timelines — a traditional search-and-retrieval system over a well-maintained knowledge base is more reliable than an LLM, because the LLM may generate plausible-sounding wrong answers for questions slightly outside its training.
Data Quality Is the Prerequisite Problem
Every AI implementation depends on data quality. LLMs trained on poor data produce poor outputs. RAG systems over disorganized, inconsistent, or outdated documents retrieve irrelevant chunks and generate wrong answers. Recommendation systems trained on incomplete behavioral data make poor recommendations.
Before investing in AI capability, assess the state of the underlying data:
- Is the data collected consistently? Same formats, same schema versions, same units?
- Is it complete? Are there significant gaps in historical records?
- Is it accurate? Have the definitions changed over time in ways that affect comparability?
- Is it timely? How quickly does new data become available for model input?
In most small and mid-size business environments, the data infrastructure isn’t ready for meaningful AI. The CRM has five years of inconsistently structured customer records. The ERP exports don’t include key fields. The operational logs aren’t structured. The business intelligence reports are built on spreadsheet exports, not queryable databases.
Fixing these problems — which falls under data engineering and analytics — is more valuable than any AI implementation that builds on poor foundations. The data infrastructure investment pays dividends beyond AI too: better reporting, faster operations, cleaner integrations.
The honest conversation: “You want AI to give you better customer insights. Before we build that, let’s look at your customer data. Here’s what it would take to get the data quality to a point where the AI outputs would be reliable.”
Automation Without AI
A significant category of “AI” requests turns out to be automation requests. The customer wants their process to run with less manual intervention. They’ve heard that AI can automate things. What they actually need is workflow automation — Zapier, Make, or custom scripts — not machine learning.
Process automation is a well-solved problem for a large class of business workflows: trigger on event, transform data, call API, send notification, update record. This handles the bulk of what businesses want when they say “automate this.” It’s deterministic, auditable, debuggable, and cheap.
Where AI legitimately adds value over pure automation: cases where the transformation step involves unstructured input that can’t be handled with rules. Extracting structured data from free-text — pulling invoice details from a PDF invoice image, categorizing support tickets from natural language descriptions, summarizing long documents into key points — these are tasks where the unstructured-to-structured gap requires something more capable than a rule engine.
The design pattern that works: automation handles the workflow structure (trigger, routing, notification), AI handles specific transformation steps where unstructured input can’t be handled otherwise. Keep the AI component as isolated and bounded as possible.
The Maintenance Cost Nobody Mentions
AI implementations have ongoing maintenance costs that are often invisible in the initial project evaluation.
LLM API costs change as providers reprice models. The model version you build against gets deprecated and requires migration. The outputs shift with model updates in subtle ways that may not be detected until a user complains. The evaluation pipeline you need to catch regressions requires ongoing attention.
Fine-tuned models require periodic retraining as the underlying domain changes. A customer support model trained on last year’s product documentation produces increasingly wrong answers as the product evolves.
RAG systems require document collection maintenance — someone has to ensure that the knowledge base stays current, that outdated documents are removed, that new content is ingested promptly.
These aren’t reasons to avoid AI. They’re costs to account for in the project evaluation. A project that’s economically justified when implementation cost is considered may not be justified when ongoing maintenance is included. Evaluating honestly means including the full lifecycle cost.
When AI Is Genuinely the Right Answer
To be clear: there are many cases where AI is the right approach, and the right implementation creates meaningful value.
Tasks where AI has genuine comparative advantage over alternatives:
- Unstructured text understanding and extraction (documents, emails, transcripts)
- Content generation where quality variability is acceptable and human review is applied
- Pattern recognition in image or audio data
- Personalization at scale where the preference space is too large for explicit rules
- Complex decision support where the relevant factors are numerous and relationships are non-linear
The key characteristics: unstructured input, pattern matching at scale, and cases where the cost of being occasionally wrong is acceptable or where human review catches failures before they matter.
The distinguishing question: “What is the cost of a wrong answer from this system?” If the cost is high — a medical recommendation, a loan approval, an automated infrastructure action — the human review step is non-negotiable and the system needs explicit handling for the cases it’s uncertain about. If the cost is low — a content recommendation, a categorization that gets reviewed anyway — the higher error rate of AI systems is acceptable.
Our AI consulting and implementation practice has declined engagements where the honest evaluation was that AI wasn’t the right approach. That’s the only way to build a track record of implementations that actually work. The alternative — building AI into every engagement regardless of fit — produces projects that don’t deliver the expected value and organizations that learn to distrust AI recommendations.
If you’re evaluating whether an AI use case is genuine, start with the substitution test: could a simpler solution solve this? If yes, build the simpler solution. If no — if there’s a genuine gap that requires AI capability — then the next question is whether the data quality and maintenance capacity exist to support it.