AI Product Development · Technology Strategy
When AI is the wrong answer
Three conditions that determine whether a workflow should use a language model, and what to build instead when they are not met.
- Author
- Jordan Dias
- Published
- Last updated
- Reading time
- 6 minutes
The question this answers
A model-backed feature has been requested and nobody can say whether it will pay for itself.
Key takeaways
- Three conditions must hold before evaluation work starts: language-driven task, verifiable output, and manual effort large enough to justify the running cost.
- Verification, not accuracy, is the binding constraint — an undetectable wrong answer is a design failure, not a prompt failure.
- When the conditions fail, structured data capture and validation rules solve the same problem with no variance and no inference cost.
- A fixed-scope evaluation that concludes against building is a successful evaluation.
The test
Most requests to add AI to a product arrive as a solution looking for a problem. A useful filter is to check three conditions before any evaluation work begins. All three must hold.
- The task is repetitive and driven by language or documents rather than by numbers and rules.
- A correct answer can be verified — by a person, a rule, or a downstream system that would reject a wrong one.
- The current manual effort is large enough that removing part of it justifies building and running the feature.
Why verification is the binding constraint
Accuracy is a distribution, not a guarantee. A feature that is right most of the time is only usable if being wrong is detectable and recoverable. Where a wrong answer flows silently into a customer record, a payment or a compliance obligation, the correct design is not a better prompt — it is a review step, and the review step is where most of the engineering effort belongs.
This is why classification and triage tasks succeed more often than autonomous decisions. Triage narrows a person's work; a decision replaces their judgement, and replacing judgement requires a level of confidence that is expensive to establish and easy to lose.
What to build instead
When the conditions fail, the underlying problem usually remains real. The alternatives are less interesting and more reliable: a structured form replacing free text, a validation rule at the point of entry, a scheduled job replacing a manual check, or a single system of record replacing three spreadsheets.
These carry no inference cost, no variance, and no explanation burden. When the workflow later becomes a genuine candidate for a model, structured data is exactly what makes the evaluation possible.
Evaluating before committing
Where the conditions do hold, the next step is a fixed-scope evaluation on real data — not a demo on curated examples. Record accuracy, latency and cost per run, and keep the failure cases. Failure cases determine the review design, so they are the most valuable output of the exercise.
An evaluation is worth writing down in a form a non-technical stakeholder can read. The shape below is enough.
{
"task": "classify inbound compliance queries",
"sample": 400,
"accuracy": 0.91,
"unrecoverable_errors": 3,
"median_latency_ms": 820,
"cost_per_run_usd": 0.004,
"decision": "build with human review on low-confidence band"
}Common questions
Does this mean AI should be avoided in regulated workflows?
No. It means the review step, retention rules and audit trail are part of the feature rather than additions to it. Regulated workflows often have clearer verification than unregulated ones, which makes them better candidates than they first appear.
How large should an evaluation be before a decision?
Large enough that the failure cases repeat. In practice a few hundred real records is usually sufficient to see the pattern; curated demo examples are not, at any volume.