The test

Most requests to add AI to a product arrive as a solution looking for a problem. A useful filter is to check three conditions before any evaluation work begins. All three must hold.

  • The task is repetitive and driven by language or documents rather than by numbers and rules.
  • A correct answer can be verified — by a person, a rule, or a downstream system that would reject a wrong one.
  • The current manual effort is large enough that removing part of it justifies building and running the feature.

Why verification is the binding constraint

Accuracy is a distribution, not a guarantee. A feature that is right most of the time is only usable if being wrong is detectable and recoverable. Where a wrong answer flows silently into a customer record, a payment or a compliance obligation, the correct design is not a better prompt — it is a review step, and the review step is where most of the engineering effort belongs.

This is why classification and triage tasks succeed more often than autonomous decisions. Triage narrows a person's work; a decision replaces their judgement, and replacing judgement requires a level of confidence that is expensive to establish and easy to lose.

Practical note: If nobody in the workflow can tell a wrong answer from a right one within a few seconds, the feature needs a review step before it needs a model.

What to build instead

When the conditions fail, the underlying problem usually remains real. The alternatives are less interesting and more reliable: a structured form replacing free text, a validation rule at the point of entry, a scheduled job replacing a manual check, or a single system of record replacing three spreadsheets.

These carry no inference cost, no variance, and no explanation burden. When the workflow later becomes a genuine candidate for a model, structured data is exactly what makes the evaluation possible.

Evaluating before committing

Where the conditions do hold, the next step is a fixed-scope evaluation on real data — not a demo on curated examples. Record accuracy, latency and cost per run, and keep the failure cases. Failure cases determine the review design, so they are the most valuable output of the exercise.

An evaluation is worth writing down in a form a non-technical stakeholder can read. The shape below is enough.

Evaluation record

{
  "task": "classify inbound compliance queries",
  "sample": 400,
  "accuracy": 0.91,
  "unrecoverable_errors": 3,
  "median_latency_ms": 820,
  "cost_per_run_usd": 0.004,
  "decision": "build with human review on low-confidence band"
}