How I decide whether an AI idea deserves a build

A demo shows the best moment: a document goes in and a polished answer comes out. A product has to handle everything around that, like checking access, dealing with missing information, recording corrections and recovering when a later step fails. So I write the task out with a clear start and end. If the feature drafts a reply, the task isn't done when text appears. It's done when the reply has been checked, approved, sent and attached to the right record. Written that way, the model call often turns out to be a small part of the build next to identity, retrieval, the interface, audit records, exception queues and support tools.

Then I look for a realistic way to catch errors. "A human will review it" doesn't tell me much. Which human, what will they see, and how long can they spend on each one? For a summary they might need links to the passages behind each claim. For extraction, the source image beside each value. For classification, the category definitions and somewhere to send uncertain cases. Some errors only show up later, when the team receiving a misrouted request notices, or support notices a wrong account update, so their corrections need a way back to the original record. If nobody can catch a wrong result before it gets expensive, I'd narrow the feature or keep it away from that action.

Generation time is easy to show off, but review time gets spread across people and quietly drops out of the calculation. I'd use the current task as the baseline, with active handling time and waiting time measured separately, then run real cases through the new process and record:

• preparation time before the model can run
• model and retrieval time
• reviewer time, including checking sources
• correction and escalation time
• rework from failed or duplicate runs

The sample needs ordinary and difficult cases, since short clean inputs give you a number that won't hold up in normal use. If the new workflow moves work onto another team, that still counts.

I also compare the idea with at least one conventional change. A lot of AI ideas point at a real problem (bad search, missing fields, weak templates) that something simpler could fix. Could a better query find the records? Could a required field remove the need to infer a value? Could a rules engine route the stable cases and leave only the ambiguous ones for review? Both options should face the same cases and acceptance criteria. I wouldn't let an AI answer through just because it sounds reasonable when the simple rule has to be perfect.

So whether an AI idea deserves a build comes down to what happens around the demo: who'd catch a wrong result, what they'd check it against and whether the time saved is still there once review and correction are counted. Before implementation I'd put that in a one-page brief naming the user, the task, the current baseline, the proposal, the sources it needs, what errors would cost, how review works and what result would stop or reshape the build. The evaluation set goes with it, including inputs missing evidence and cases where the system should refuse to go ahead, and I'd keep measures like field accuracy, routing, unsupported claims, corrections, handling time and unresolved cases separate for now, because one combined score can hide the failure that matters. Then I run those cases manually, review time and corrections included, and that's usually enough to estimate the build or drop the idea.