What embedded AI delivery uncovers in the workflow

When you sit beside the people who'll actually use an AI system, you start seeing things the brief never had in it. Someone checks a field against a spreadsheet before they'll trust it. A submission sits there until one particular person replies. An operator copies an identifier into a private note because the application won't show it again. (What I'm describing is a composite of patterns, not one engagement.) The brief came from policy and the process people could explain in a meeting, so it was never going to capture the calls they make under time pressure.

So early in delivery I'd follow a handful of tasks from arrival to done, with the operator using the systems and sources they normally use, and note where they pause, switch tools, ask someone, repeat a check or leave a field blank for later. You don't need names, record values or commercial details for this, just the type of source, the role and the decision. Then compare it with the brief. Some gaps are defects in the existing process and some are controls the documentation missed, and you want to know which is which before automation makes either one faster.

Distrusted fields are the ones that catch AI features out. If the team always checks a field against a dated export or another system, a feature reading the official field will look inconsistent to them even when it's following the documented rule, and telling people to trust the model won't settle a source disagreement. I'd write down the disputed field, which source people prefer and why, and how often the difference matters, then decide whether to fix the source, show freshness and provenance, or require confirmation. Don't let the model quietly blend conflicting values. Show which source it used and when it was retrieved.

Exceptions and approvals need the same attention. If the assistant can't tell whether a record is waiting, failed or deliberately paused, it'll recommend the ordinary next step at the wrong time, so recurring exceptions are worth turning into named states or reason codes. For each approval, ask what decision is being made, who's authorised and whether it's recorded. If nobody can explain a control, don't remove it as part of a software change. Give it an owner and schedule the decision. The AI can prepare the evidence and route the item, but the interface has to keep its recommendation separate from the person's actual approval, along with who approved it and when.

Notes on their own don't change the product, so I'd turn each material observation into a confirmed requirement, an open decision, a defect or an evaluation case, with its source and how confident you are in it ("observed once" carries less weight than "confirmed by the process owner"). Before widening the release, I'd check:

• Which sources do users still verify outside the application?
• Which exceptions still live in private lists or comments?
• Which approvals have no named rule or fallback owner?
• Which AI outputs can't be traced back to current evidence?

Think about what the brief never had in it: the field someone checks against a spreadsheet, the submission waiting on one person's reply, the identifier copied into a private note. Those are the things an AI system built from the brief alone will get wrong, which is why I'd want them written down as requirements, decisions, defects or evaluation cases before the release widens. From that list, I'd pick the one unresolved item most likely to cause a wrong action and settle its owner, its state and how it gets verified before adding another automated step, even if the fix turns out to be a small workflow change.