Article chapter 08 of 08
Run a regular operating review
From Budgeting and governing an AI workflow after the prototype
In early operation I'd run this monthly, or at whatever interval suits the workflow, and adjust it once things settle down. Get the business, technical and source owners in the room looking at the same record, instead of three separate reports on cost, quality and risk.
Go through:
- completed volume and backlog by workflow state
- cost per completed unit and total spend against the assumptions
- model, infrastructure and supplier charges
- review time, escalation reasons and queue age
- failed, uncertain and corrected runs
- evaluation results for any material system changes
- source freshness, access and processing failures
- incidents, complaints and requested corrections
- upcoming supplier, policy or source changes
- decisions, owners and due dates
Reconcile the bills against your internal usage records. Gaps can come from retries, test environments, untagged projects or a supplier reporting late. Every production request should carry enough metadata to tie it to a workflow and environment, without putting any private content into billing labels.
Check the workflow is still doing what it was approved to do. Users find adjacent tasks and start relying on outputs the review policy was never designed to cover. When that happens, either stop that use or assess it as a new workflow with its own boundary, evidence and owner.
I'd finish each review with operating decisions instead of an overall health score: carry on under the current policy, change a named control, fund more review capacity, narrow the accepted inputs, run a specific evaluation, pause one action, or retire the workflow. Write down when each decision gets checked again.
Before you approve the next budget period, pick a sample of completed, failed and corrected cases and trace their full cost and evidence. Can you connect the supplier usage, review records, source work and support activity back to the same unit? If you can't finish that trace, fix the records first, before anyone makes a confident claim about savings.
The prototype budget had a model, some development time and a data store. Once the workflow is running every day, the model call is one line in a cost that's mostly review, exceptions, failed runs, source upkeep and the people who own each of those. I'd budget the whole path from the trigger to the finished state, give each of those pieces a named owner, and swap the assumptions for measured values as soon as controlled use starts. If that sample trace can't be finished, that's where I'd start, because the budget and the governance both depend on those records.