Article
Designing an auditable AI capability assessment
If you let an AI run the conversation in an assessment, I'd keep it well away from the score. Here's how I'd set up the rules, fixtures and audit record so a reviewer can work out exactly how a result was reached.
Picture an assessment where the respondent chats with an AI and then gets a score and a polished report in one go. If someone asks why they got that score, nobody can really say. The respondent can't see which answers counted, a reviewer can't redo the calculation, and if someone tweaks the prompt next week, the interview and the score can both change without anything useful being written down.
The way I'd build it is to give the AI a smaller job. It runs the conversation, picks out what looks like evidence and asks for clarification when an answer is fuzzy. Then a separate, deterministic scoring service takes the confirmed answers and applies a versioned ruleset to them. The report carries the rule version, the evidence references and a record of the calculation, so anyone with access can reproduce it.
I'm describing an implementation pattern here. It isn't a write-up of a particular assessment or engagement. The actual questions and rules still need someone who knows the subject to review them. The architecture just keeps that methodology visible in the software where people can check it.
On X
Got a take on this?
Share it on X. I read the replies.