A conversational assessment still needs deterministic scoring

A conversational interface can make an assessment much easier to get through. It can clarify a question, take an answer in the person's own words and ask for the detail that's missing. The trouble starts when the language model is also deciding the score, with no stable method behind it. So I'd keep the two apart: the conversation captures evidence, and a versioned rule set calculates the result. (This is a composite architecture. The actual rules depend on the assessment and need review by whoever is accountable for using it.)

Write the rules before you design the conversation: dimensions, allowed responses, weights, thresholds, exclusions and what happens with incomplete information. Give each rule a stable identifier and a version. If a response maps to an option worth three points, put that mapping in data or code you can test. If it's buried in a system prompt, a wording tweak can change the scoring without anyone noticing it was a methodology change.

The conversation layer should know which item is active, what evidence it needs and which responses the scoring contract accepts. It can phrase things naturally, but what it writes to the record is structured: the original response, the proposed mapping and, where a model mapped free text, the model output, reason code, prompt version and the exact option chosen. None of that needs to show up in the chat (nobody wants to read "mapped to option B"). When an answer doesn't give enough evidence, ask a follow-up or mark the item unresolved, because forcing everything into the nearest option gives you tidy data that's wrong. Ambiguous evidence, conflicting answers and high-consequence outcomes need an explicit route to a person, and it's worth limiting what personal information gets collected, since people tend to say more in a chat than on a form.

The scoring itself is ordinary code. Validate inputs against a schema, reject unknown option identifiers and rule versions, and decide on purpose whether a missing answer counts as zero, neutral or incomplete, otherwise the model's defaults decide for you. The output should carry the rule version, dimension scores, rules applied, unresolved items, threshold decisions and any manual overrides. When the methodology changes, publish a new rule version instead of editing the old one, with its effective date, approver, reason and migration decision, and version the question and mapping prompts as well. If someone makes a manual change, keep the previous value, the reason, who did it and when.

I'd build a small set of fixtures early: clear low and high cases, boundary values, incomplete and contradictory answers, irrelevant free text, and wording that could plausibly map to more than one option. Test the mapping separately from the scoring so you can tell whether a failure came from understanding the answer or applying the method, and rerun the scoring fixtures with a comparison report whenever the rules change.

So the release check I'd run on a conversational assessment is reproducing a result without the conversation. Take a completed assessment, throw away the generated narrative and run its stored scoring inputs through the recorded rule version, then compare the total, dimension scores and threshold decisions. If they don't match, the score is still depending on the model somewhere, so hold the result and fix the record or the engine before spending any more time on the chat. The conversation can collect the evidence however naturally it likes, as long as the score comes from a versioned rule set a reviewer can rerun and get the same answer.