20 August 2024 / AI governance

AI governance has entered the product backlog

The EU AI Act is now in force, with obligations arriving in stages. If you build product, I'd start by writing down where AI is actually used and who's responsible for each use.

Circular openings and railings in a pale concrete building.
Photo: Joel Filipe (opens in a new tab)

The EU Artificial Intelligence Act entered into force this month. Most of the requirements apply later, and working out how a particular use is classified needs proper legal and domain review. Product teams can still start the inventory now, though, including AI features that came bundled inside other software you already use.

I'd keep one record per use of AI, because a list of model vendors is too coarse. The same model can draft internal text in one workflow and influence a decision about a person in another, and those two uses have different users, data and consequences when they go wrong. Describe the job in plain language (summarising a support conversation, ranking items for review, extracting fields from a document) and then note where the output goes and whether it can trigger an action. The fields I'd want on each record:

  • product and feature
  • intended purpose and affected users
  • model or service, including the deployed version where available
  • input data and output data
  • whether a person reviews the result
  • connected tools or downstream actions
  • product owner and technical owner
  • current release status

The Act also uses defined roles such as provider and deployer, and you can't work out which one you are from the invoice or from the fact that someone else trained the model. So for each use I'd map who developed the system, who puts it into service, who operates it, who supplies the underlying model and who faces the end user. That tends to surface practical gaps early. Can you tell which version is running? Does the supplier tell you when behaviour changes? Can you keep the logs you'd need for an investigation, and does the contract actually allow the data you're sending?

A short screening step is there to get each use in front of the right reviewers. It can't give you the legal classification on its own. Ask whether it's used around employment, education, access to important services, biometrics, safety, law enforcement or another sensitive area, whether it ranks or decides something about a person, and whether someone can change the result before it takes effect. If an answer is uncertain, mark it uncertain and give it to someone. Please don't pick the low-risk option just so the form can be submitted.

Evidence is much cheaper to collect in the tickets while people still remember the decision. For an AI feature that means the intended use, known exclusions, evaluation cases and results, the human review design, data sources, release approval and rollback method, all linked to a specific version of the prompt, model and surrounding code. Things like "human oversight required" or "handled by policy" need turning into backlog items with an owner and an acceptance test, otherwise a tester has nothing to check.

A spreadsheet or small database is fine if the team will keep it up to date, with a review triggered by model changes, new data sources, new user groups or a change in purpose. So that's how I'd treat the AI Act as product backlog work: most obligations come later, but knowing where AI is used, who's responsible for each use and what evidence sits behind it is something you can start in the tickets now. I'd fill in one live use completely, assign every unknown to a named person and ask the reviewers to work from that record. Whatever they couldn't answer from it tells you which fields and backlog items to add next.

Got a take on this?

Share it on X. I read the replies.

All blogs