← ~/blog

Model Cards Nobody Reads and Audit Logs Nobody Checks

 /  ai  /  323 words

AI governance in most companies is a stack of artifacts that exist so someone can say they exist. The model card gets written once at launch, describing training data nobody has verified and limitations copied from another card. The audit log records every prediction faithfully into a bucket that no human query has ever touched. Compliance theater, running nightly.

The tell is asking a simple question: when this system misbehaves, what artifact will help us? A model card describing the model as of eight months and four fine tunes ago will not. Neither will terabytes of logs with no tooling to answer "show me every decision this model made about this person."

What separates governance that works from governance that decorates, in my experience, is whether the artifacts are load bearing for some process that already runs. A model card that the on call engineer actually opens during incidents, because it lists known failure modes and their workarounds, stays current, because stale entries burn the people who maintain it. An eval suite wired into CI that blocks deploys governs behavior daily, while a fairness assessment PDF governs nothing. A log becomes an audit log the first time someone builds the query that reconstructs a decision end to end, input, model version, features, output, override. Until that query exists and gets run, you have storage costs, not accountability.

If you own one of these programs, the cheapest honest test is a drill. Pick a real past decision, hand the artifacts to someone uninvolved, and time how long reconstruction takes. Hours means you have governance. "We would need to check with the team that left" means you have paperwork.

Regulation is coming for the paperwork version, and honestly, fair enough. The teams doing the real version will barely notice the audit. The teams doing theater will discover, at the worst possible moment, that a shelf of unread documents was never the same thing as control.