P(doom) Is a Distraction, P(oops) Is Tonight: Practical AI Risk for Normal Teams

The public AI risk conversation is dominated by extinction scenarios, and whatever you think of those debates, they have a side effect on normal engineering teams: risk feels like someone else's philosophical problem instead of this sprint's engineering problem. Meanwhile the risks with probability approaching one are already in your backlog, unticketed.

Call it P(oops), the chance your AI feature causes a mundane, expensive mess this quarter. The catalog is unglamorous. The support bot that confidently invents a refund policy, which a court or a regulator may decide you now honor. The summarizer that drops the one negation that mattered, "not approved" becoming "approved" in an executive digest. The agent with API access that does exactly what a malicious document told it to. The model whose accuracy quietly rotted as the world drifted away from its training data, discovered only when a customer complains louder than your monitoring, because you have no monitoring, because it is "AI" and not "software."

That last clause is the root cause. Teams that would never ship a payment service without alerts, rollbacks, and an on call rotation ship model driven features with none of the above, as if statistical software were exempt from software discipline. It is the opposite of exempt. Deterministic code fails loudly and consistently. Models fail plausibly, which is worse.

The practical program is not exotic. Write down what the feature must never do, and test for it in CI like any regression. Log inputs and outputs so incidents are reconstructable. Monitor quality in production, sampled human review counts. Bound the blast radius, confirmation on irreversible actions, least privilege on tools. And rehearse the rollback.

None of this resolves the doom debates. It just means that whichever future arrives, you shipped software instead of a liability with a demo.