Secrets in Env Vars: A Love Story That Ends in a Git History

Ran a secret scanner against our own repos on a slow Friday, expecting nothing, because we use a vault and we are professionals. Found a live AWS key that had been sitting in git history for 14 months.

The commit message was the best part. "quick fix, will move creds to vault later." That commit was mine. I have no memory of it. Past me committed a credential to get unblocked and present me got to rotate it during what should have been a quiet afternoon.

The thing about secrets in git is that deleting the file does not delete the secret. It lives in history forever, and anyone with clone access to the repo has it, including that contractor from two years ago and every laptop that ever pulled main. Rewriting history with filter-repo is possible and miserable and does not un-ring the bell for clones that already exist. Rotation is the only real remediation, which is why rotation needs to be cheap, because if rotating a key is a two day project, nobody rotates and everybody prays.

What we actually run now, in order of value delivered. Pre-commit hooks with a scanner so secrets get caught before they exist in history at all. The same scan in CI as a backstop, verified detectors only so we don't drown in false positives from test fixtures. Scheduled scans over full history quarterly, because the pre-commit era started at some point and everything before it is archaeology. And short lived credentials wherever the platform allows, since a key that expires in an hour is a much smaller prize than one from 14 months ago.

Env vars themselves are fine, by the way. The sin is not the env var, it is the file that populates it getting committed. Guard the file.