One key, six places to change
Changing one key meant changing six places, so it never got changed.
(Illustrative scenario, not a real incident)
The security consultant said, “Keys should be rotated regularly.”The engineering lead nodded and started counting:
Production, staging,the deployment pipeline, the config files for two services,and a script on one colleague’s laptop.Six places. Miss one,and some service breaks in the middle of the night.
The conclusion: “Let’s do it next quarter.”
At this twenty-five-person logistics tech company,one payment keystayed in use, just like that, for four years.
Until one day,a contract engineer who had handled that keywas found to have a compromised laptop.
The company decided to rotate it at once.
It took the whole night.Production updated, the scheduled jobs broke;scheduled jobs fixed, they found an old serverwhere the reconciliation script was still using the old key.
Not rotating is a risk;rotating is a risk, too.The real problem underneath:no one knew where the key was used.
Sweat more in peacetime, bleed less in war.Emergency rotation hurts the most,because without practice,you can only search as you go.
- For every key, list every place that uses it.
- Have programs read the key from one place, not from copies.
- Pick a key that doesn’t matter much, and practice rotating it once.