The calm before the storm
A key we knew was high-risk but put off revoking because it was a hassle was abused just over a month later.
(Illustrative scenario, not a real incident)
In the forensic report, one line was written in bold:one key was “highest risk, still valid.”
It was also within the scope of what had been taken.
We all knew it was dangerous,but revoking it was a hassle.Several live services depended on it;the same keyhad been copied into many codebases and computers.
Revoke it, and everything goes offline.
So we decided:move things off it one by one first, then change the lock.
The month that followed was very quiet.So quiet that I almost believed it was fine.
Then, late one night,someone took that keyand hammered the cloud's AI services all night long.The next morning, the suspension notice arrived.Every service relying on it stopped at once.
That was just over a month after the breach.
I used to tell people: don't shoot at every shadow.Later I learned that with hackers,you have to turn that around:he will come back,and right when you breathe a sigh of relief.
It's not that we weren't prepared.We were just quietly hoping our luck would hold:it's such a hassle to change, surely it can wait a bit longer.
Every time we count on luck,we're helping the hacker gather strength.
The most painful part: after the suspension, deleting that keycost nothing at all.The very same thing could have been done a month earlier.
The pain of downtime is a time you choose;the pain of suspension is a time someone else chooses.
Laozi said: “Plan the difficult while it is still easy; do the great while it is still small.”Do the hard thing while it is still easy.
Calm seas don't mean the rain won't come.The quiet you think you hearis sometimes just him waiting in line.
For a key you know has leaked, revoke it first and migrate after. The one that feels like too much hassle is the one to put on this week's calendar, so you choose when the downtime happens.