The city deleted by mistake
An intern deleted the wrong cluster, and the backups had never been turned on.
(Illustrative scenario, not a real incident)
On a Friday morning, intern Xiao-Yu got a task:delete a test system nobody was using.
On screen, test and production looked almost the same.He picked one and pressed delete.No one took a second look for him.
A minute later, the alarms went off.What he had deleted was the live production system.Every customer website in the region went down with it.
The cloud provider said: once deleted, it can’t be recovered.They tried to restore from backupand found the backups had never been turned on.
Luckily, the customers’ data was still there.The team spent nearly an hourmoving the services back, one by one.
The post-mortem was honest:the intern’s account was powerful enough to delete production,and an action this dangerous had no one reviewing it.
What you see is one wrong click.What you don’t seeis a key that can tear down the whole city,handed to an apprentice still learning the trade.
Laozi said: “What is at rest is easy to hold.”Things are easiest to protect while they are still calm.
Apprentices make mistakes; that is what apprentices do.Letting one mistake tear down the whole cityis the master’s doing.
An extra lock on the delete buttonisn’t distrust of people.It protects the personwho, someday, will press the wrong one.
Use low-privilege accounts day to day. Irreversible actions like deletion should need a separate request and a reviewer. Backups shouldn’t just be on—restore from them for real at least once.