Skip to main content

Man Accidentally Deletes His Entire Company With One Wrong Command

Man Accidentally Deletes His Entire Company With One Wrong Command

SysAdmin deletes server
SysAdmins often have this nightmare when they run the dreadful and deadly command ‘rm -rf /’ as root. How horrifying!
If you didn’t know already, / represents root. And running ‘rm -rf /’ will delete root directory and all of its content. In Linux file hierarchy, root contains everything. Deleting root means your system is gone, forever.
No wonder this is compared to drunken driving in the Linux world.

Sh*t happens

But shit happens in the IT world. And apparently it happened with this hapless SysAdmin Marco Marsala who runs a web hosting company serving over 1500 customers.
As per the question posted on Serverfault few days back, Marsala tried to run a Bash script that had the following command in it: rm -rf {foo}/{bar}. But it turned out to be ‘rm -rf /’ due to undefined variables and the inevitable happened.
In Marsala’s own words:
I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.
All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).
How I can recover from a rm -rf / now in a timely manner?
Oh, poor guy!! What did you just do?
sudo rm -rf funny Linux command

What next?

What next? This is what Marsala wanted to know. Is there a way to recover from ‘rm -rf /’?
But chances of recovering all the data from a rm -rf / are thin. No wonder, this post started getting sarcastic (but honest) comments like:
If you really don’t have any backups I am sorry to say but you just nuked your entire company
Another one went like:
You’re going out of business. You don’t need technical advice, you need to call your lawyer.
Few people suggested to shutdown everything, don’t overwrite anything and use data recovery tools to get at least some data back.
And it seems like, it did work to a larger extent for Marsala as he did mention “luckily we recovered almost all data” later on.

Lessons to learn

As some people are speculating that it’s a hoax, there are still few lessons to learn for all of us.
  • Backup everything. If it’s a professional server, have multiple, offline backups
  • Don’t use a random tool or script from the internet and use it on a production machine directly
  • Have test machines identical to that of production for testing out new stuff without risking the production system
Anything to add to this scary incident?

Comments

Popular posts from this blog

fix idm integration on chrome

Chrome Browser Integration I do not see IDM extension in Chrome extensions list. How can I install it?  How to configure IDM extension for Chrome? Please note that all IDM extensions that can be found in Google Store are fake and should not be used. You need to install IDM extension manually from IDM installation folder. Read in step 2 how to do it . 1. Please update IDM to the latest version by using  "IDM Help->Check for updates..."  menu item 2.  I don't see  "IDM Integration module"  extension in the list of extensions in  Chrome . How can I install it? Press on  Chrome  menu ( arrow 1  on the image), select  "Settings"  menu item ( arrow 2  on the image) and then select  "Extensions"  tab ( arrow 3  on the image). After this open IDM installation folder ( "C:\Program Files (x86)\Internet Download Manager"  by default,  arrow 4  on the image) and drag and drop  "IDMGCExt.crx"  ( arrow 5  on the image) file int

sxhkd volume andbrightness config for dwm on void

xbps-install  sxhkd ------------ mkdir .config/sxhkd cd .config/sxhkd nano/vim sxhkdrc -------------------------------- XF86AudioRaiseVolume         amixer -c 1 -- sset Master 2db+ XF86AudioLowerVolume         amixer -c 1 -- sset Master 2db- XF86AudioMute         amixer -c 1 -- sset Master toggle alt + shift + Escape         pkill -USR1 -x sxhkd XF86MonBrightnessUp          xbacklight -inc 20 XF86MonBrightnessDown          xbacklight -dec 20 ------------------------------------------------------------- amixer -c card_no -- sset Interface volume run alsamixer to find card no and interface names xbps-install -S git git clone https://git.suckless.org/dwm xbps-install -S base-devel libX11-devel libXft-devel libXinerama-devel  vim config.mk # FREETYPEINC = ${X11INC}/freetype2 #comment for non-bsd make clean install   cp config.def.h config.h vim config.h xbps-install -S font-symbola #for emoji on statusbar support     void audio config xbps-i

Hidden Wiki

Welcome to The Hidden Wiki New hidden wiki url 2015 http://zqktlwi4fecvo6ri.onion Add it to bookmarks and spread it!!! Editor's picks Bored? Pick a random page from the article index and replace one of these slots with it. The Matrix - Very nice to read. How to Exit the Matrix - Learn how to Protect yourself and your rights, online and off. Verifying PGP signatures - A short and simple how-to guide. In Praise Of Hawala - Anonymous informal value transfer system. Volunteer Here are five different things that you can help us out with. Plunder other hidden service lists for links and place them here! File the SnapBBSIndex links wherever they go. Set external links to HTTPS where available, good certificate, and same content. Care to start recording onionland's history? Check out Onionland's Museum Perform Dead Services Duties. Introduction Points Ahmia.fi - Clearnet search engine for Tor Hidden Services (allows you