Hey

After someone wrote about ":(){ :|:& };:" and "rm -rf" on main post comments, I'd like to start this thread. What did you do yourself / cleanup after others?

I was hit by:

• 'kill <pid of ssh>' while working remote (looked at wrong one in the ps tree),
• 'shutdown'-ing / 'reboot'-ing while on remote shell by accident, thinking it was my box
  
• 'iptables -F'-lushing while INPUT was DROP by default
• mixing up partitions in software raid (noticed "rebuilding" too late)
• 'tar -zcf *' instead of 'tar -zcf archive *' when the first 'aaaaa' file is the most crucial one
• not noticing >> instead of << when running a command from history
• setting too aggressive brute-force logins blocking rules and missing password 3 times in a row next time when connecting (ip block for a day)
  

I cleaned up after others, that were hit by:

• forcing bash update before libc package (lame update scripts on slackware - beware updating by more than 1 version - swaret will not warn you)
  
• rm -rf /lib (I don't think we're in chroot anymore)

Fortunately I've had backups always. But now I know more than others. And you have to do it once - no matter how much you think of it and how careful you are - it will hit you. I know many people that did some of this things. You should expect you'll hit every little thing like this and think before that, how would you fix it :)

So what is your story? 

I once murdered a hobo.

I once accidentally removed /bin (meant to remove a subdir named bin, somehow a / got in there). This naturally left almost everything crippled. emerge refused to work with /bin/(ba)sh missing, for example, and of course, sash is also in /bin ... The fortunate thing was, I still had things like wget and scp, so I could scp off some /bin binaries from another ssh host I had access to until I had enough to get emerge to rebuild the rest.

Also, not linux, but I somehow once managed to practically destroy a box running win2k once. It was an (older-than-dirt) packard bell, and while just goofing around, I set up a cmd.exe forkbomb, which led to BSOD, which led to the computer never able to boot again (tried floppy, etc). I'm thinking, user-assisted "natural" causes?

I once wanted to do "kill -9 $$" to kill the current shell without writing to the history (some bad typos I didn't want to repeat in C-r). The only problem was, I did "kill -9 -1"...
 

I have done something pretty similar - forgetting to add the dot to:

 rm -rf ./*
 

viraptor:

• 'iptables -F'-lushing while INPUT was DROP by default

My friend (yes, really :-P) once did something similar, but backwards: he was setting up the firewall rules over ssh, and he set the policy to DROP (or maybe it was DENY in those days?) before setting any rules.  Fortunately the box was a minute or two's walk away, so he easily managed to fix it.  Didn't stop him from doing exactly the same thing 15 minutes later, though....

As for me, the most interesting thing I can remember was when I was trying to learn bash history substitution... as root.  I accidentally recalled an "rm *" command, but fortunately I was in /root and there weren't any particularly important files there. 

Just having acquired basic Unix skills, I tried to show a friend, with whom I was sharing the account, a stupid joke, involving a shell command. So I did "cat > name" and typed a simple echo command or something like that. However, "name" was the file where my friend had stored his nearly finalized essay in nroff (we're talking 1985 here) that was printing at that very moment. We could never recover the file, but it completed printing fine...

 

Another f*ck up, much later, was removing (or perhaps moving, I can't remember) an essential dynamic library, perhaps even libc. Suddenly all commands quit working. It was a headless box, so the only thing I could do was try to repair it using commands from /bin, since these still were functioning (good thing they don't rely on dynamic libraries)! Had I logged out or lost the connection, the box would have been rendered completely useless.

 

Another stupidity was installing DEC Ultrix (from tape!) on a DEC-station and answering "yes" to the question whether I really didn't want to change anything to my set-up. I thought it meant the current configuration of the work station, but in reality it referred to the default values just shown. So the installation procedure happily started to repartition my drive, losing 300Mb of files.

Deleted the partition table. Recoverable fortunately.

rm -fr . in my root instead of the broken copy of root. Ctrl+C wasn;t enough to stop the nuking of /lib and worse /etc. That meant a reinstall.

sudo apt-get dist-upgrade from Dapper to Edgy. Breaks X, and I hate aptitude so I ended up going down the reinstall route, ultimately back to Dapper. As much of an Ubuntu WTF as a me WTF really.

Not having up-to-date backups :D

Having sshd running but having forgotten to open port 22 in the firewall.

Not a WTF as such since it worked, but I needed to DL and burn a DVD iso without having the free disk space. So I wgetted it to a named pipe and burnt the pipe with growisofs.

Compiling a Linux kernel (back in the 2.0 days) and forgetting to include my root filesystem.

Two hours later, reconfiguring my kernel, I came across an option for 'loopback interface', and thinking "gee I don't need that." (was a hopeless newbie at the time.) Took me four weeks to find out why I couldn't log in when running multiuser.

Just four months ago, I did the exact same iptables fuckup.

Just two months ago, was working on the company's main developmestruction machine as one of the disks in the (Highpoint IDE) raid-1 set had failed. Unknown to me, Highpoint cards don't hide the physical disks that are part of the raid set. "Hey, there's two spare disks in this box. I'll just 'dd' the data over there...." and copied the failed disk onto the good one.

By the time I cleaned that up I gave my notice and found a better place to work which could actually afford backup.

In grad school I wrote simulation code on my laptop and then ran the simulations on a workstation in our laboratory. So I would routinely sftp the code from my laptop to the workstation and later sftp the simulation results back the other way.

One day I was in the middle of sending new code and getting back data. I sftp'ed, navigated to a data directory containing the results from several month-long simulations, and entered get *. I got confused when I listed the directory contents and found that all the files were zero bytes.

I had forgotten that I had already ssh'ed into the workstation and was trying to sftp the files back onto itself. Unfortunately sftp didn't issue any warning that I was doing something stupid and decided that resizing the files to zero bytes was the proper response.

Since I was not wanting to do dual-boot anymore (I could run Windows with VMware if needed to), I decided to format my Windows partition and use it as /home. Partition list: hda1 -> / hda2 -> /media (where I kept all my stuff) hda3 -> Windows partition Booted from a Live-CD and typed 'mkfs.reiserfs /dev/hda1' instead of 'mkfs.reiserfs /dev/hda3', so I essentially formated the Linux partition! :(

Nothing to do with Linux, but... 

I was messing around on my computer at my temp job at FirstUSA (old credit card company absorbed by BankOne, who was absorbed by Chase). I was playing around in the MS-DOS prompt when I noticed the deltree command. I decided to try it out on c:\temp\somerandomfolder so I started typing...

deltree /y c:\

And ya know, that forward-slash key is so close to the Enter key that they were both hit almost simultaneously...

AlpineR:

I had forgotten that I had already ssh'ed into the workstation and was trying to sftp the files back onto itself.

That reminded me of great log filtering command:

cat log.file | grep -v "worthless line text" > log.file

.... unfortunately file is opened for writing and truncated before cat is run. Don't try this at home :/

grep -v "worthless line text" < log.file > log.file

doesn't work either - what you're looking for is:

cat log.file | grep -v "..." | tee log.file > /dev/null

I've dropped a production database a couple times.

I also uninstalled openSSL once, on a new fedora box that was at a hosting facility.  Turns out nothing runs.  No new SSH sessions.  The package manager won't even run.  So then you try to wget the openssl package, only to discover that both wget and curl require openssl...

 

viraptor:

AlpineR:

I had forgotten that I had already ssh'ed into the workstation and was trying to sftp the files back onto itself.

That reminded me of great log filtering command:

cat log.file | grep -v "worthless line text" > log.file

.... unfortunately file is opened for writing and truncated before cat is run. Don't try this at home :/

grep -v "worthless line text" < log.file > log.file

doesn't work either - what you're looking for is:

cat log.file | grep -v "..." | tee log.file > /dev/null

 

Personally, I just output to temp.file, and then mv temp.file log.file once I'm sure everything worked as planned.  Better to play it safe.   

Worst I've done is accidentally rm -rf'ing /usr/lib32/. Could've been a lot worse, but it still wasn't fun to fix. I've also done "do-shit foo > foo" or something similar a few times, but I've luckily never lost any important files to that.

And how do you get a password wrong *three* times in a row? I've got a 20-character password for disk encryption that I have to enter at every boot, and two cats who very much like to sit on the keyboard to get my attention, and I've still never gotten it wrong even *twice* in a row.

On Windows, but one time when I was changing my password, I managed to do the SAME typo on both entries :o Good thing it was on a network where I could just ask the admin to reset it.

Entering the wrong password several times is also quite easy if you can't remember which password you use for the system you're trying to log in to. That has happened to me. Heck, some places online I've managed to forget my USERNAME, never mind my password!

Doing some file clean up; (okay this is pretty basic, no fancy IP tables or nuttin'):

 "rm *.bak *.tmp *       .log"  (extra spaces included for emphasis)
 

The Real WTF:

And how do you get a password wrong *three* times in a row? I've got a 20-character password for disk encryption that I have to enter at every boot, and two cats who very much like to sit on the keyboard to get my attention, and I've still never gotten it wrong even *twice* in a row.

1. Become a new sysadmin.
2. Get a card with password written down.
3. Discover, that it's random 16-chars string.
4. Try to login with ssh, where you don't even see stars for characters, with a keyboard that will or will not recognise a keypress - depending on sun, wind direction, accumulated dust, noise in the room....

Normal passwords? I'm writing my own mixed characters/numbers/height with eyes closed and hands tied behind ;)

rm * .bak

(same like marvin_rabbit)
 

removing or renaming (can't remember) /bin/sh

(had to reinstall the OS then)

tar cvf instead of tar xvf (destroying the tar file)

tar xvf instead of tar cvf (overwriting the files with older versions)

"scp x y z" instead of "scp x y z destination" (overwriting z)
 

-9 Killing an Informix database during recovery probably wasn't the best way to get it working again... (eventually, it was, since it forced me to reinstall it then)

But the by far worst one was when we had this Unix workstation (pretty expensive then) with the hard drive not securely mounted (physically, we didn't have the right screws), just inserted into the case; when we wanted to go to a customer with this machine, the hard drive (pretty expensive then, IIRC USD10K) fell out and was dead.
 

ammoQ:

(had to reinstall the OS then)

You couldn't've used a livecd to put one back? Even if your controller or filesystem is not supported by said livecd, you could probably figure out something like passing root=\dev\cdrom to the kernel when booting from your regular bootdrive and still be on your way in considerably less time than it would take to reinstall.

Minor hardware-related one.... during the course of pranking each other at our summer internship, my coworker and I discovered that a DEC Alpha terminal will lock up hard if the mouse is unplugged while it's on.

Mixing up data, swap, and boot partitions during a manual install.

Curiously, Linux didn't have any problem with formatting a swap partition as ext2 and formatting a data partition as swap -- nor even mounting them.  Or reformatting them while mounted.  Only when I ran out of space during the install did I notice anything was wrong.  It's not often you see a df listing of -64Z free.  And twenty gigs of swap was certainly not what I wanted.

 I'd made a big list of what all the partitions should be then built them in the wrong order.  Guhhhhh.  That could have been ugly.

Back in the '80s working on a dual-floppy PC, I typed

DOS is not UNIX.