Here in The Netherlands we have this very populair website: [url]http://fok.nl[/url]. It's a big community with every kind of news, forums, weblogs and more. There are around 280.000 users registered, And even much more people are visiting this site every day.
A bit more than week ago, they started to move all their servers (during daytime) from one datacenter to another. While they did this, they also replaced a few old servers with new ones (1st WTF, doing this at the same time). From this time, the problems were starting.
The site came online again and seemed very fast. But it only lasted a hour till the site was unreachable: "Kernel Panic" on the fileserver. They placed a downtime-page online on another server, saying they couldn't reboot the server remote because they didn't wrote down to which powerplug the server is connected. (2nd wtf). Also everyone was already at home, and they don't wanted to go to the datacenter anymore that day, so a day passed.
Next day, they did the usual server maintaince. Replace some memory, look up in the logs and try to disable/change some settings. They put the site online again, went home and bam. Site offline again. This story repeated itself for a few days long. Everything was done everyday by only 1 guy, An unpaid volunteer.
Few days ago they finally tried to get help from DELL (where they got their servers from). I don't know exactly what he did but at least the server could now work for 10 hours straight! Hooray. But the problem wasn't over yet.
Now they came up with a *great* solution. They created a little script, which reboots this server from time to time (main wtf!). Yes, it works. Now the site is only around 2 / 3 times a day , 10 minutes offline. But it seems like they gave up on it and just keep this solution. They're even refusing most help.
So I guess we just have to deal with this maintaince page a few times a day:
Ofcourse there are many more WTF's, in this story and in the site itself. A little while ago a scriptkiddy also 'hacked' the server and got hands on all passwords. They were stored plaintext in the database! All kind of WTF you'll expect from a site with only 100 users, not 280.000.
Just, wtf..
BTW. Sorry for my bad English!