Priceless re: servers down

[Bertha Venation]

http://thedailywtf.com/Articles/Im-Sure-You-Can-Deal.aspx

Not too long ago, there was a small furor in the local media about a major disaster at The State's Technology Services Division. The details were a bit sketchy – mostly because The State was “unable to comment on an ongoing investigation” – but what was reported was that, for two full days, employees of The State were unable to logon to their computers or access email, and that this caused business within The State to grind to a halt.

As the “investigation” carried on, the media lost interest in the story and moved on to more newsworthy stories like who Paris Hilton was partying with last weekend. Fortunately for us, a certain employee of The State named J.N. works in the Technology Services Division and decided to share what really was behind those fateful days.

When employees of The State came in to work following a three day weekend, they found their workstations overloaded with "cannot logon" and "Exchange communication" error messages. The Network Services folks had it even worse: the server room was a sweltering 109° Fahrenheit and filled with dead or dying servers.

At first, everyone had assumed that the Primary A/C, the Secondary A/C, and the Tertiary A/C had all managed to fail at once. But after cycling the power, the A/Cs all fired up and brought the room back to a cool 64°. At the time, the “why” wasn’t so important: the network administrators had to figure out how to bring online the four Exchange Services, six Domain Controllers, a few Sun servers, and the entire State Tax Commission’s server farm. Out of all of the downed servers, those were the only ones that did not come back to life upon a restart.

They worked day and night to order new equipment, build new servers, and restore everything from back-up. Countless overtime hours and nearly two hundred thousand dollars in equipment costs later, they managed to bring everything back online. When the Exchange servers were finally restored, the following email finally made its way to everyone's inbox, conveniently answering the “why”

From: ----- -----------
To: IT Department
Re: A/C constantly running.

To whom it may concern,

I came in today (Monday) to finish up a project I was working
on before our big meeting with the State ----- Commission tomorrow,
and I noticed that there were three or four large air conditioners
running the entire time I was here. Since it's a three day weekend,
no one is around, why do we need to have the A/C running 24/7?

With all the power that all those big computers in that room use, I
doubt it is really eco-friendly to run those big units at the same
time. And all computers have cooling fans anyway, so why put the A/C
for the building in that room?

I got a keycard from desk and shut off the
A/C units. I'm sure you guys can deal with it being warm for an hour
or two when you come in tomorrow morning.

In the future, let's try to be a little more conscientious of our
energy usage!

Thanks,
-----

As for the employee who sent it, he decided to take an early retirement.

[mikeytherat]

1. On one of my IT gigs, there was a router at a remote site which kept going down.

When I went to the site to investigate, I found the router inside a closet with the steam pipes for the building. The temperature in the closet was over 140 degrees.

I moved the router out into the hallway (thankfully I had spools of Ethernet with me), and started figuring out where it could be placed.

mikey_the_rat

[madinmaryland]

2. EPIC FAIL!!

Being in IT, that one really hits home. I was setting up a small data center for a client 10 years ago, and the cleaning crew came in to clean up, and plugged a vacuum cleaner into the UPS system, and complete blew the system up! I think they had to replace all of the batteries in the system!

[Wapsie B]

3. For doing something like that he should've been forced into some kind of restitution.