E-Mail Outage Forces White House to Operate the Oldfangled Way

[RamboLiberal]

Source: Washington Post

The guy on the computer help line at the White House seemed a bit harried yesterday afternoon.

Shortly after the workweek began, the tech-savvy Obama administration was hit with a mysterious "server outage" that shut down all incoming and outgoing e-mail for more than eight hours, forcing aides to resort to old-fashioned phone calls and face-to-face conversation.

"We're getting a few calls," the worker deadpanned after answering phone calls from e-mail-starved employees at 1600 Pennsylvania Ave.

How is it possible that the e-mail system for the White House could go down that long, a caller asked? Press secretary Robert Gibbs had just apologized on live television for the lack of any e-mail contact.

"We still don't know," the help-line guy said, adding that two e-mail servers had been rebooted but that two others remained mysteriously down, with no immediate explanation.



Read more: http://www.washingtonpost.com/wp-dyn/content/article/2009/01/26/AR2009012602087.html?hpid=topnews

[jwirr]

1. Oh, oh could this be the missing W revenge? Why do we have such

an inadequate computer system to run our nation with?

[bluesmail]

2. It's not so mysterious if you think about it in context with KKKRovian

operatives. Messing with President Obama. Sure you want to do that? Edited for grammar and spelling and just in case-iness.

[boppers]

3. FOUR FREAKING SERVERS?

W. T. F.

It's the white house, not "Bill's Bait Shop".

[merwin]

4. 4 servers is plenty for what they're doing... they're just edge servers delivering mail.

At least, they are by the way it's being described. I'm guessing that it wasn't the 4 e-mail servers that went down, as that would be too much of a coincidence... probably a firewall down or a cut line.

[boppers]

5. It's still a major facepalm.

For something like the Whitehouse, you'd want at *least* 4 edge MX presence servers, in 4 physical locations, each with redundant inbound IP links, with different OS's, etc.

It's the office of the Whitehouse, their communications should be able to withstand a tactical nuke that takes out a whole data center, or even a whole city. That's the whole reason DARPA-net was created.

[kristopher]

6. maybe it is intentional and involves a review of vulnerabilities in the system? nt

[merwin]

11. I agree with that. The article says that the network was in shabby conditions when they found it.

That was probably one of those conditions :)

It could also be that one of the techs fucked up while they were migrating e-mail and setting up the new accounts. I've been known to click the wrong button a time or two, however never to the extent of taking down 4 e-mail servers. heh.

[NBachers]

23. Yes, I can remember throwing the wrong circuit breaker

in a panel where the labeling was messed up. As soon as I tripped it and heard a frantic "Oh My God!" in a nearby cubicle, I knew I'd F'd up.

[crikkett]

27. that was my 1st thought, firewalls. 2nd thought: microsoft servers.

Back in the bubble I helped disentangle a realtor's office from its reliance on a spaghetti network.

[merwin]

34. The chances of it being a Microsoft problem is slim, being that all 4 servers went down.

Plus, Exchange is a rock solid product, especially in a clustered environment.

[boppers]

35. Exchange is rock solid?

For varying values of "rock", and "solid", I'll agree. :)

PMDF on an open VMS cluster, now that's solid, but all it takes to make a Microsoft cluster insane is frequent rebooting, or clocks out of sync. :evilgrin:


edit: typo

[givemebackmycountry]

7. It's a joke - it should be the most sophisticated office in the world...

And instead it's like the freaking DMV.

[crikkett]

28. no worse! like a VA office.

[Lars39]

8. Do we need to send Elad, EarlG and Skinner to straighten things out?

One hand behind their back, blindfolded! :D

[lapfog_1]

9. Sigh...

As part of the team that originally connected the White House to the internet, let me say this for the record. Of just about all government facilities that I ever had a chance to run into, review, talk about or just hear about in my 10 year NASA career, the White House was the single most technologically backward facility in the national government. Seriously.

Not only that, but at the time, they had ZERO people allocated to IT. ZERO. Everyone was borrowed from some tech savvy agency, like NASA, or one of the DOE labs.

They simply never saw a big need for it (other than Al Gore, bless his heart), and congress never really funded it.

That said, by the time I left the government (and shortly before Clinton/Gore left), we had it more or less functioning with a web site, email servers, computers in the West Wing wire connected (no wireless back then, secure or not). And back up servers. And back up tape subsystems (that would be ME).

Lord knows what's happened in the last 9 years (actually about 10 since I had anything to do with it).

But it sounds like the Bushies fucked it all up, likely as not on purpose (handing the functions that they needed to that guy that got killed in a small plane crash and out of executive branch hands).

Obama needs to revamp the whole system.

Hey, I'm out of a job right now... I could use the "stimulus"... Seriously.

[merwin]

12. They're probably on Exchange 2003, and I would hope at least Server 2003.

Unless they're running a Unix system, which seems unlikely because of the Blackberry Enterprise Server (BES) that runs the blackberry communication.

[lapfog_1]

14. When I connected them to the Internet... way back when,

we had them running a POP3 server on a SGI PowerChallenge deskside server (mounted in a rack), a different one for the web.

I'm sure they have done a technology refresh since then, but I would have no idea if they moved to an Exchange Server or not.

I think the Barackberry would be a new thing, and I'm not even sure that's part of the White House environment (for example, I knew that there WAS a situation room somewhere in the White House, but I never got near it... it's run by DOD people and they do their own thing).

[TrogL]

17. Do you really want the White House email on that insecure a system?

Exchange is a problem waiting to happen. It also makes backup and recovery of individual emails nearly impossible.

[lapfog_1]

18. I know the issues with Exchange well...

for a brief moment in time I worked for a startup which had a method to backup and restore exchange email. That was their total business plan. I would not recommend exchange for the White House (or nearly any other business) EXCEPT it's become this standard. Which sucks.

But I SUSPECT that the Bushies put exchange servers in place (or rather, in the hands of that guy's company in Kentucky or where ever it was... you remember, the guy who also was getting all of the Ohio electronic tabulated vote sent to him BEFORE it went to the county election officials... the guy who just a few weeks ago died in a private plane crash).

Because it's a "business standard" now.

I hate exchange. I never lose email from my old clunky NetBSD POP3 server. Never, not once in some 20+ years of running it (moving it from hardware platform to hardware platform, upgrading the version of NetBSD... adding new and better spam killers, etc, etc).

But that's just me.

[merwin]

20. I doubt they're using Exchange as the public mail server, so there's no security issues.

They've most likely got some sort of UTM scanning for viruses (and logging in the CIA database of course:). That means there is no way into the Exchange server and no possibility of exploits.

And the E-mail database in Exchange is very robust. It's a fully transactional database with write-ahead logging to prevent corruption.

[jtrockville]

33. At some point (maybe) the WH switched from Lotus to MS

According to Perino the WH switched from Lotus to MS, but how much is a liar's word worth?
http://www.citizensforethics.org/node/27675

[Seldona]

21. You should apply. Seriously.

You sound like you know that the hell you are doing. If that system is that antiqued and useless, it should be a top priority.
Me? I always figured the President would be the most wired man in the world. He certainly SHOULD be.

[stillcool]

10. holy shit...

How is that possible? Anyone have any extra tinfoil?

[appleannie1]

13. The WH should have the best services in this country. WTF was the Bush Admin doing for 8 years?

[lapfog_1]

15. Sadly, that's never been true.

Antiquated would be the best description of the White House even before Bush came in. Doubt he did much to improve it.

[tanyev]

26. Very little of the Bush administration's work was done on WH servers.

Anything done there was for public show only. All their "important" communications were fed through RNC servers or contracted out.

[snot]

16. I thought I read or heard that as part of Bush's "cooperation" with the transition,

there had been specific discussion of how brand-new equipment would be installed for the Obama admin, and the Bush team let them pick what they wanted. I remember thinking hmmm, how convenient that all the old servers etc., together with any evidence they might contain, would be completely removed and replaced.

Now I can't help wondering about some kind of sabotage, which could come from a wide array of sources. I'd have expected the Obamanians to be pretty tech-prepared.

[happygoluckytoyou]

19. FIRST TIME D.C. SERVERS WERE EVER USED WITHOUT THE VIRTUAL SHREDDER TURNED ON ! ! !

[OnyxCollie]

22. Is this when bushco taps the White House?

:tinfoilhat: :hide: :tinfoilhat:

[dipsydoodle]

24. Oldfangled way

Two baked bean tins and a length of string ? :shrug:

[annabanana]

25. Check that smoking closet next to the old VP's office..

You remember.. the "electronics" that caught fire.

http://afp.google.com/article/ALeqM5jp5EwGNIs1P7UsrUSxH6VgR-ITDA


Maybe the emails were routed through there....

[johnfranks999]

29. White House E-mail Outage

Price Waterhouse Cooper and Carnegie-Mellon’s CyLab have recent surveys that show the senior executive class to be, basically, clueless regarding IT risk and its tie to overall enterprise (business) risk. Data breaches and thefts are due to a lagging business culture – and people aren’t getting the training they need. For example: Microsoft patched for this worm 4 months ago. As CIO, I’m constantly seeking things that work, in hopes that good ideas make their way back to me - check your local library: A book that is required reading is "I.T. WARS: Managing the Business-Technology Weave in the New Millennium." It also helps outside agencies understand your values and practices.
The author, David Scott, has an interview that is a great exposure: http://businessforum.com/DScott_02.html -
The book came to us as a tip from an intern who attended a course at University of Wisconsin, where the book is an MBA text. It has helped us to understand that, while various systems of security are important, no system can overcome laxity, ignorance, or deliberate intent to harm. Necessary is a sustained culture and awareness; an efficient prism through which every activity is viewed from a security perspective prior to action.
In the realm of risk, unmanaged possibilities become probabilities – read the book BEFORE you suffer a bad outcome – or propagate one.

[Jokerman]

30. Maybe they are down for "security" reasons.

Purely speculation but as an IT manager I know that if I had just taken over a system from someone I do not trust, I'd be scouring it for trojans and other "bombs" that may have been deliberately left behind.

Sometimes you tell the users that you don't know why a system is down when in reality, you may not want them to know the real reason.

:tinfoilhat:

[formercia]

31. Every internode should be fiber-optic

for security reasons, speed and EMP resistance. The only Copper should be for power.

For what they spend, it's peanuts.

[glinda]

32. They need to go over everything electrical, mechanical & computer related there

and update everything as well as check for bugs.