Maybe the folks that decided September should be "Disaster Preparedness Month" did some research beforehand. After all, here we are in September and we've definitely had our share of problems. Heat waves, hurricanes and now an unprecedented power outage in southern California that affected 5 million people. This may not seem like a big deal to other parts of the country but it was a pretty big deal here since it had never happened before.
We're actually headquartered in San Diego so we were definitely affected. On September 8th, about 3:40PM the lights went out, not a big surprise since we'd been in a heat wave blasting our A/C like everyone else. We just figured the power grid operators decided there was too much strain on the system and shut down some areas to get it under control. I was actually on the phone with one of our suppliers and fortunately I have a UPS on my desktop and phone. Our computer room is also UPS protected so the systems were up as well as our connection to the internet. I wrapped up my phone call, shut down my computer and joined everyone else mingling around the office. I checked on our IT guy and he was in the computer room as expected. We tried to contact some of our managed IT clients but at this point phone service was down in the area.
By 4:10 we're wondering if the power is coming back on. A few people are checking for news on their smart phones and we found out that this seemed to be a very large power outage, a couple hundred miles in every direction actually, including into Mexico. This was not going to be our typical power outage...
Being a few days before September 11 you'd think our first thought would be some sort of terrorist event. In our office however, Zombie Apocalypse came out as the first possible reason. After all we had some people out sick that morning. Hmmm, one of the sales guys just left of the office not an hour ago not feeling well... You know, one of the operations people left a little while ago too... OK, what's going on? Famous quotes from Zombie movies start coming up as we laugh and joke around.
So by 4:20PM we're preparing to close up for the day. All our servers have been shut down and the alarm systems are on battery backup with mobile backup lines to central monitoring so that's all good. We head home to join everyone else in traffic.
The good news is that for most of southern California the power was back on again within 8 hours. Soon enough that if people kept their fridges and freezers closed their food didn't go bad.
The next day was pleasantly uneventful. Due to shutting down our servers properly we had zero hiccups to our data. By the way, whether we had been here or not our UPS shutdown software would have shut down our servers. We reached out to key accounts and our managed IT clients to make sure all was well. For one of our managed IT clients this was very timely. We had just installed and configured a new UPS system for their servers not 2 weeks ago. When they came in the next day everything powered up clean and they were back to business as usual, no issues, just another day at the office.
Unfortunately the bad news is I did speak to some clients that didn't have UPS power or in some cases did but never bothered to setup their shutdown software and weren't at the office when the power went out. One IT person I know was scrambling to get back to the office but got stuck in traffic, I'm not sure if they made it in time. Another poor IT guy spent all day Friday and this whole past weekend scrambling to get all his systems up and running again, he was on his way to Fry's Electronics last e-mail I got...
So the lesson here simply put is to "Be Prepared". Think about what possible problems could happen and plan for the more obvious ones. Power outages are going to happen. Theft unfortunately happens. Unexplained data corruption happens. Mother Nature, well, she happens too... We're all busy, if you're honest with yourself you're always going to be busy, you just have to make the time to plan a little. The good news is you probably don't have to add Zombie Apocalypse to your list of likely disasters, at least let's hope not! If you do though, please share with us what you put into your plan :)
