Valued Partner News October 2011
|
|
"Rule#1: Cardio"
- Zombieland
 Maybe the folks that decided September should be "Disaster Preparedness Month" did some research beforehand. After all, September definitely brought it's share of problems: heat waves, hurricanes and an unprecedented power outage in southern California that affected 5 million people. This may not seem like a big deal to other parts of the country but it was a pretty big deal here since it had never happened before.
We're actually headquartered in San Diego so we were definitely affected. On September 8th, about 3:40PM the lights went out. Not a big surprise since we'd been in a heat wave blasting our A/C like everyone else. We just figured the power grid operators decided there was too much strain on the system and shut down some areas to get it under control. I was actually on the phone with one of our suppliers when it happened and, fortunately, I have a UPS on my desktop and phone. Our computer room is also UPS protected so our systems were up as well as our connection to the internet. I wrapped up my phone call, shut down my computer and joined everyone else mingling around the office. I checked on our IT guy and he was in the computer room as expected. We tried to contact some of our managed IT clients but at this point phone service had gone down in the area. By 4:10 we were wondering if the power would be back on any time soon. A few people were checking for news on their smart phones and we found out that this appeared to be a very large power outage, a couple hundred miles in every direction actually, including into Mexico. This was not going to be our typical power outage... Since this occurred a few days before September 11 you'd think our first thought would have been some sort of terrorist event. In our office, however, Zombie Apocalypse came out as the first possible reason. After all, we had some people out sick that morning. Hmmm, one of the sales guys had just left the office an hour ago not feeling well... You know, one of the operations people left a little while ago too... OK, what's going on? Famous quotes from Zombie movies started coming up as we laughed and joked around. So by 4:20PM we were preparing to close up for the day. All our servers had been shut down and the alarm systems are on battery backup with mobile backup lines to central monitoring so that was all good. We headed out the door to join everyone else in traffic. The good news is that for most of southern California the power was back on again within 8 hours. Soon enough that if people kept their fridges and freezers closed their food didn't go bad. The next day was pleasantly uneventful. Due to shutting down our servers properly we had zero hiccups to our data. By the way, whether we had been here or not our UPS shutdown software would have shut down our servers. We reached out to key accounts and our managed IT clients to make sure all was well. For one of our managed IT clients this was very timely. We had just installed and configured a new UPS system for their servers not 2 weeks before. When they came in the next day everything powered up clean and they were back to business as usual, no issues, just another day at the office. Unfortunately, the bad news is I did speak to some clients that didn't have UPS power or, in some cases, did but never bothered to setup their shutdown software and weren't at the office when the power went out. One IT person I know was scrambling to get back to the office but got stuck in traffic. I'm not sure if they made it in time. Another poor IT guy spent all day that Friday and his whole weekend scrambling to get all his systems up and running again, he was on his way to Fry's Electronics last I heard... So the lesson here simply put is to be prepared: "Rule# 4: Wear Seat Belts." Think about what possible problems could happen and plan for the more obvious ones. Power outages are going to happen. Theft, unfortunately, happens. Unexplained data corruption happens. Mother Nature, well, she happens too... We're all busy, if you're honest with yourself you're always going to be busy, you just have to make the time to plan a little. The good news is you probably don't have to add Zombie Apocalypse to your list of likely disasters, although Halloween is just around the corner... so if you do, please share with us what you put into your plan. As always, your team at Partners is available to help.
"Rule# 8: Get a Kick Ass Partner." :)
~Your Partners at Partners Data Systems
 |
