The Night of Derecho (AKA The Day the Data Center Died)
Last Friday evening, a derecho (a widespread and long-lived straight-line windstorm that is associated with a fast-moving band of severe thunderstorms) hit the eastern part of the U.S. The storm was like a tornado without the funnel or a hurricane without the rain.
Similar to those types of weather events, it left a trail of destruction and widespread power outages from Indiana to New Jersey – with over 3 million without power on one of the hottest days on record.
A few things have become apparent with this storm. (1) You CANNOT underestimate the power of social media; (2) The Cloud Can Fail; (3)Consumer dependence on power has clearly soared over the last decade; and (4) Smart Grid would have made little difference.
The last time I experienced an extended power outage was in 1989 after Hurricane Hugo. We lost power for 5 days, but NEVER landline telephone service. This time around, I was out for 2 days (although many are still out 4 days in) and landline phone service died with the power.
I was actually out of town when the storm hit and both Twitter and Facebook proved invaluable to get information about the storm. It was the only way I was able to get weather information and perhaps more importantly, updates on the aftermath. My profound thanks to everyone who provided updates and let me know that my house was okay.
In the path of this storm was Amazon’s Web Services, which provides cloud-based infrastructure services to hundreds of companies including Netflix, Instagram, Pinterest, and Heroku. Although back-up power sources kicked in for many of its data centers – it didn’t for one of them – leaving the customers in the dark (literally) when they tried to access their services. One twitter post about it was particularly funny – “I guess all of those people will now have to actually drink their coffee rather than post picture about it…”
Amazon said a “power outage caused storage failures” for customers who rent space on Amazon’s servers through its cloud service. The outage highlighted the fact that the cloud is not always dependable and how a single failure can have widespread impacts across the web – especially when the company hosts thousands of customers.
Although services such as Amazon allow customers to to spread their data across eight regional data centers around the world; many, especially smaller companies do not do this or even have a disaster recovery plan in place. Even though the risk is low that failure will occur – it can and it does and companies need to make sure this is part of their strategy – lost revenue is never a good thing when it can be prevented.
The Need for Power
Last time I experienced an extended power outage – I did not own a cell phone, tablet, or computer. I did not access the internet, have an email account or stream my entertainment. I still played records and cassette tapes. And I still had a transistor radio and a landline – which I used extensively.
Nowadays, nearly everything I do requires being connected to power to access my services – whether it is Skype, Netflix, iTunes, shopping, banking, etc. Yes, I can do alot with my smartphone, but I still need to sit in my car to charge it up. In the past, I was only worried about having batteries – now I panic when I can’t play Angry Birds….perhaps its time to re-evaluate a few things.
The complaints about the slow restoration process are endless. People tend to forget how their power is actually delivered to them. The outage was so widespread, our local utility has had to call on assistance from utilities from Florida to Texas.
Would a Smart Grid have helped? Not really. A Smart Grid will make the network run more efficiently, but it cannot prevent a natural disaster. This storm not only took out distribution lines, but transmission lines as well, affecting numerous substations.
There are some aspects that would have been useful – such as the ability for smart meters to provide automated real-time alerts to the utility – it was particularly frustrating to not be able to reach the outage reporting centers via phone or web. Additionally, smart meters might be able to help identify nested outages (multiple breaks to the line). Finally, utilities could use smart meter data in combination with Google Maps to inform customers where power was out or more importantly which areas had power.
What has been most evident, is how little attention utilities pay towards vegetation management and really just simple maintenance of the network. A fair amount of the outages probably could have been prevented – if they would regularly trim trees whose branches near the power lines.
Regardless of these events – I can say that we have experienced more outages in the past year than the past 20 years. Grid modernization is critical and necessary as these increasingly common catastrophic weather events continue to demonstrate how vulnerable the electrical network is, not only to our daily lives, but our national security.
Thank you AEP, APCO, ON Power, North Huston Pole Line and all the other utilities for everything you have done this week.