Jim Panttaja

Arthur Bergman posted on O’Reilly Radar about the power failure in San Francisco on Monday (Failure Happens). I was corresponding with the Network Manager for Sapias (now with Wireless Matrix), Jeff Dao, and he had the following lessons learned (fortunately, the Sapias production systems were not in the floors that lost power at 365 main): 1) Do a dry run to see how your failure strategies will work (see 2)). 2) Don’t trust anyone to tell you how great their data center is, and how it can’t possibly fail 3) although there are advantages to being ‘just up the street’ from the data center, maybe the data center should be a long way away (Jeff did encounter issues in the corporate data center - which was impacted by the power hits, and was pleased that he didn’t have to be chasing production issues at the same time).

When Sapias first went live (almost five years ago), we were hosted at a data center that claimed that their building had not had a power outage in 40 years. Dual paths - dual everything. Generators on the top of the building. Within a month of going live - they had a power outage. They ran on generators for months as they coped.