Jesse Robbins at O’Reilly Radar has followed up on the outages at 365 Main. (I had written about this a couple of days ago: Hosted Data Centers and Outages). His post is a post mortem on the failure itself, and includes some commentary on the design approaches that 365 Main has taken.
The challenge in design and implementing architectures to deal with failure, is that often the complexity of the solution increases the likelihood of failure. One of the comments in my earlier post from Jeff Dao was that you should do a dry run of your solutions - simulate, or create a failure. But the more complex your solution, the more difficult it may be to simulate a failure - and in particular, the more difficult to simulate every failure scenario.
Having said that, in the case of 365 Main, it seems like they could have tested the three generators that failed to start up.
Jim on July 28th 2007 in Problem Solving, Technologies
I always am amused by this bumper-sticker. I was reminded of it yesterday as we drove by Gray’s Lake in Idaho, and saw Sandhill Crane’s (see Mary’s blog on Tincup Creek for more comments on Gray’s Lake). I remembered my 9th grade English teacher, Miss Davies, teaching us a poem about the Sandhill Crane. She pointed out that the rhythm of the poem matched the gait of the crane. A fact lost on me at the time (not a lot of crane’s in San Jose) - but watching them walk - sure enough - the poem matched the gait.
Jim on July 26th 2007 in Uncategorized
Arthur Bergman posted on O’Reilly Radar about the power failure in San Francisco on Monday (Failure Happens). I was corresponding with the Network Manager for Sapias (now with Wireless Matrix), Jeff Dao, and he had the following lessons learned (fortunately, the Sapias production systems were not in the floors that lost power at 365 main): 1) Do a dry run to see how your failure strategies will work (see 2)). 2) Don’t trust anyone to tell you how great their data center is, and how it can’t possibly fail 3) although there are advantages to being ‘just up the street’ from the data center, maybe the data center should be a long way away (Jeff did encounter issues in the corporate data center - which was impacted by the power hits, and was pleased that he didn’t have to be chasing production issues at the same time).
When Sapias first went live (almost five years ago), we were hosted at a data center that claimed that their building had not had a power outage in 40 years. Dual paths - dual everything. Generators on the top of the building. Within a month of going live - they had a power outage. They ran on generators for months as they coped.
Jim on July 26th 2007 in Technologies
Yesterday, the Santa Rosa Press Democrat published a Q/A between Nate Halverson, staff writer at the Press Democrat, and me, titled Make information technology work for you.
Jim on July 24th 2007 in Technologies
Mary wrote an entry today about starting a business 20 years later, the similarities and the differences. It is still true that people are the core — and finding the right partners and team members is essential to success. Mary has written a lot about catching the updraft, and the importance of both your own, and your entire teams approach to the creative process. Starting a business is a creative process. You need to make sure that you and your team are working together to Envision, Align and Embody your results.
You can’t just grab a bunch of smart people and go. You have to get the right collection of smart people. The right skills, the right skills mix, the right view of the goal, and the common desire - make that expectation - of success.
What is different now, is that we know more - and know more people than we did 20 years ago.
Jim on July 23rd 2007 in Uncategorized
There is a lot that is special about live performances. We recently attended the Oregon Shakespeare Festival for a week. We saw seven plays over the course of five days. It is so much better than watching a movie that I am reluctant to make the comparison. You feel the risk that the performers are taking, and the success that they are achieving. And you get to make a connection with the performer. At the Oregon Shakespeare festival, some of that connection is achieved by seeing the same actor in multiple roles, and some is achieved because you see the actors around town. Last Sunday, I attended a masterclass put on by Sir James Galway, and his wife Lady Jeanne Galway. I have been lucky enough to attend a concert with Sir James in San Francisco, and a duo concert with the two of them in Santa Rosa in the last couple of years. Being in the room with them is such a different experience from listening to a CD - or even viewing a concert on TV. At one point, late in the afternoon, they played a duet. Although it was clearly planned - there was time only for each to pick up a flute from the table, and start to play. You get a different level of appreciation of their art - seeing them - just play. Lady Jeanne made the comment that you have to be ready before you walk in the room - and Sir James had walked in several hours before.
Last week, we attended an evening of Jazz sponsored by the Healdsburg Jazz Festival - and saw a local favorite, Julian Lage with his trio. After a couple of numbers, Julian introduced his drummer, Lorca Hart and bassist, John Shifflet - and them introduced them to each other - they had never met before. That added to the magic of the evening as they listened to each other - found common ground - and then found new ground.
I think about the musician who were alive in my lifetime - that I never took the opportunity to hear/see live. There are always legends (Dave Brubeck will play in Santa Rosa in August - sponsored by the Healdsburg Jazz Festival), and wonderful young musicians like Julian Lage, playing locally. I need to make sure that I spend more time seeing them, hearing them, learning more about them than I might learn watching them on TV, or listening to a CD.
Jim on July 22nd 2007 in Arts Commentary
This about getting stuck down a path. Going down just one path or exploring just one solution, you are not able to see any other possible solutions. This is similar to another one of our techniques: The buzzsaw moment. In that post, I described having a single solution taking all of your attention. And perhaps it isn’t even really a solution at all.
In this case - you get so drawn in to one path - that you don’t even notice (can’t even notice?) the other paths anymore. If you are on a waterslide - there is really no choice until you come out the other end. Hopefully you can find a way to jump paths - or at least notice that there are other paths to consider.

[The cartoons were done for us several years ago by Marc Schmid. His website is www.cartooncity.net ]
Jim on July 13th 2007 in Problem Solving
I use Adium on my Mac - and have my Yahoo and AOL accounts on it. I also use iChat - thought not as much. Because I have an older PowerBook - I don’t have a built in camera. iChat wins when I want to use the camera - but it means unpacking it (and finding it first)… Most of the time, I use Adium.
The key features I look for are the basics. The ability to maintain multiple conversations simultaneously (and asynchronously). Also the ability - offered in Adium - to have ‘group chat’. We found these very effective at Sapias when we were doing deployments - everyone was listening in - and adding their status and questions. This is available in other tools as well.
Today I looked at Velvet Puffin. The following is from their web site:
Socially Active Instant Messenger
VelvetPuffin is a new kind of instant messenger, built on the idea that IM can be more intuitive, real-time, and socially active.
Through an Instant Messaging interface
Chat, share photos, blogs, videos & polls with friends
View your friends’ photos, blogs, videos and polls.
Receive instant alerts when friends share new content
View/share videos from YouTube, Google Video & Metacafe
Mobile Access
You can do all of the above on your mobile phone with
VelvetPuffin Mobile. (If the phone is Java based)
Velvet Puffin covered the basics (though I didn’t see the group feature). The key aspect is the ability to share pictures and videos with the people you are IMing with. There is value here from a social networking perspective - but I didn’t see any particular value added for the business setting.
Jim on July 9th 2007 in Technologies, Companies
Wireless matrix announced today that it is acquiring Sapias, Inc.:Â Wireless Matrix Further Accelerates Expanding Mobile Resource Management Application Capabilities With Acquisition of Sapias.
I am proud of the team that we built at Sapias, and the technology and systems that Wireless Matrix is committed to utilizing as the next generation of their services.
Jim on July 9th 2007 in Companies
Mary and I want to share our calendars. We have utilized Google Calendars, and now have subscribed to those in iCal. This allows us to edit the calendars in Google (or send invitations from Outlook, Entourage or other sources), and then view them (read only) in iCal. We have a Google Calendar for each of us, and a joint one. In iCal you right click in the Calendar pane (left hand pane) and choose subscribe, entering the iCal URL from Google Calendar. It isn’t a perfect solution - but works for the moment.
I had hoped with a 2 megapixel camera, that we could use ScanR - sending faxes, entering business cards, and OCRing documents. But for the moment, Apple is resizing the images to 640×480 before emailing them (to ScanR) - which isn’t enough resolution for ScanR.
I am now using my iPhone to view my eBay to check on items I am selling, Google Analytics to check on access to our web site, and the TextDrive admin site to administer our hosted system.
I have added both an Exchange email account, and my personal email account (hosted by TextDrive) - as IMAP accounts. This does allow me to have folders that I save items to on my iPhone - so I have a way to ’store’ files of interest - spreadsheets and other items. It seems like there should be an independent folder system - but at least I can save content.
Note that the iPhone is turning out to be, primarily, a read-only device. There are limited places that you can write stuff: add contacts, add calendar entries, send emails, take pictures, add crude notes in the Notes application.
The map is quite useful. Although we travel with a GPS device (a portable one that can work in the car - as well as tell us how far, how fast and how high we have gone on our bike), we found the iPhone quite handy. It can give us a map of any of our contacts (I keep contact info for my 2700 closest friends on mine) by clicking on the address in the contact. You can then use that as a from or a to for a trip. The iPhone does not know its own location - so you have to provide both ends. But you now have directions - including a list you can step through. I click on the ‘car’ in the bottom right of the map page, and you get an overlay of traffic conditions.
Jim on July 3rd 2007 in iPhone