Build versus Buy

Over the past few weeks I've been thinking quite a bit about how software development companies, and specifically those such as ours that operate under a SaaS model, deliver their offering.  While there is obviously some grey area, in my mind companies generally fall into two camps - those that while integrating certain libraries and commercially available technologies choose to build a custom solution versus those that take a purely COTS approach.

I can remember several years ago being in an airport waiting for a flight and striking up a conversation with a Microsoft evangelist and how completely surprised I was by how the conversation unfolded.  We got into a general architecture discussion and how my company opted to build a custom horizontally scalable solution to meet our application's transactional requirements also adhering to strict up-time service agreements.  His take was that the approach we had taken was a wrong one and that if we ever wanted to mature as a business we needed to completely rethink our approach and consider using a highly integrated end-to-end COTS solution.

My counter argument at the time was the cost savings we had achieved - even today we must accommodate severe price compression in our market - think of the license savings I said rather sheepishly.  He then proceeded to rip that answer to shreds saying that in fact our custom solution cost us more in both development costs and that we couldn't go out and find engineering talent that "knew" our custom solution.  With COTS he argued, engineers that know that technology grow are easy to find.  I left that day a bit troubled with what he had said and really up until the last few weeks I wasn't able to construct a strong counter argument that makes me feel comfortable with our initial decision to go the custom route.

About the time I had my airport run-in with Mr. Microsoft, our company in fact decided to make a course change of sorts with respect to pursuing a more COTS based implementation.  We were then and still are an Oracle shop and while I was not part of those making the decision, we decided to look at how we could better utilize Oracle's product offering.  Probably the most significant system change that came out of this effort was to ditch our custom developed replication solution that had allowed both of our collocation facilities to be live (i.e., access for users could seamlessly switch between the two) and instead go with Oracle's Data Guard product in an active/standby configuration (i.e., one site would be live and the other would be a DR back-up).

The reasoning behind the decision was two fold.  First, with the custom solution there was a perception that what we had developed was "too difficult to maintain".  Developers would invariably create a new table but in the process forget to create the custom triggers necessary to keep the table in sync across sites.  As one would expect, the replication would get further and further out of sync requiring us to repair the data by hand or in some extreme cases use a feature in our storage appliance to completely re-sync the two sites from scratch.  Second, as stated by Mr. Microsoft, and supported by the many Oracle sales representatives we spoke with, if any issues did crop up engineers that knew Data Guard would be so "easy" to find.

Now roll the clock forward several years and we are still feeling the effects of this decision.  We have gone through several senior Oracle DBAs and have found none of them to have experience using Data Guard with the amount of transactions our database processes every day.  We've also paid licensing for "Active" Data Guard which is supposed to allow queries (e.g., for reports) to be issued against the standby instance only to find that the standby instance crashes after only a few hours in this mode.  And most importantly the decision cost us our true "live-live" configuration where at any point in time we knew we had a truly live secondary site where we could point users should we want to do maintenance on the primary.

Having read all of this, you might be thinking to yourself that personally I prefer the build model and you would be correct.  Now that I am the one responsible for making these sorts of key strategic decisions I look back on our change in course going with a COTS based approach and think it was a wrong one.  We gave up too much and didn't really gain anything back in return.  In practice, what I've learned is that experts on COTS solutions (especially Oracle) don't grow on trees and if you do manage to find someone talented they are prohibitively expensive.  They also aren't the typical types that want to come on as a full time employee because quite frankly they don't need to with the consulting gigs they always seem to get.

Moreover COTS offerings don't always work as advertised especially when you push them past the boundaries of what the software can do right out of the box.  Our experience with Data Guard is a prime example.  I'm pretty certain the Oracle sales representative didn't provide us any clue that we might have an issue keeping our standby in an "Active" state where we could execute queries against it.  And when you do run into these problems as we found out all too painfully your staff COTS "expert" can't think outside the box and you typically end up engaging the aforementioned prohibitively expensive consultants.

My other past experience with COTS solutions is that one typically must at least perform some amount of custom development to get them to work the way you want them to work.  Who is going to do this development you might ask?  Certainly not the COTS "expert" you hired to support the solution as they are not remotely comparable talent wise to the type of engineer that you would hire for the building a custom solution.  This is at its core why I prefer the custom build route as I believe in order to be a successful small start-up company you must first build a small team of talented innovative A-players who prefer and demand building over buying and then they in turn will go out and build you a world-class custom solution.

Never Criticize Creative Thinking

The other week while visiting on-site with one of my contractor partners I spent one late evening at the office with some of the more senior leadership of the company.  We somehow began discussing ideas that some of their more junior developers had floated.  I remember that during the discussion that we all out of hand dismissed some of their ideas concluding that while they certainly had creativity, there were real business reasons why they most likely would not find legs to stand on.

That night back at the hotel it occurred to me that in our discussions we had been too fast to pass judgment.  I actually thought of similar situations where I had experienced the same where my ideas were "poo-poohed" by senior management.  While experience in the real world can provide a more realistic outlook on what is possible to implement at the same time it can breed pessimism that in fact leads to less creative thinking.  I would say even in IT there is something magical about the innocence of youth when it comes to dreaming up truly innovative (and sometimes disruptive per an earlier post) ideas that are worth exploring.

As an example, about ten years ago at my last wireless start-up I had one of these ideas that received similar dismissive treatment.  My idea was how cool would it be to stream content from my cable box over the net to my desktop at work or wherever.  I remember vividly the reaction I received from many of the senior engineers at the company.  "Your idea will never work!" they told me, "there won't ever be enough bandwidth to support your idea".  Several years later the SlingBox was born and today a company called Aereo is taking on the network giants allowing consumers to stream over-the-air broadcasts.

That evening we also had a discussion about the usefulness of patents.  Put aside the argument of whether patents by in large are defendable - our discussion focused purely on the usefulness of them with regards to forming new ideas and lines of business.  Take a company like Yahoo for example.  If one spends time reading what they are patenting many times a typical reaction even from someone who is even seasoned from a technology perspective might be "well that is the craziest, dumbest idea I've ever heard of -  why would you even want to patent that idea?".  And sure, maybe some of them are crazy and stupid but it occurred to us that just going through the patent process itself can lead to great ideas that are not so far out of left field.

Ultimately, criticism or immediate dismissal of anyone's ideas usually results in that individual shutting off any further creative thinking.  When this 'tune-out' effect occurs it means that future great ideas are being lost from that individual.  One of our responsibilities as leaders in IT is to build environments that in fact fosters and even encourages this type of "crazy and stupid" thinking.  We always have to always ourselves that for the dozens of ideas that while they may be creative but are not unique or realistic there is that one gem of an idea that is truly original. 

Building a Winner

Let me start by saying I know that this topic has been covered countless times which would make it logical to ask what I can add that is unique or differentiating.  Even given that premise, I'm going to still try my hand at adding something to the discussion.  Building a winning team sounds so obvious, I mean who wants to build a losing team right?  But to be able to put together the right team that has chemistry, talent and drive to succeed it isn't easy.  The difference between a winning and losing team is a scarily fine line.

The other evening I was watching a documentary produced by NFL Films on the success of the San Francisco 49'ers under Eddie DeBartolo's ownership.  The documentary focused mostly on interviews with former players and coaches who tried to explain why he was so successful in building a winning organization.  There seemed to me to be two components to his success - creation of a tight-knit family friendly organization and instilling an intense desire to win.

Let's start first with his obsession to instill a winning mindset throughout his organization.  I'll lead off with a DeBartolo quote that I think perfectly sums up the type of individuals you don't want to be on your team:

Show me a good loser and I'll show you a loser.

In building those successful teams, he searched for people who just had it in their DNA a desire and passion to win.  That's not to say that those teams didn't experience failure and that even in some degree losing helped build successful teams in the future.  But he didn't want anyone around the team that didn't absolutely detest losing.  He also said that when you've brought someone on board that doesn't fit into the culture admit you've made a mistake and be prepared to address the situation by cutting that person from the team.  Admitting failure and getting rid of that individual is critically important to building a winning organization.

One might envision based on how driven he was to win that his organization was not family friendly or that there might be tension or animosity amongst employees.  Nothing could be further from the truth.  It was apparent watching the documentary that DeBartelo compensated for his brutally tough passion for winning by truly caring about everyone in his organization.  He made it a point to know every employee personally - from the head coach down to the cook in the team kitchen.  He knew their names, the names of their children, ages of their children, everything about them.  He somehow found time out of his day to make these connections with everyone - he realized that a family atmosphere within his organization was a key component to building a winner.

I think two precepts of DeBartolo's success in building a winning team are important.  Surround yourself with individuals that have the same desire you do for producing something truly special.  While they don't all have to be type-A's, it is important to find those that don't want to be on a losing team.  At the same time it's important to never forget once you have this type of an organization established how special of a situation it is and focus on making it a family by personally getting to know all of your employees and the portion of their lives that occurs outside the office.