Over the past few weeks I've been thinking quite a bit about how software development companies, and specifically those such as ours that operate under a SaaS model, deliver their offering. While there is obviously some grey area, in my mind companies generally fall into two camps - those that while integrating certain libraries and commercially available technologies choose to build a custom solution versus those that take a purely COTS approach.
I can remember several years ago being in an airport waiting for a flight and striking up a conversation with a Microsoft evangelist and how completely surprised I was by how the conversation unfolded. We got into a general architecture discussion and how my company opted to build a custom horizontally scalable solution to meet our application's transactional requirements also adhering to strict up-time service agreements. His take was that the approach we had taken was a wrong one and that if we ever wanted to mature as a business we needed to completely rethink our approach and consider using a highly integrated end-to-end COTS solution.
My counter argument at the time was the cost savings we had achieved - even today we must accommodate severe price compression in our market - think of the license savings I said rather sheepishly. He then proceeded to rip that answer to shreds saying that in fact our custom solution cost us more in both development costs and that we couldn't go out and find engineering talent that "knew" our custom solution. With COTS he argued, engineers that know that technology grow are easy to find. I left that day a bit troubled with what he had said and really up until the last few weeks I wasn't able to construct a strong counter argument that makes me feel comfortable with our initial decision to go the custom route.
About the time I had my airport run-in with Mr. Microsoft, our company in fact decided to make a course change of sorts with respect to pursuing a more COTS based implementation. We were then and still are an Oracle shop and while I was not part of those making the decision, we decided to look at how we could better utilize Oracle's product offering. Probably the most significant system change that came out of this effort was to ditch our custom developed replication solution that had allowed both of our collocation facilities to be live (i.e., access for users could seamlessly switch between the two) and instead go with Oracle's Data Guard product in an active/standby configuration (i.e., one site would be live and the other would be a DR back-up).
The reasoning behind the decision was two fold. First, with the custom solution there was a perception that what we had developed was "too difficult to maintain". Developers would invariably create a new table but in the process forget to create the custom triggers necessary to keep the table in sync across sites. As one would expect, the replication would get further and further out of sync requiring us to repair the data by hand or in some extreme cases use a feature in our storage appliance to completely re-sync the two sites from scratch. Second, as stated by Mr. Microsoft, and supported by the many Oracle sales representatives we spoke with, if any issues did crop up engineers that knew Data Guard would be so "easy" to find.
Now roll the clock forward several years and we are still feeling the effects of this decision. We have gone through several senior Oracle DBAs and have found none of them to have experience using Data Guard with the amount of transactions our database processes every day. We've also paid licensing for "Active" Data Guard which is supposed to allow queries (e.g., for reports) to be issued against the standby instance only to find that the standby instance crashes after only a few hours in this mode. And most importantly the decision cost us our true "live-live" configuration where at any point in time we knew we had a truly live secondary site where we could point users should we want to do maintenance on the primary.
Having read all of this, you might be thinking to yourself that personally I prefer the build model and you would be correct. Now that I am the one responsible for making these sorts of key strategic decisions I look back on our change in course going with a COTS based approach and think it was a wrong one. We gave up too much and didn't really gain anything back in return. In practice, what I've learned is that experts on COTS solutions (especially Oracle) don't grow on trees and if you do manage to find someone talented they are prohibitively expensive. They also aren't the typical types that want to come on as a full time employee because quite frankly they don't need to with the consulting gigs they always seem to get.
Moreover COTS offerings don't always work as advertised especially when you push them past the boundaries of what the software can do right out of the box. Our experience with Data Guard is a prime example. I'm pretty certain the Oracle sales representative didn't provide us any clue that we might have an issue keeping our standby in an "Active" state where we could execute queries against it. And when you do run into these problems as we found out all too painfully your staff COTS "expert" can't think outside the box and you typically end up engaging the aforementioned prohibitively expensive consultants.
My other past experience with COTS solutions is that one typically must at least perform some amount of custom development to get them to work the way you want them to work. Who is going to do this development you might ask? Certainly not the COTS "expert" you hired to support the solution as they are not remotely comparable talent wise to the type of engineer that you would hire for the building a custom solution. This is at its core why I prefer the custom build route as I believe in order to be a successful small start-up company you must first build a small team of talented innovative A-players who prefer and demand building over buying and then they in turn will go out and build you a world-class custom solution.
No comments:
Post a Comment