Where Less is More and Free is Costly

While it's true that testing is a means to verify the expected behavior of a system, what it really boils down to is uncovering and fixing all the things that prevent that behavior.  It's an iterative process by nature, the very purpose of which is to find problems, whether by manual or automated means.  And nothing uncovers problems like a good load test - bugs, bottlenecks and errors of all kinds come to the surface that may have nothing to do with performance.  

So the process of load testing should really be focused on finding and fixing the problems encountered along the way and NOT on the load testing itself.  To whatever degree your load testing tool slows this iterative process down or complicates it in any way is the measure of what it is REALLY costing you - most often there are three to five people involved in a load testing project, an expense that can dwarf the cost of just about any tool after a day or two of testing.  And, paradoxically, it is the open source tools that are "free" which often take the most time to learn, master and use.

And herein lies the paradox of pricing for testing services like CapCal - less is really more when it comes to load testing and to charge customers for the amount of testing they do can actually be self-defeating and non-productive in the long run.  Nobody knows in advance how much testing will be required for a given website and it can even vary substantially between releases of the same website. Fortunately the pricing of Linux instances on EC2 is low enough to make it affordable to most companies.  But if the real advantage of a tool like CapCal is how much time it saves you does it not seem counter-intuitive to charge for the amount of testing you do?

It does, and that is why we lean towards a monthly pricing model that allows unlimited testing with the customer paying the AWS charges.  The real ROI is in the time saved creating, modifying and running tests, which not only squeezes time out of the development process but also decreases time to market, which results in lower costs and higher revenue.  Lowering costs, reducing risks and increasing revenue are what CapCal is aiming for and our customers are telling us that it works.  


So if you are in the market for a load testing service and have received a quote from SOASTA or one of the others, you may discover you are paying more for less if you don't check out CapCal first!