Why Self-Service and On-Demand are Critical

Last week I got a call from the senior developer at an online retailer with an urgent request - they were seeing errors in production at load levels higher than they could generate in the lab and needed to do some very large load tests immediately. I instantly put myself in his shoes and imagined how I would feel if it were my server and the word "immediately" took on a very visceral meaning!

Within an hour and a half we had negotiated the business terms and he had been trained on CapCal and was building and running his own tests! To make sure each user and session was unique, he created a few thousand test accounts and generated a CapCal replacement data set with user id, password and so forth. That took a couple hours, and then to make a short story even shorter a series of 5,000 user load tests using 10 Amazon instances uncovered a bug in their PHP framework that was causing the errors. Problem solved! (After hours of re-coding and unit testing that is).

They'll be spending a week or two doing more testing now that they've solved their immediate problem. They want to reach 150,000 simultaneous users while maintaining sub-second response times with no errors, which is pretty awesome in itself. What's even more awesome is how quickly and easily that kind of load is generated by CapCal using the Amazon cloud!

Click on the chart above and see how quickly it ramps from 1 to 5,000 in the Users column with excellent response times and zero errors. But at about 3 minutes into the test, server errors begin showing up and rapidly climb. Response times hardly degrade at all, which once again reminds us that scalability and performance go hand in hand but scalability is king!

I hope to get permission to publish their name and get a quote for the blog but I have to say that this particular developer is extremely bright and caught on to CapCal faster than practically anyone I've ever seen. But even mere mortals are capable of getting the same results in the same amount of time since there's no programming involved.

The buzz around here now is about moving CapCal to EC2 and running load tests on itself to test the autoscaling and load balancing features. Stay tuned!