If you followed the comments on the Geeks Are Sexy blog post, you saw that someone named Brad from SOASTA, "the Cloud Testing Authority", left a rather odd comment about how a million users happens all the time, just not all at once. It's the "all at once" part that's the point - give me enough time and I'll do a trillion user load test! Just ask the Election Commission in India who got 8.6 trillion hits over 8 hours and it flatlined the server yesterday!
Anyway, I challenged Brad to a contest to see who could reach the million user mark the fastest in a way that could be fairly judged by a third party and he declined, calling it a "scientific experiment". Where I come from it's called "competition", and in this case the competition is really not either one of us, it's the 800 pound gorilla in the load testing space, HP, who is not going to watch idly as their LoadRunner franchise evaporates. SOASTA may think of themselves as the 800 pound gorilla in the cloud testing space, but I say the cloud testing space is too new for a gorilla. A baby gorilla or two, maybe, but no 800 pound gorillas.
CapCal has the decided advantage of having been around longer and having designed for a much more challenging "cloud", the Internet, and a lot of those earlier decisions and trade-offs are starting to pay off in a big way on the Amazon cloud. I don't know if SOASTA uses a Windows agent or a Linux agent but if it's Windows we could easily have a 10 to 1 advantage! Windows takes too much network and disk time to copy, takes too long too boot up, requires too much memory, and is more restrictive than Linux. And even if your agent runs on Linux, how you scale and control your agents has a lot to do with how many virtual users you can handle on a given instance, for example.
Either it seems obvious or counter-intuitive, but an application that tests the performance and scalability of other apps can itself have performance and scalability issues, and these kinds of things only come out at really high load levels. This is where CapCal has had lots of attention focused and a lot of work done - the scalability of the controller-agent design has proven itself over and over again to be virtually limitless. Even so, the very first million user test - a record that will stand until someone else beats it - will be a challenge, and it will be reported here "live", so come back soon!
Good post. Not many think about the VU agents of a Load testing software . The more efficient they are,the better the overall efficiencies and costs can be and ofcourse with less complexities.
ReplyDelete