The Need for Standard Performance Ratings

You can't manage something you can't measure, which means the performance of distributed web apps, whether they are running on "bare metal", local virtual servers or remote ones on the cloud, is totally unmanageable. There is no, and I mean NO standard way of comparing the performance of two servers, or even two versions of the same server, much less with bandwidth and connectivity taken into account. We can measure all the bits and pieces, from CPU speed to upload and download speeds, but we can't measure the whole and make valid and useful conclusions of any kind.

That's because the "whole" includes so many things, from the web server to the database to the OS to the network stack. How on earth can you measure all of that and say that it is faster or slower than anything else?

You do it the same way they do it in other industries, by defining and adhering to standard benchmarks that are reliable, repeatable, easy to use, and most of all, done in the same way each and every time. This has been a challenge in the past because competing operating systems, databases and so forth make it very difficult (though not impossible) to compare apples to apples.

The CapCal Server Performance Rating Test is a packaged performance test that is quickly and easily installed on any server and contains everything necessary to run a 2 minute performance test against a standard Web app, the Java Pet Shop (which has a .NET equivalent). The total number of pets purchased by 20 users over 2 minutes will be your rating number, which is large enough to be granular (in the hundreds or thousands) and small enough to be easily categorized. Since it exercises the entire stack (including the storage subsystem), this test is not only a good performance indicator but an excellent health indicator as well - all the important subsystems have to be up and running for the test to even start.

Following posts will show the CapCal Rating Test being run in a variety of instances so we can dissect them and see what they are telling us.