CPU Benchmarks for Amazon EC2 and Rackspace Cloud Servers

The cloud has made it very easy to allocate and use computing assets like CPU, Disk, and other services. Zoompf uses virtual servers to power our free scanning service, our performance scanner, and our WPO application. When I originally created these servers in early 2010, I choose Rackspace’s Cloud Servers. Zoompf has been growing a lot recently and so I re-examined different cloud server providers to see who I should use to build out additional servers. I thought I’d share my findings.

Choosing Rackspace to Start

I wish I could tell you there was some amazingly insightful reason for choosing Rackspace Cloud Servers in the first place. There really wasn’t. Rather circumstances and my lack of skills made it the logical choice.

You see, in early 2010, Zoompf was in the “feast-or-famine” mode of a bootstrapped startup and our cash flow was not particularly stable. I didn’t have the money to invest in a server and collocate it somewhere. In fact, I didn’t even really know how to do that assuming I even had the cash. So lack of money lead me to look at virtual servers in the first place.

Zoompf’s technology is written in C#, mainly due to my experience writing web crawlers and security auditing tools using Microsoft’s stack at SPI Dynamics and HP. In early 2010, I had a working performance audit tool, and was building a web interface. I didn’t want to waste time porting it, or trying to get to work with a Linux/Mono environment, or debugging a web interface in something besides Visual Studio. So that pushed me to use virtual Windows servers.

Finally, I’m hardly an IT operations guy. At the time I was focused on building Zoompf’s technology, not on trying to learn about administering virtual boxes. I knew I would need good documentation and some handholding to get a server up and configured properly. I couldn’t really afford managed hosting and I don’t think I even knew what it was at the time. I just needed someone smart to whom I could ask a lot of dumb questions.

So my requirements were: cheap, Windows, and idiot proof. Rackspace fit the bill. Their Windows Cloud Servers were cheaper than Amazon EC2, and Rackspace support is amazing. I’m happy I went with them. The process was easy, I was able to add addition systems as need be, and Rackspace’s live chat has been critical in helping me configure and administer Zoompf’s servers. But in 2012, was that still the best choice?

Problems and solutions

Rackspace has some downsides. There currently isn’t any elastic block storage like EBS. You also cannot “suspend and store” server instances as you can in EC2. Lacking these features in 2010 was annoying. Lacking them in 2012 is a real problem. Rackspace is beta testing its OpenStack cloud services, which includes an elastic block storage system and a “suspend and store” feature. Luckily neither of these problems are directly effecting Zoompf right now. Disk space hasn’t been a problem due to the number and size of our servers. Zoompf’s load is fairly constant and only increasing. We don’t currently need to spin up and spin down a lot of servers. When we add a server to increase our capacity, it’s because we need that capacity to satisfy customer demand. We don’t ever get rid of servers. So, while I don’t need those features now, I certainly will be happy and breathe easier when they become available later this year.

Benchmarking Windows Instances

So, while Rackspace has problems, they are getting addressed and don’t currently effect Zoompf. So I can ignore those differences and compare Amazon and Rackspace purely on the performance characteristics of their cloud servers. Earlier this month, I was looking to deploy additional servers to run Zoompf’s performance scanner. Our scanner is highly threaded and tends to be CPU bound. While disk space and memory are nice to have, it’s not my primary concern. My interest is in which cloud server instance gives me the most computational power for the least monthly cost.

Amazon and Rackspace each describe the CPU power of their instances with fairly meaningless units. An “EC2 compute unit” is difficult to conceptualize, let alone compare from Amazon to Rackspace. Instead, I tested different instance types from Amazon and Rackspace using Geekbench. I then divided the score by the monthly cost to determine a “CPU score per dollar” ratio. The higher the ratio, the better the performance value. For all instances, I used Windows Server 2008 with the latest Service packs and security updates.

This data reveals some interesting facts about Amazon and Rackspace’s virtualized offerings.

First of all, to run even a medium sized virtual server with Amazon or Rackspace will cost around $2000 a year. Always remember you are paying a high premium for the advantage of pay-as-you-go and use-what-you-need hosting.

Second, this chart illustrates a principle that is not as well known outside of the IT operations community: Getting more power by purchasing a few faster and larger machines is not as cost effective as buying several, less powerful, smaller machines. In this case, we see that moving from a Rackspace 4096 instance to a Rackspace 8192 instance is a 100% increase in price, but only a 72% increase in CPU performance.

Third, we see that Amazon’s High-CPU instances are not a good value. The EC2 Medium Instance is said to have 2 EC units of CPU power. The High-CPU Medium instance is 24% more expensive, offers 54% less RAM, and supposedly has 5 EC units of CPU power. However Geekbench testing reveals it is only 17% faster than the an EC2 Medium instance.

Finally, Rackspace instances provide more CPU power per dollar, than Amazon. If your application is CPU-bound, Rackspace may be a better and cheaper alternative.

Windows Tax

Using Windows over Linux on virtual servers is more expensive to run on a per-hour basis. Please note that this excludes time and resources spent maintaining a box. Of course, Microsoft maintains that Windows is cheaper because Linux has a higher Total Cost of Ownership (TCO). The nice thing about a virtualized environment is that many of these TCO costs are taken care of for you. You don’t have to worry about hardware replacement. With the wide range of community created instances, the initial configuration is largely taken care of already as well.

In my experience with Zoompf, the only ongoing system maintenance I need to do is keeping software up-to-date with security patches. Both Windows and Linux distributions like Ubuntu have automatic updating mechanisms. We have spent very little time managing our systems, beyond the initial setup of Zoompf specific packages.

This means the cost largely comes done to the price-per-hour. Windows instances cost more per-hour than Linux instances. Amazon charges between 43% and 73% more per-hour for a Windows instance than a Linux instance. Small, Medium, Large, and Extra Large Windows instances cost 1.43 times the equivalent Linux instance, while it goes up to 1.73 times for the High CPU instances. Rackspace charges only 33% more per-hour for Windows instances. If you want to purchase Windows instances, Rackspace instances will cost less per-hour than Amazon.

Recommendations

Geekbench is does not measure all aspects that affect the performance of a server. I did no testing about disk or network I/O, RAM performance, or other factors which affect performance in the real world. Remember that Geekbench is a synthetic, CPU focused benchmark.

We have seen the advantage of the “lots of small servers” over the “a few huge servers” approach. If your application can support it, and (more importantly) you have the technical expertise to manage and maintain it, I suggest spreading your application across a large number of smaller instances.

I suggest using Rackspace for people who need virtual Windows instances for CPU bound applications. Specifically, I recommend Rackspace 2048 instances. While they don’t have the highest power/cost ratio I like them for several reasons. First, 2 gigs is a reasonable amount of RAM for CPU intensive operations. 80 gigs of disk is enough for a large database while having a good deal of slack space for temporary data. Finally, its easier to maintain multiple 2048 instances than twice that number of 1024 instances. All and all Rackspace’s 2048 instance hits a nice sweet spot of cost, power, RAM, disk, and maintainability, at least for Zoompf.

While there is a lot of buzz around Dev/Ops, and cloud, and virtualization and such, remember that a cloud server is not always the best choice. You are paying a premium for the flexibility of using a pay-per-hour, scale-as-needed server. In many cases you are wasting money using a cloud server instead of a purchasing dedicated hardware. I recommend cloud servers only in the following situations:

• You are experimenting with a project and need a small server to test with.
• You do not have the capital to outlay for a large expense like a server.
• Your application has large, dynamic spikes in load, and you need to be able to both scale up and scale down very rapidly (as in a matter of hours)

Want to see what performance problems your website has? Zoompf tests yours web application for nearly 400 performance issues. You can get a free performance scan of you website now and take a look at our Zoompf WPO product at Zoompf.com today!