4 Reasons to use Linux Agents vs Windows Agents (WebPageTest AMIs)

If you run a private WebPageTest instance on Amazon EC2, one of the choices you’ll have to make is whether to use Linux or Windows test agents. I had always been an advocate for the Windows AMIs – as most of the world runs Windows instead of Linux, I wanted the performance to represent what the majority sees.

But last week I switched over to Linux test agents, and I don’t think I’ll ever go back to Windows – here’s why:

1. Linux Agents are more consistent

Recently there have been updates to the driver that runs the the Windows test agents, due to Chrome’s pending security updates that will restrict code injection so the performance can’t be measured using the old methods. New AMIs had been available for several months, but I waited to switch until I was ready – and looked what happened – here’s a graph of stackoverflow.com’s performance over a 30 day period:

What used to be steady performance became erratic, and that’s the opposite of what you want when you’re monitoring your site speed. There are several theories about why this performance drops – could be related to video processing, or perhaps the RDP session – but what’s obvious is that CPU is much higher than it used to be. Take a look at this CPU chart before, and after using the new Windows AMIs (this is the same exact url, less utilization is better):

Okay, perhaps that site has some weird issues going on. Let’s look at a bunch of sites in aggregate. Here’s 10 pretty typical sites (including Amazon and Bing, not just mom-and-pop sites):

Nope, there’s definitely a problem across the board.

I reached out to some other WPT users, and they confirmed the same issues. One person tried to fix it by beefing up the hardware from a large to xlarge (which doubles the compute power), but that didn’t make a big difference.

Switching to the Linux AMIs is what made the results more reliable, as seen in the smoothing out of the lines in both of the MachMetrics screenshots above (after July 19th).

2. Linux Agents are less expensive

The pricing for Windows servers on Amazon EC2 is over twice as much as the Linux alternative. Here’s Amazon’s current pricing for a c5.large server in the US East region:

• Windows: $0.177 per Hour
• Linux: $0.085 per Hour

To put that in perspective, if you’re running your agents for 12 hours per day, that’s a difference of $33.12 per month, per agent!

On top of that, the Windows images are billed in hourly increments, while the Linux instances are billed by the second. When you’re using autoscaling, having your agents spin down before the end of an hour and not getting charged for that whole hour can add up to lots of savings.

3. New Chrome versions are released equally on Linux and Windows

One of my fears was that the Linux version of Chrome would be less supported than the Windows version. I didn’t want to be testing using an older browser than most of my traffic. But in checking into the Chrome release history, you can see that the Windows, Linux, and MacOS versions of Chrome all get released at the same time:

4. Linux Agents are used by the industry professionals

Pat Meenan, the creator of WebPageTest, said last week, “If at all possible, I HIGHLY recommend using the Linux AMI’s for Chrome and Firefox and only use Windows if you REALLY need IE testing.” His opinion is always very respected, being the industry professional he is.

Also, the public WebPageTest locations mostly use the linux agents, as do some of the other premium site speed monitoring services, “so it gets quite a bit of testing and eyeballs inspecting it.” And now that MachMetrics does as well that’s even more industry leaders using it.

Consider the Linux Agents

There is one big reason to stay on the Windows AMIs, and that’s only if you absolutely need to support IE/Edge testing – Microsoft only releases Internet Explorer for Windows, not other platforms.

But if you’re open to using Chrome and/or Firefox (like most of the world does at the moment), then give the Linux test agents a try. Let me know in the comments what your experience is!