I’ve talked a lot about trust, so this exchange from the Simpsons last night was incredibly funny to me.

Homer: Oh Marge, let the kid have his embarrassing secrets and lies.

Marge: That is totally irresponsible parenting!

Homer: Not according to the Internet.

Marge: Where on the Internet?

Homer: I’m sure some idiot somewhere agrees with me.

Marge: Why would you believe him?

Homer: Because his avatar is Darth Maul! The second coolest Darth! And what he says is true because its in all caps. Are you going to argue with caps?

Marge: Why does that make it right?

Homer: He took the time to press the shift key Marge! I think he knows what he’s talking about.

Trust, but verify? No way, if its in all caps, its perfectly fine. TRUST ME, I’M HOLDING THE SHIFT KEY.

While debugging a customer issue this weekend, I uncovered a problem with chunked encoding in general, and ASP.NET in particular, that can reduce your website’s performance.

Let’s start with some background.

Digicure is a web security and performance services company in Denmark. They are also a Zoompf customer. At the end of last week, they contacted Zoompf support to tell us that some of our Zoompf WPO pages were timing out. Zoompf WPO is our web performance scanner delivered as a SaaS. User’s log in to the web interface and can conduct performance scans, review scan results, and generate reports. Zoompf WPO’s web interface is written in ASP.NET. This is largely because our performance scanner is written in C#.

By default, ASP.NET does not use chunked encoding. When an HTML page is being dynamically generated, ASP.NET buffers all of the output, and sends all of the content at once. This response includes a Content-Length header because the entire response is created before being delivered to the client, so the web server knows how long it is. This is called the Store-and-Forward approach.

Store-and-Forward Vs. Chunked

Store-and-Forward is not necessarily bad. In fact, it’s how HTTP/1.0 transmits dynamic responses when Connection: Keep-Alive is used. But Store-and-Forward does not create the ideal the user experience. This is because no content is sent to the user until the application tier has finished generating the markup. This means the user sees no content. More importantly, the web browser doesn’t have any HTML yet, so it cannot start downloading other like CSS or JavaScript files while the HTML loads.

A better approach is chunked encoding. Chunked encoding was added in HTTP/1.1 and allows the web server to stream content to the client without having to know how large the content was ahead of time or having to close the connection when it’s done. We can see how the chunked encoding approach compares to Store-and-Forward in the figure below:

Chunked encoding is great, because the user starts getting content almost immediately. The application is faster because the browser can start to download other resources while the HTML is still being generated and streamed to the client.

Since chunked encoding can improve performance, I looked for appropriate places in Zoompf WPO’s web interface to use it. Now, there are a few areas of WPO where generating the HTML can take a long time. One is when WPO is generating the list of affected URLs for a specific performance issue. This list can be thousands of items long. Streaming this HTML to the client using chunked encoding provides a better user experience than completely generating the HTML and then delivering the page to the client. So I disabled output buffering for areas like that and ASP.NET uses chunked encoding to send pieces of HTML to a visitor as they are generated. I thought everything was fine.

Too Chunky

That is, until I heard from Digicure. They experienced pages which were loading slowly and would sometimes timeout. Specifically, lists of the affect URLs were appearing very slowly. I did not see this behavior when I tested the application, and no other customers had this issue. A quick check showed the web server and database were not under excessively load. Network checks showed there was plenty of available bandwidth to transmit the data quickly.

I decided to see the traffic the web server was actually sending However I did not use an HTTP proxy because I wanted to make sure that trying to measure what was happening did not affect what was happening. Instead I used Wireshark to capture the HTTP traffic between the browser and web server while fetching the slow pages. Here is what I saw:

2f
<a name="affected"><h2>Affected URLs</h2></a>


12
<div class="nb">

17
<ul class="url_list">

4
<li>

9
<a href="

34
ShowResponse.aspx?scan=7519&amp;got=96&amp;check=300

2
">

1a
http://XXX.XXXXX.XXX/de/de

6
</a>

7
</li>

4
<li>

9
<a href="

36
ShowResponse.aspx?scan=7519&amp;got=1659&amp;check=300

2
">

...

That shows some of the response body, encoded into chunks. The problem is each chunks is really small. As in, just a few bytes small. And there are so many chunks. Way too many. And then I noticed something with a sinking feeling: the way content was divided into chunks looked familiar. Oh crap, I know what the problem is. I went and looked at the source code generating this list:

if (!alreadyShown)
                {
                    fout.WriteLine("<div class=\"nb\">");
                    fout.WriteLine("<ul class=\"url_list\">");
                    foreach (IBasicItemInfo info in infos)
                    {
                        fout.Write("<li>");
                        HtmlUtils.RenderInternalLink(fout, "ShowResponse.aspx?scan=" + scanID + "&got=" + info.ID + "&check=" + issueID, StringUtils.Truncate(info.Name, 256));
                        fout.WriteLine("</li>");
                    }
                    fout.WriteLine("</ul>");
                    fout.WriteLine("</div>");
                }

See the problem? With buffered output disabled, anytime the application writes bytes to the response, those bytes are immediately sent to the client. Even if you are just writing a simply <li> tag! This HTML response is around 300 kilobytes, and ASP.NET is streaming that just a few bytes at a time. Really big pages sending all those chunks over a high latency connection like to Digicure are going to be slow.

There is another performance problem with overly “chunky” responses. Chunked encoding adds overhead. For each chunk, there are a few bytes to represent the length of the chunk, and then 4 bytes to represent two CRLF sequences. For small chunks, like sending an <li> tag, the overhead of the chunk is larger than the data in the chunk! For some pages Zoompf WPO was sending 75 kilobytes of chunked encoding overhead to transmit 300 kilobytes of data!

Control over this is very limited in ASP.NET. In fact, it seems to be Boolean. Either output buffering is enabled, and no content is sent to the client until everything is generated. Or output buffering is disabled, and every function call to write results in a chunk. There was no middle ground. I’m not an expert in IIS or ASP.NET and its possible I’m missing something, but Google queries returned no useful information about this.

In the end, I disabled buffered output and implemented my own output buffering. Now, using fout.Write() or fout.WriteLine() calls my class instead, which buffers data into 8 kilobyte chunks before sending the data to the client. Ideally, I should adjust the size of the buffer dynamically, based on how large I think the response is going to be. This is reasonable in WPO, since I know how many links will be in the affected URLs list, but isn’t always possible.

Testing Your Website for “Chunky-ness”

This isn’t exclusively an ASP.NET problem. At the end of the day, the problem Digicure experienced which resulted in slow pages was that the response was sliced into too many chunks, each of which were very small. ASP.NET just exasperated the problem by chunking every call to Write or WriteLine. This problem inherently exist in other frameworks, and can exist if your application tier excessively flushing the output, resulting in a large number of small chunks.

How do you detect if a website has this problem? That’s a little trickier. As I said, most web traffic tools like proxies or plug-ins normalize the traffic before showing it to you. The uncompress it or they de-chunk it or normalize HTTP headers. You can’t count the chunks if you can’t see them. I suggest using Wireshark. This allows you to capture all of the traffic as it is that actually transmitted and received by a network interface on your computer. (Note that on Windows, Wireshark cannot capture traffic for localhost).

Here is how you can see if a webpage is “too chunky”:

1. Load the page you want to test in your web browser.
2. Select an interface in Wireshark and start a capture.
3. Reload the page in the browser window.
4. Stop the capture in Wireshark and enter “http” into the filter to exclude everything accept we traffic.
5. Find the HTTP request, right click, and select “Follow TCP Stream”, as shown below:

If you see a large number of chunks, or very small chunks, examine your application code to see if you are flushing the output stream excessively, such as instead of a loop.

Conclusions

Chunked encoding is awesome. It allows us to transmit variable length content and use persistent connections. However each chunk has overhead, and you need to ensure your application isn’t too chunky.

I wish I could tell you that testing for “chunky-ness” is one of the 400 checks Zoompf can test your website for. Unfortunately that’s not currently the case. The HTTP library we wrote doesn’t expose chunk information, so we cannot detect this. Hopefully this is something we can change in the near future. Until, use Wireshark to test your website, and Zoompf’s free or paid offerings to find other types of performance problems.

There are a lot of interesting findings that came out of my analysis of how the Alexa Top 1000 is using HTTP compression. One finding was that JavaScript is the most common type of content served without compression. I hypothesize that this is due to websites linking to all these 3rd party JavaScript libraries and widgets. Normally a developer links to a 3rd party file to enable advertising, web analytics, user feedback and chats systems, or even social sharing widgets. But you cannot control these systems, and so those resources might not have compression enabled the way the files on your own website are enabled for compression.

Today I want to focus on a 3rd party library which wasn’t using compression because it leads to an important and often unmentioned performance lesson: Don’t link to resources inside of source code repositories.

html5shiv is a JavaScript library that enables Internet Explorer 8 and earlier to understand and properly render new HTML5 tags like <article> or <aside>. This is incredibly helpful because it allows websites to use these new semantic elements when retaining backwards compatibility. While html5shiv is open source and free to be copied and used anywhere, websites often link to a specific URL: http://html5shiv.googlecode.com/svn/trunk/html5.js.

In fact, the html5shiv article on Wikipedia includes the following source code snippet, which references this URL.

<!--[if lt IE 9]>
     <script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->

I would estimate that 95% of the time I witness a site using html5shiv, they are loading it from http://html5shiv.googlecode.com/svn/trunk/html5.js. Unfortunately, is this bad for performance for several reasons.

It’s Not Served Using HTTP Compression.

The reason I noticed this in the first place is that html5shiv.googlecode.com is not properly configured and does not serve JavaScript files using HTTP compression. Reducing the size of content is a well known category of performance optimization and HTTP compression is an easy way to accomplish this. (Well, easy in theory at least).

It’s Not Cached.

Google’s web server instructs the browser (and all shared caches) to only cache this file for 180 seconds. 180 seconds! By the time you are done reading this article, a copy of html5shiv sitting in your cache would have expired.

The whole point of linking to a commonly JavaScript file on a 3rd party instead of hosting it yourself is to use network effects and have that resource already be cached. That’s the entire reason that JavaScript CDNs like Google Libraries API and Microsoft’s (poorly named) Ajax Content Delivery Network exist. Of course it doesn’t always work. Regardless, 180 seconds is way too short to matter even if JavaScript CDNs were useful.

It’s Not the Most Recent Version.

This branch of html5shiv is not actively developed. As Paul Irish mentions, development for html5shiv has shifted to github, under the care of Alexander Farkas and others. You can find the most recent version here: https://raw.github.com/aFarkas/html5shiv/master/src/html5shiv.js.

(And no, don’t link to that either! It’s not minified, and the Content-Type header is wrong so, ironically, Internet Explorer will not execute the JavaScript that’s supposed to help it!)

This new version of the html5shiv has had numerous bug fixes and performance improvements. In addition, if you actually minified this new version of the html5shiv, you’d find it’s also smaller than older one on Google code. In short, the code at html5shiv.googlecode.com is obsolete.

It Says It Supports Range Requests When It Really Doesn’t.

Since we are on the subject, I might as well bring out all the issues. When you request html5shiv.js, the web server responds with an Accept-Ranges header, indicating that it supports HTTP byte ranging (also known as partial responses) as shown in the screen shot below:

This means the server supports conditional requests, allowing you to download just a small portion of the file. This is helpful when the client has already downloaded some of the response, such as when recovering from a network error. As I wrote in a previous post, supporting partial responses is good for performance. Only googlecode.com doesn’t support partial responses. I used REDBot (which is awesome) to detect this issue, and you can see the results here. If you send a request and specify a byte range using the Range header, the web server still gives you the full response.

The Biggest Problem of all

While these are important issues, they are really just a symptom of a much bigger problem: You’re linking to a source code repository.

Think about that for a second. You are directly linking to a resource in an external source code repository, which you do not control! Giant sirens and flashing lights should be going off in your head.

First of all, because this is a source code repository, the contents could be updated at any time. This is why there is not a far-future expires header. To function properly, this URL can never have a far future expires header. This is most likely the reason that byte range requests fail as well. The content may change, and, even though Last-Modified and ETags could be used, the web server may be able to determine whether it can return a subset of the (potentially new) response body. So, by the very nature of being a code repo, you have two performance optimizations that simply can never happen.

Secondly, this is a source code repository! How many times have you accidently checked in code to a repository that broke the build? I’ve done it lots of times. In fact, it’s so common, that many departments have funny forms of punishment when a developer breaks the build, like forcing them to wear an embarrassing hat or shooting them with automatic NERF machine guns (Skynet is nigh!)

(And the real reason this is an embarrassing hat? Comic Sans MS.)

Sure, this problem exists whenever you link to an external resource beyond your control. But it’s not like we are talking about linking to the Google Analytics JavaScript file here. A simple mistake or typo when using SVN and BOOM! Someone at Google didn’t just break the build, they broke the Internet.

Marco Arment, the creator of Instapaper, has a great podcast, Build and Analyze, over at 5by5. Sometimes he and Dan even talk about development, so I highly recommend it. One of the things he has advocated repeatedly on the show (including the most recent episode), is to always avoid external dependencies you cannot control. Direcly linking to a resource in someone else’s code repository is a perfect example of an external dependency that can easily be, and should be, avoided.

Finally, at the end of the day, we are dealing with a library that is 3854 bytes in length. The new version is 2337 bytes. When served with compression it’s only 1166 bytes. Why on earth are you linking to a 3rd party to serve 1166 bytes? In the time it takes your browser to do the DNS lookup on the html5shiv.googlecode.com hostname, you could have already transmitted the file! Creating an HTTP connection to a 3rd party to download 1166 bytes is just silly and wasteful.

The Moral of the Story

The moral of the story is: “never never never link to a resource inside a source code repository”. This includes your own source code repository and 3rd party ones like googlecode or github. The characteristics inherent in a source code repository are fundamentally at odds with frontend performance best practices. Breaking code in a source code repository or version control system is much easier due to its dynamic nature. Additionally, you should not be exposing internal infrastructure systems like code repositories, version control systems, or continuous integration systems to the public internet. This is highly dangerous and increases your attack surface.

The Moral of the Story (part 2)

The other moral of the store is one I’ve written about before: “Trust, but verify”.

Top search results will tell you to link to the copy of html5shiv.js hosted on Google’s servers. Wikipedia includes a code snippet which links to this file. And you might not think that’s a bad idea. After all this is Google we are talking about! “Let’s make the Web Faster” Google! The creator of awesome performance technologies like PageSpeed and mod_pagespeed and SPDY. The employer of frontend performance greats like Steve Souders, Patrick Meehan, and Mike Belshe. That Google should know what they are talking about.

Well, that Google still got 4 things wrong trying to serve you a JavaScript file that’s less than 4 kilobytes in size. Now I’m not trying to belittle or make fun of Google. My point is that “just because” it’s from Google, doesn’t mean its right. For that matter, “just because” it’s from Zoompf doesn’t make it right. People make mistakes. As the Buddha once said:

Believe nothing, no matter where you read it, or who has said it, not even if I have said it, unless it agrees with your own reason and your own common sense.

You can trust, but verify.

Conclusions

All of this was discovered because of a JavaScript file that was served without HTTP compression. You never know what even a minor performance issue will lead you to discover. Thius is where Zoompf can help you. Our performance scanner tests your web application for nearly 400 issues affecting web performance. You can get a free performance scan of you website now and take a look at our Zoompf WPO product.

Yesterday, frontend madman and performance nut Paul Irish reached out to me asking if I had any stats on the use HTTP compression.

I’ve written a bunch about the benefits of HTTP compression, as well as the challenges in implementing it. Surprisingly, I realized that, no, I did not have any figures about HTTP compression usage by major sites. The most recent stats I had were from the talk I gave during Velocity 2011′s Ignite sessions. The more I thought about it, I saw there were lots of interesting stats to gather beyond a raw “X number of sites use compression”. I decided to survey the top 1,000 Alexa websites and get a deeper understand of how they are using HTTP compression. The results reveal how large websites manage their content and apply the most basic web performance optimization.

Methodology

How could I get data about how HTTP compression is used by the top websites? I could certainly pull from Zoompf’s database of free scans or our paid customers, but this is not a great sample. It contained lots of the same sites scanned over and over again, and all the scans are made by people who know about performance and are actively trying to improve it. It didn’t really matter. All of the Alexa Top 1,000 websites are sadly not Zoompf customers, (yet!), so I wouldn’t even have the data anyway.

My next thought was to use the awesome HTTP Archive. After all, they’ve done the hard work of fetching all the content and provide HAR files for each site. Sadly the HTTP Archive can’t help me here for a few reasons. The first reason is that HAR, while a helpful format, has short comings. The biggest is that the response bytes are not included in the HAR file. This is a critical requirement. I wanted to determine what types of content are or are not getting compressed. The MIME type in the Content-Type header is not a reliable at all, so I would need the response bytes to determine content type.

Another reason for needing the full responses including headers and body bytes is the type of analysis I needed to do. Understanding how top website use HTTP compression is much more involved than simply checking if, say, an HTML file is lacking a Content-Encoding header. For example, HTTP compression can make certain responses larger. To determine if a website is using compression properly, I need to determine if responses that were not served using HTTP compression would truly be smaller if they were compressed. This means that I needed the response bytes to compress and see the result.

In the end, there was no getting around the fact that I needed to download the actual content from Alexa’s top websites. So how do I do it?

I started by downloading Alexa’s Top 1,000,000 sites list. This list is downloadable as a CSV file, so I needed to strip out just the hostnames, and I only wanted the first 1,000. This was accomplished with the following Linux/Unix/Cygwin commands piped together:

$ cat top-1m.csv | cut -d"," -f2 | head -n 1000 > top-1000-sites.txt

This gave me a list of just the host names of the top 1,000 sites on Alexa’s list. Next, I used an internal tool we have at Zoompf for running our performance scanner against a list of hosts. For each host, the scanner visits the home page, and downloads all dependent resources like CSS, JavaScript, images, fonts, Flash and more. This simulates exactly what a visitor’s web browser would do when accessing the main page for the site. I ran the bulk scanner on the list of the Alexa top 1,000 sites to download all this content. This took about an hour and half from my dev box. Then I analyzed the data. Basically, this involved opening the scan data for each scan, and examining each response. In total, I examined 90,517 responses served from 4,597 distinct hostnames.

First I needed to determine which responses could be compressible. This means I was looking for a response whose format is not natively compressed, and so it could be compressed with HTTP compression. As I discussed in the Lose the Wait: HTTP Compression post, this includes more than just text responses like HTML or CSS. Luckily we have built a pretty large database of common web file formats internally at Zoompf, which includes an attribute about whether the format is natively compressed. I simply examined the bytes, determined the file format, and did a quick check to see if it was natively compressed. I also took various measurements including how much content could be compressed, what its type was, and where it came from.

Big Questions To Answer

I wanted to use all this data to answer some big questions. Specifically, I wanted to know:

• How much of the content was compressible, and how much was actually getting compressed?
• What types of files get compressed more or less often than others? For example, are most people compressing HTML but forgetting CSS?
• How are websites doing as a whole? Are some websites applying compression perfectly and some completely failing? Or is it that all websites are overlooking one or two little things?

Ultimately, I wanted to use these answer to try and determine broader answers about how performance optimization is implemented at big companies and why.

The Findings

Of the 90,517 responses I examined, only 14,316 responses (15.81%) were compressible. This is an interesting stat, because it goes to show how much of the web is dominated by binary content like images. This is why I’m a big proponent of image optimization, and it’s nice to see the topic of image optimization on the radar of more mainstream tech bloggers like Daring Fireball’s John Gruber and Jeremy Keith. As I said in my Take it all off: Lossy Image Optimizations talk at Velocity 2011, a 20% reduction in image size has more of an effect on total content size than an 80% reduction of text content.

Let’s dig into those compressible responses. There are 3 categories that a compressible response can fit in:

1. Properly Compressed Responses – A response that could be HTTP compressed, and was in fact served to Zoompf using HTTP compression. For example, a CSS file which is served with HTTP compression. This is a good category, since the content owner is optimally serving the content.
2. Properly Uncompressed Responses – A response which is not natively compressed, but compressing it makes the response larger. For example, a small HTML file which is actually larger when served compressed. This is a good category, since the content owner is optimally serving the content.
3. Responses Missing Compression – A response which is not natively compressed, which should be compressed, and which will be smaller if HTTP compression is used, but which is not compressed. For example, a SVG image served without compression, since SVG images are not natively compressed. This is a bad category, because the content owner is inefficiently delivering content to the client.

You can see the breakdown of responses in the graph and table below:

So, across all Alexa Top 1,000 websites, 23.37% of content is not getting compressed. The median savings if compression was used would be 4.4 kilobytes with is a median savings of 62.3%. That’s quite a bit of savings, considering HTTP compression has been a well known, supported, and recommend optimization for over 15 years.

How are sites themselves doing? Of the Alexa top 1,000 websites, 642 of them are serving at least 1 item that is missing HTTP compression. In other words, 64% of the Alexa Top 1,000 are not properly applying HTTP compression.

The median number of responses missing HTTP compression per website was 2, so it’s not like the occasional, single response is getting through without compression. This is a larger issue.

To try and understand why content is slipping through, we first need to know which types of content isn’t being compressed. The table below shows this.

(Generic Binary and Generic Text are responses whose format Zoompf could not determine. This typically indicates an incorrect MIME type, and we could not conclusively determine a type by examining the first 500 characters or so. Manually spot checking revealed that most of these responses were JavaScript files served with an incorrect MIME type and which had a mix of HTML tags in them which confused our scanner. For purposes of analysis, I will ignore these are not count them as any other file type.)

This table tells us lots of interesting things about how content is compressed by the Alexa Top 1,000:

1. Approximately 20% of all HTML, JavaScript and CSS files are served without compression. Major websites are having real problems compressing even the most basic and common types of compressible content.
2. JavaScript is the single largest source of compressible content, yet it is served compressed less often than CSS or HTML. I believe this is due to the widespread use of 3rd party libraries and widgets which are served from a website you don’t control. While people can configure their own sites to compress content, 3rd parties serving their JavaScript files appear to be using compression less often than other sites. Based on their URLs, the majority of JavaScript resources missing compression appear to be for analytics scripts.
3. Lesser known compressible file types are forgotten far more often than HTML, JavaScript, or CSS. While there are fewer instances of these formats on the web, they are more than twice as likely to be uncompressed.
4. Atom feeds are not very popular.
5. Someone is using BMP images on an Alexa Top Website? Wow. At least some of them are being served with compression.
6. SVG images are present largely as fallback for web fonts.

Another interesting area are 404s. 404s are often overlooked because, while the request might be for a file like logo.png, the response is HTML. If your server is not configured properly, it will see the file extension .png and not apply compression, even the response is compressible text.Of the 1513 response which had a 404 status code, 490 of them were not compressed. In other words, 32.4% of all 404 handlers in the Alexa Top 1,000 are not using HTTP compression properly.

As I wrote about before, the use of Content-Type: deflate is incredibly problematic and broken. Luckily I did not see it in wide use in the Alexa Top 1,000. Of the 8825 response I saw which were using HTTP compression, only 23 were using DEFLATE. What is troubling is that, based on the HTTP response headers, virtually all of these Content-Type: deflate responses came from a Juniper Network’s DX series load balancer or web accelerator. A company like Juniper should know better.

See For Yourself

I’m a big believer in transparency, so here are my raw data files so you can review it yourself. In fact, if you work at an Alexa Top 1,000 site, there is even a text file full of exactly which URLs are missing compression!

• Website stats – Each website, how many responses are compressed, etc
• Items missing compression by file type
• URLs missing compression – List of all sites, and the URLs of responses that are not served with compression.

What Does This All Mean?

This data supports much of what I discussed in the Lose The Wait: HTTP Compression post. Specifically:

• HTTP compression, though easy in theory, is not properly implemented in practice. The majority of Alexa Top 1,000 websites are not completely implementing HTTP compression.
• The most commonly compressed content, (HTML, CSS, and JavaScript) are not properly compressed 20% of the time. This is most likely due to incorrect configuring the web server to use HTTP compression based on incorrect or missing file extensions, and incorrect or missing MIME types.
• Less common text formats, like RSS and XML, are more than twice as likely to be served uncompressed. People are forgetting about these files, and common configuration examples on the web exclude them.
• Non-natively compressed files formats, such as ICO, SVG, and various font files are more than twice as likely to be served without HTTP compression. People are forgetting about these files, and common configuration examples on the web exclude them.
• Nearly 1/3 of all 404s handlers do not use HTTP compression. This figure is nearly 50% higher than the 20% of regular, non-404 HTML files which are served without HTTP compression. This is most likely caused by web servers configured to use the requested URL’s file extension to decide if HTTP compression should be used for the response.

As I wrote about before, the use of Content-Type: deflate is incredibly problematic and broken. Luckily I did not see it in wide use in the Alexa Top 1000. Of the 8825 response I saw which were using HTTP compression, only 23 were using DEFLATE. What is troubling is that, based on the HTTP response headers, virtually all of these Content-Type: deflate responses came from a Juniper Network’s DX series of load balancers and web accelerators. A company like Juniper should know better.

This data also reinforces a great quote from Mike Belshe, one of the creators of SPDY, about optional features:

“Experience shows that if we make features optional, we lose them altogether due to implementations that don’t implement them, bugs in implementations, and bugs in the design.” – Mike Belshe

Compression was an optional after thought for HTTP, and so 20 years later we still have problems using HTTP compression appropriately.

The Million Dollar Question

What was startling when I really dug into the numbers was who seemed to be having the most trouble, and how widespread it was. There are a number of very large websites with obviously capable staffs were not compressing the majority of their content. For example, major news sites like The Washington Post, ABC News, The New York Post, CNBC, Sky in the UK, and NPR all served 80%-90% of compressible resources without compression. Even the Pirate Bay has 52 RSS feeds referenced on its main page using <link> tags, and none of them are compressed. The Japanese Social networking site Ameblo has an RSS feed of user activity that’s over 4 Megs.

So why is this happening? The guys who run The Pirate Bay are technical geniuses. The IT departments and budgets for ABC or The Washington Post are huge. How is it that the biggest and most popular websites in the world, who have the most to gain and the most to lose from web performance, and who are the best equipped, staffed, and funded to solve these problems, can’t seem to solve these problems?

This is the million dollar question. And it is literally the million dollar question, because it’s costing these websites millions of dollars.

How is it that the biggest and most popular websites in the world, who have the most to gain and the most to lose from web performance, and who are the best equipped, staffed, and funded to solve these problems, can’t seem to solve these problems?

The short answer is “They don’t know they have a problem.” I know it’s true because whenever I tell one of these websites about the problem, their immediate answer is “we aren’t? Crap we should. Let’s fix that”. And then they do.

Frankly, this reaction opens up an entirely different and very scary box. Because “They don’t know they have a problem” is just another way of saying “testing for front-end performance issues is very immature even at the largest organization.” That is an incredibly huge problem for our industry with many different facets. Its too big of a topic to stick at the tail end of this post, so I’ll put all my thoughts on this in a future blog post.

More Testing

More testing is clearly needed. The results for font files are new and interesting to me. While I know fonts like WOFF are natively compressed, it is interesting that OTF and others can be compressed using HTTP compression. This indicates no native compression, or a poor compression scheme so much so that a second pass of deflate makes it smaller. Also, the use of HTTP compression should be something that is charted over time, to see if we are improving. My last figure, from 2010, showed 78% of Alexa Top 1000 sites having at least 1 resource served without compression by the HTTP. Perhaps this should be added to the HTTP Archive.

If you want to find out whether your site is properly applying HTTP compression, Zoompf offers a free performance scan of you website. HTTP Compression, and the various implementing issues surrounding it, are just a few of the nearly 400 performance issues Zoompf detects when testing your web applications. You can also take look at our Zoompf WPO product.

Today Google released their HTML/CSS Style Guide. While it is full of great advice to help manage a growing code base among multiple developers, I thought it would be interested to review the web performance implications of each of its recommendations. Despite being a just style guide, nearly all of its rules had performance ramifications. For each rule I will:

• Briefly quote Google’s style guide.
• Link to the rule for more information
• Discuss how the rule affects performances (either positively or negatively)
• Mention any other related performance rules, and whether Zoompf’s free or paid offering currently detects the performance issue.

Let’s get started!

Omit the protocol from embedded resources.

Omit the protocol portion (http:, https:) from URLs pointing to images and other media files, style sheets, and scripts unless the respective files are not available over both protocols. (link)

This is a great recommendation for performance. I’ve talked about the advantages of protocol relative URLs in the past. Not only do they both reduce page size, protocol relative URLs prevent you from needlessly creating SSL connections. As an added benefit, protocol relative URLs are a bulletproof way to prevent mixed content warnings.

Applicable Zoompf Checks:

• SSL Resource on non-SSL Page
• Excessive Absolute URLs (Same Site)
• Excessive Absolute URLs (External Links)

Bonus Performance Tip. While this advice is about linking resources, Zoompf recommends to you relative URLs for web content. So instead of linking to http://zoompf.com/blog/some-awesome-post you would use /blog/some-awesome-post. Not only does this reduce page size, but makes it easier to transfer content to new hostnames or restructure your site in the future.

Indent by 2 spaces at a time

Indent by 2 spaces at a time. Don’t use tabs or mix tabs and spaces for indentation. (link)

More of a style rule than performance. You should be minifying your HTML when you move it into production anyway which removes unneeded whitespace and resolves this problem.

Applicable Zoompf Checks:

• Dynamic HTML Not Minified
• Static HTML Not Minified

Use only lowercase

All code has to be lowercase: this applies to element names, attributes, attribute values (unless text/CDATA), selectors, properties, and property values (with the exception of strings). (link)

Using lowercase in HTML makes your pages more compressible, resulting in smaller pages.

Zoompf does not check specifically for this issue but does normalize your HTML to lowercase as part of our HTML minification analysis.

Trailing whitespace

Trailing white spaces are unnecessary and can complicate diffs. (link)

A good style rule, and since the document mentions diffs, this is mainly to make internal code management easier. As I said before, post processing your HTML, or minifying it when promoting it to the production environment resolves this issue and can slightly reduce the size of a page.

Applicable Zoompf Checks:

• Dynamic HTML Not Minified
• Static HTML Not Minified

Use UTF-8 (no BOM)

Make sure your editor uses UTF-8 as character encoding, without a byte order mark. Specify the encoding in HTML templates and documents via <meta charset="utf-8">. Do not specify the encoding of style sheets as these assume UTF-8. (link)

Standardizing on a single encoding is a good idea, and getting ride of the BOM saves 3 bytes. However I don’t suggest using anything inside the HTML, like <meta charset="utf-8">, to specify the charset. The Content-Type header should be the only thing you use to tell the browser about the charset. Encoding inside the document what the encoding format for a document is creates a chicken or the egg situation where the browser must parse the document to figure out how to parse the document. This creates performance problems and exposes you to XSS attacks. Google’s own documentation even references Zoompf for this.

Applicable Zoompf Checks:

• Unnecessary <meta> Character Set (http-equiv)
• Unnecessary <meta> Character Set (charset)

Comments

Explain code as needed, where possible. (link)

Comments are absolutely important, but they should not make it into production code. They needlessly increase the size of the resource, and often expose you to security risks.

For things like CSS and JavaScript, minification will remove the comments for your automatically. For HTML, especially dynamically generated HTML, consider using the code comments in the server-side language instead of HTML comments.

So instead of this:

    ...
    <a href="/public/">public</a>
    <!--
    Feature is not ready yet, disable until next release cycle
    <a href="/public/">public</a>
    -->
    ...

Replace it with something like this:

    ...
    <a href="/public/">public</a>
    <?php
    /*
    Feature is not ready yet, disable until next release cycle
    <a href="/public/">public</a>
    */
    ?>
    ...

Applicable Zoompf Checks:

• Bloated HTML Tag (<comment>)
• Excessive HTML Comments
• Commented Out HTML
• Dynamic HTML Not Minified
• Static HTML Not Minified
• JavaScript Not Minified (External File)
• CSS Not Minified (External File)
• JavaScript Not Minified (<script>)
• CSS Not Minified (<style>)
• Feed Not Minified

Action items

Highlight todos by using the keyword TODO only, not other common formats like @@. Append a contact (username or mailing list) in parentheses as with the format TODO(contact). (link)

TODOs are a quick way to remind you of additional tasks. Just make sure that TODOs are not included in content sent to the user. Since TODOs are placed inside of comments minification fixes this issue.

Zoompf does not check for TODO items specifically, but this issue will be flagged more generically by Zoompf comments and minification checks.

Use HTML5

HTML5 (HTML syntax) is preferred for all HTML documents: ‘<!DOCTYPE html>’. (It is recommended to use HTML, as ‘text/html’. Do not use XHTML. XHTML, as ‘application/xhtml+xml’, lacks both browser and infrastructure support and offers less room for optimization than HTML.) (link)

Smaller DOCTYPE means smaller bytes. XHML is also more bloated than HTML, requiring you to use closing tags and <CDATA> which [needlessly increases the size of your page](http://perfectionkills.com/optimizing-html/). Avoid XHTML, especially on mobile sites.

Zoompf does not currently check for HTML5 use, or mobile sites using XHTML.

Use HTML according to its purpose

Use elements for what they have been created for. (link)

Google’s example for this rule shows a <div> using a onclick to replicate a standard <a href=""> tag. Another common example is that developers with use an <a> tag with an empty href attribute, but use an inline onclick handler which simply navigates to a web page.Reinventing the wheel like that almost always results in larger markup, increase the size of the page.

Zoompf does not check for using elements properly.

Separate structure from presentation from behavior

Strictly keep structure (markup), presentation (styling), and behavior (scripting) apart, and try to keep the interaction between the three to an absolute minimum. (link)

Placing JavaScript and CSS into separate external files allows you to reuse common content and reduce the size of all of your pages. These external files can also be cached while HTML often cannot.

This is another good example of advice that is good for development and not good for production. Despite the guide advice, Developers can and should separate code into files as it logically makes sense, such as separating libraries. The same is true for CSS. However, you should not reference 8 JavaScript files in the HTML when the website moves to production. Those files should be combined and versioned as part of the publishing process.

Bonus Tip. Smaller files should be inlined into the document. Software like mod_pagespeed can do this automatically.

Applicable Zoompf Checks:

• Common Style Attribute
• Common Content (JavaScript Block)
• Common Content (Style Block)

Do not use entity references

There is no need to use entity references like —, ”, or ☺, assuming the same encoding (UTF-8) is used for files and editors as well as among teams. (link)

All of these symbols appear in UTF-8, so there is no reason to use HTML entities, reducing the size of your page.

Zoompf does not currently check for using entity references on UTF-8 encoded pages.

Omit optional tags (optional)

For file size optimization and scannability purposes, consider omitting optional tags. (link)

I disagree with this advice. While it makes sense to omit end tags of elements that have no meaningful inner HTML (<meta>, <link>, <img>, <script src="">), you are just asking for maintainence issues if you start omitting things like </p>. In fact, Google’s example of recommended HTML usage is actually a better example why this is a stupid practice.

If you truly want to do something like this, don’t try to do it in your source HTML or template files. Add these aggressive HTML minification optimizations as part of your publishing process when moving a website to a production environment.

Zoompf does not check for optional end tags being used.

Use valid CSS where possible

Unless dealing with CSS validator bugs or requiring proprietary syntax, use valid CSS code. (link)

CSS minification tools are very primitive. While JavaScript minifiers like YUI Compressor or Closure Compiler actually tokenize, parse, and build syntax trees to guide minification, CSS minifiers use regular expressions or primitive lexers. Invalid CSS often breaks these tools. Make sure your CSS is valid.

While Zoompf does not check for invalid CSS syntax, it does check to make sure that CSS resources referenced with a <link> tag or @import statement are, in fact, CSS files.

Use appropriate lengths for ID and class names

Use ID and class names that are as short as possible but as long as necessary. Try to convey what an ID or class is about while being as brief as possible. (link)

Shorter names are better than long names for performance. Zoompf does not currently check for long ID or class names.

Avoid qualifying ID and class names with type selectors

Unless necessary (for example with helper classes), do not use element names in conjunction with IDs or classes. (link)

In other words, ul#ref-list is not needed when #ref-list will do. This has three performance implications. First, simplifying how CSS selectors match speeds up rendering. Additionally, the rule is more generic, allowing it to be reused in more situtations and prevents the creation of redundant rules. Finally, smaller selectors mean smaller CSS files, which is always good for performance.

Zoompf does not currently check the specificity of CSS rules.

Omit Default or Unneeded Values

Google’s document lists a number of rules for omiting default or unneeded values from CSS. All of these rules lead to smaller CSS files, which is a good thing for performance. All of these rules can be automatically fixed with CSS minification.

Applicable Zoompf Checks:

• CSS Not Minifed (External File)
• CSS Not Minifed (<style>)

Conclusions

Every organization should use a style guide. For the most part, Google’s HTML/CSS Style Guide promotes good style advice that is also good performance advice, making it a good standard to use. Just remember, you should not make things easy for developers and the expense of your visitors. At the same time, don’t force a style on the developers that makes code or content harder to undestand for the sake of performance. Instead, use a build process to apply many of these optimizations automatically.

Want to see what performance problems your website has? I’ve mentioned only a few of the nearly 400 performance issues Zoompf detects when testing your web applications. You can get a free performance scan of you website now and take a look at our Zoompf WPO product at Zoompf.com today!

Sunday was Earth Day. As with many holidays or anniversaries, Google celebrated Earth Day with an animated doodle. This is certainly not the first animated doodle that Google has done. Since I have been speaking so much about images recently, I thought it would be interesting to see how Google creates small and fast animations. The solution surprised me, and changed my mind about animated images on the web.

What Google is doing

The Google Earth Day animation is a non-repeating linear sequence of images. Unlike previous animated doodles, it is not interactive. The animation is 486 pixels wide by 182 pixels tall. Traditionally, this could be accomplished using an animated GIF image. In fact, that is what I thought they were doing at first. However Google took what I think is a different and better approach.

Instead of an animated GIF Google stored all the animation frames vertically inside a single JPEG image that is 468 pixels wide by 2912 pixels tall. There are 16 frames of animation stored this way, somewhat resembling a strip of film, as shown in the screen shot below:

This is kind of like a CSS sprite, but each image is a frame of animation. To clarify things, I will call this animation technique filmstrip animations. Another animated doodle which uses filmstrip animation is this Gumby logo from October 2011.

So how does Google make the filmstrip animation work? First they enclose the <img> tag JPEG with all the animation frames inside a <div> tag that has a fixed height of 182 pixels and which hides overflow. This creates a fixed window so to speak, which masks all but current animation frame. The image is animated using JavaScript, which changes the top property for the absolutely positioned image to slide it up a fixed interval with the setTimeout() function. A diagram of this concept is shown below:

The code for this looks something like this:

<div style="height:182px;position:relative;width:468px;overflow:hidden">
	<img border="0" src="source.jpg" id="filmstrip" style="position: absolute; height: 2912px; top: -0px; display: block; ">
</div>
<script>
function naiveAnimation(id) {
	var img = document.getElementById(id);
	var offset = 0;
	var animate = function() {
		//slide the image correct frame of animation given by  offset
		img.style.top = -offset + "px";
        //calculate offset to next frame
		offset = Math.floor(offset + 182);
		//if we are not yet on the last frame...
		if(offset < 2912) {
			//call me again in half a second
			window.setTimeout(animate, 500);
		} else {
			//at last frame, so all done!
		}
	};
	//start the animation
	animate();
}
naiveAnimation('filmstrip');
</script>

However, this animation is jerky compared to Google’s Earth Day. In Google’s animation, the frames seem to fade into each other. How is Google doing this?

Generating Intermediate Frames

Google uses a neat trick. They use opacity to fade between two different frames of animation and create, in effect, additional intermediate animation frames. This makes the animation look smoother.

To accomplish this, Google uses two <img> tags inside of the container <div> and uses absolute positioning and z-index to place them directly on top of each other. You can see this clearly using Firefox’s 3D DOM viewer, as shown in the screen shot below:

Each image is pointing at the filmstrip image. Let’s call the “lower” image tag <img> #1 and the “upper” image tag <img> #2. To accomplish the effect, <img> #1′s top property is set to show animation frame 1 and <img> #2 is set to show animation frame 2. The opacity property of <img> #2 is set to 0, making it full transparent so animation frame 2 is not seen. This setup is shown in the diagram below:

Google then uses a fast JavaScript timer to increase the increase the opacity of <img> #2 every 50 milliseconds. Increasing the image’s opacity makes it less transparent, so more and more of <img> #2 is visible and less of <img> #1 is visible. This makes animation frame 2 fade in on top of animation frame 1, as shown in the screen shot below:

Google slowly fades the image in over 10 steps of 50 milliseconds. After 500 milliseconds:

• <img> #2, displaying animation frame 2, is fully opaque and covers <img> #1.
• The top property of <img> #1 (which is now hidden by the fully opaque <img> #2 ) is increased so that <img> #1 now points at animation frame 3.
• JavaScript is now used to decrease the opacity of <img> #2 every 50 milliseconds This means <img> #2 (displaying animation frame 2) slowly becomes transparent, allowing <img> #1 (now displaying animation frame 3) to fade into view.
• The process repeats

In short, using two images, changing opacity up and down, and moving the position of the two copies of the same filmstrip images, Google can create an animation with fading transitions between frames!

But what about animated GIFs?

My initial thought was that Google should be using an animated GIF. Why didn’t they? To find out I needed to create a GIF animation of the same content as the Earth Day filmstrip animation and compare the two. I took the filmstrip JPEG and sliced it into 16 separate images using Imagemagick’s convert -crop command. To create the animated GIF, I used GIMP, inserting each animation frame as a separate layer of a single image. I then exported it as an animated GIF with 500 milliseconds between frames. The resulting animation is shown below:

The animated GIF approach was bad for several reasons.

• GIF’s are not ideal at storing photographic images and all the frames of the animation must share the same palette of 256 colors.
• The resulting GIF image is huge at 980 KB! GIF animations can be optimized to store the differences between frames. However, since my source image was a JPEG, the color values for pixels vary between each frame. In other words, the color value for the dirt color in the top left corner of each animation frame is different. This means there is little commonality between frames, so more graphics data must be stored for each frame, resulting in a large image. Since the original sources of each animation frame were photographs taken of a 3 dimensional biological scene (with different lighting effects and shadows, etc), I think it’s unlikely that having access to the original source images would have allowed me to create a more optimized animation.
• The animation is jerky. It is only 16 frames and after one complete frame is shown it is replaced with next complete frame 500 milliseconds later. The animation runs at 2 frames per second.

Contrast this with Google’s JPEG filmstrip approach:

• The image is a JPEG, allowing millions of colors and photographic realism.
• The JPEG animation is only 270 KB, plus a few hundred bytes of HTML and JavaScript.
• Google’s animation is much smoother. Even though the JPEG contains only 16 frames, the fading transition between frames occurs in 10 “steps.”. This essentially creates a 160 frame animation running 20 frames per second. Our animated GIF would grow significantly larger to create this same effect.

A New Animation Format?

Animated transitions have been popular with image galleries and carousels. However, the intended effect was not to create a single smooth piece of animation. By coupling tradition transition animation with a filmstrip of animation frames, Google has created a new and highly adaptive animation format for the web which is supported by every modern browser. A single image is used to store the animation frames and JavaScript advances the frames and drives addition effects. Consider the benefits of filmstrip animations:

• Near universal compatibility. No new image format needs to be supported.
• Use the best image format for the job. Need photorealism? Use a JPEG. Need an animation with alpha transparency? Use PNG! GIF animations cannot adapt like this.
• Reduced size. The number of animation frames can be reduced by using JavaScript to generate intermediate frames. In our example, Google used opacity for fading, but this is just one method. Wipes of any kind are possible, in addition to moving specific parts of the image.
• JavaScript can adapt to the platform. Look at navigator.network.connection and see you are on a slow network? You can respond and fetch a lower quality filmstrip image, or one with fewer animation frames.
• Better control over frame order. With animated GIFs, you move from the first to the last frame, and then optionally start over again at frame 1. With JavaScript you can go from the start to the end, and then backwards for the end to the start. This allows you to store symmetrical animations in half the space. Or you could revisit only certain frames. This is more flexible, allows for better transitions, and avoid sharp jumps when the animation repeats.

But it requires JavaScript!

No, not really. You can still have an animated GIF inside of a <noscript> tag like this:

<div id="filmstrip-con">
	<script>
	document.write("<img src='filmstrip.jpg' id='filmstrip-img'>);
	<noscript>
	<img src="animated.gif">
	</noscript>
</div>

(Yes, document.write is a bad idea here, but I’m using it for brevity.)

This is a big win all around. If even only 10% of your visitors have JavaScript enabled, that is 10% of visitors who don’t need to download a 1 megabyte GIF image. Anyone who doesn’t have JavaScript enabled can still see the animation, it will just load slower and potentially provide a lesser experience than the filmstrip animation.

But Why Not a CSS Animation?

If GIF animations are one extreme, CSS animations are at the other end. I don’t like CSS animations for a few reasons. The first is that browser compatibility, especially backwards compatibility is lacking. But the biggest is that the client is the one creating the animation. This is because CSS animations are extremely granular and microscopic. “Move this element in this way, at this time. Move this other element a different way at a different time.” With animated GIFs or the filmstrip approach, the content of the frames and how it changes are generated by the author, and a series of static images are presented to the client to draw.

With CSS animations, the animations are created on the client, so the computing power of the device greatly affects the experience. If the browser does not support hardware accelerated CSS transforms CSS animations are jerky and provide a poor experience. Additionally, all that microscopic “Move X to Y” makes creating and maintaining and updating animations difficult. While the filmstrip approach is more involved than simply exporting an animated GIF, it is clearly easier than composing an animation from DOM elements and coordinating their movement.

Filmstripper: A naive implementation

If you’d like to use filmstrip images, I’ve create a small proof of concept, filmstripper, that abstracts away specifics for you. The focus is on being as unobtrusive as possible. That means no external library dependencies and minimal markup or required tag decorations. Even though it’s fairly primitive (I haven’t added Google’s fading effect yet) this approach works well. If JavaScript is not enabled or an error occurs, a static image is shown. This allows animation to be added responsively.

To use filmstripper, create a filmstrip image with all the animation frames arranged vertically. Next, create a static image just containing the first frame of the animation. The code HTML is minimal:

<div id="filmstrip-con">
	<img src="static.jpg" id="filmstrip-img">
</div>

To animate, just call filmstripper(). The code is below:

/*
 * filmstripper - Simple filmstrip based animations
 *
 * contID - ID of DIV container element
 * imageID - ID of IMG element to animate
 * stripImageUrl - URL to the filmstrip image (frames arranged vertically)
 * delay - delay in milliseconds between frames
 */
function filmstripper(contID, imageID, stripImageUrl, delay) {
	var con = document.getElementById(contID);
	var img = document.getElementById(imageID);
	var frameHeight = img.height;
	var throwAway = new Image();
	var setup = function() {
		//setup the div to hide the strip
		con.style.cssText = "height:" + frameHeight + "px;position:relative;width:" + img.width + "px;overflow:hidden";
		//setup the image to scroll frames, and switch static to strip
		img.style.cssText = "position: absolute; height: " + throwAway.height + "px; top: -0px; display: block;";
		img.src = throwAway.src;
		var d = 0;
		animate = function() {
			img.style.top = -d + "px";
			d = Math.floor(d + frameHeight);
			if(d < throwAway.height) {
				window.setTimeout(animate, delay);
			}
		};
		animate();
	};
	//request the film strip image
	throwAway.src = stripImageUrl;
	//launch everything if it loads
	throwAway.onload = setup;
}

Conclusions

Animated GIFs, while simple and widely supported, have several short comings. Google uses a single image containing arranged as a filmstrip and JavaScript to advance to animation frames. This approach allows for better, smoother animations which require less content to be delivered to the browser. If you have large animated GIFs, consider replacing them a filmstrip animation instead.

Want to see what performance problems your website has? Zoompf can analyzed your website for nearly 400 issues which affect web performance and load time. You can get a free performance scan of you website now and take a look at our Zoompf WPO product at Zoompf.com today!

As I mentioned in our How Fast Is … USPS.com video and blog post yesterday, I discovered a few TIFF files on the US Postal Service’s website. I thought a follow up post about images suitable for use on the web was in order. According to the awesome HTTP Archive, the most common image formats on the web are PNG, GIF, and JPEG:

Together these images can handle the most common image use cases on the web:

• GIF – simple animations
• PNG- Figures, diagrams, screen shots, basic images
• JPEG – Photographs

These formals are so common and entrenched that the processors and SoCs used in mobile devices are hardware optimized for things like JPEG decoding or the checksums used from PNG’s DEFLATE compression scheme. And frankly, until there is a format which accomplishes these common use cases better than an existing image type, this will not change. This “what we have works fine” mentality is a big reason APNG and other GIF animation replacements have failed to take off. They are not significantly better at solving the simple animation use case, so they do not get adopted.

Unsuitable Image Formats

If these are the most common and suitable formats to solve various image use cases on the web, then what about other image formats? What make up that “1% other” on the chart from the HTTP Archive? By and large, these other image formats have one of the following characteristics that make them unsuitable for use on the web:

1. They are not natively supported by most browsers.
2. They do not compress graphical data, do not compress graphics data by default, or compresses graphics data uses inefficient or obsolete methods.
3. They contains additional information not relevant for display on a monitor.

Characteristic 1 without a doubt kills off almost any image formats you have heard of. Characteristics 2 and 3 make those image formats that are compatible wasteful. So, while there are dozens of image file formats that are unsuitable for the web, the types of unsuitable image formats you do find on the web tend be compatible (or were compatible at some point in the past), but are wasteful and bloated in size. We will focus on 3 types of image format that meet this criteria.

TIFF Images

The TIFF images I found on USPS.com are a good example of this. At one time, browsers did support them. But today TIFF images cannot be natively rendered by modern web browsers and require a plug-in to be visual. TIFF images are not compressed by default. TIFF images are also widely used in the publishing industry so they often use alternative color spaces like CMYK and contain printing information such spacing, lay out, and density. All of this makes TIFF images unsuitable for use on the web.

BMP Images

BMP images are another example of an unsuitable image format for use on the web. BMP images do not compress graphical data. While a primitive form of RLE compression is defined in the specification, this is not widely supported and rarely used. Even if it was, RLE achieves poor compression ratios compared to a more modern scheme such as DEFLATE. Images that, when saved as a BMP images will be hundreds of kilobytes are only tens of kilobytes when saved as a PNG or JPEG. With such huge sizes, BMP is definitely not a suitable image format for the web.

XBM Images

Besides TIFF and BMP, a final image format encountered on the Internet which is not unsuitable is XBM. Ironically, XBM was the very first image format used on the web and support started with the Mosaic web browser. Subsequent web browsers supported XBM to be compatible with early websites. Sadly, XBM is, to put it nicely, a horribly horribly horribly designed image format. If you can imagine all the badness of the DOM’s document.cookie “interface” somehow packaged into a image format, you’d get XBM.

Each XBM file is, quite literally, valid C source code, defining a byte array and the bytes values to populate it. In addition to not being natively compressed, this approach creates a large number of security holes as the format essentially says:

Hey web browser, I’m totally untrusted content from some random 3rd party you found on the Internet, but I’d like you to allocate an array in memory that is X bytes big, and then give me a pointer to it. Now I’m going to shove some number of bytes (who knows how many!) directly into memory starting at that address.

Security vulnerabilities like buffer overflows, heap overflows, and uninitialized memory leakage cropped up everywhere! It was so bad, that to fight the flood of issues Microsoft removed support for the image format back in XP Service Pack 2. Even in late-2008, before I stopped doing security research, I quite by accident discovered an information leakage vulnerability in Firefox using XBM images. XBM is exceedingly rare today, though Zoompf has encountered several websites with them through our free scans and from customers using our WPO product.

Why Do Unsuitable Image Formats Get Used?

If these formats are a poor fit for web usage, why do they get used on websites? There are a few reasons:

• Legacy. Once upon a time, that image format worked. The site is old, and nobody fixed it. This is common with XBM and BMP.
• Mistake. People just make mistakes. Sometimes they didn’t know they shouldn’t use one of the image types. Usually someone didn’t intend to use an unsuitable image format. 9 times out of 10, when Zoompf detects a BMP image, it has a file extension of .jpg or .png. The creator intended to save the image as a JPEG, but the actual image format that was used was BMP.
• Actually meant to use it. If you want to provide a downloadable version of your logo that is high resolution and suitable for printing, you probably do want to publish it as TIFF. If you have a collection of wallpaper images to use as a desktop background, you probably want to offer them as BMP. Having a TIFF or a BMP on your website isn’t a bad thing as long as it is downloadable. The performance problem starts when you use these image formats inside of an <img src> or a CSS background: style, because there are faster, better, and more efficient image formats you should be using.

Conclusions

You should only be using PNG, JPEG, and PNG images on production websites. All other image formats are either not supported by browsers, or are not optimized and efficient for use as images on a website. Remember, just because a filename has a .png, .jpg, or .gif extension does not mean its a PNG, JPEG, or PNG image! Review the images used by your website and 3rd party content to ensure you are using the proper image formats.

Want to see what performance problems your website has? Unsuitable Web Image (TIFF), Unsuitable Web Image (BMP), and Unsuitable Web Image (XBM) are just 3 of the nearly 400 performance issues Zoompf detects when testing your web applications. You can get a free performance scan of you website now and take a look at our Zoompf WPO product at Zoompf.com today!

Our regular video series How Fast Is…? examines real world websites and details the cause of their performances issues as well as what should be done to solve them. After all, the best way to learn about front-end web performance is to see what other people are doing right and doing wrong. In this edition of How Fast Is…? we analyze the US Portal Service’s website.

Why USPS.com?

It’s tax time again in the United States. I was at the post office this week mailing in my tax forms and as I left I saw an interesting poster:

One million people visit the US Postal Services website a day! That’s a lot of traffic! I imagine that during tax season this number increases as people try to locate post offices and are researching things like certified mail and return receipts. Since the USPS website is so important, I decided to see the types of web performance best practices they were implementing, and see there were additional optimizations to be made.

For this video, I crawled and analyzed just over 1100 pages of the US Postal Service’s website.

Interesting Lessons

I’ve been talking a lot about HTTP compression as part of Zoompf’s Lose the Wait blog series. Often people do are not compressing what should be compressed. One of the challenges with properly configuring a web service for HTTP compression is that it is op-in. You must specific specify which files, either by MIME type or file extension that should be compressed. If you haven’t conducting an inventory to understand all the different types of content your website serves, you may forget to include something.

This is exactly what happen to USPS.com. They have a large number of downloadable spreadsheets with shipping rates for various packages based on size, dimensions, and shipping locations. These spreadsheets are saved as CSV files and are served with the MIME type text/csv. USPS is not compressing these files, even though they are non-natively compressed text files which should be served with HTTP compression. Additionally, USPS is not compressing any of their 404 pages, another common mistake.

Unexpected Findings

Zoompf’s crawler found two enormous TIFF image files on USPS.com. TIFF images are typically used for high quality printing. In fact, most web browsers do native support TIFF images, and instead rely on plug-ins to display TIFF images, as shown in the screen shot below:

This is so odd and unexpected in fact, that Zoompf does not even have a check for websites which use TIFF images. Zoompf does flag when a BMP image is used a website. Using TIFF and BMP images is a bad idea, because both images do not always compress their graphical data. BMP images are not compressed at all. (While the file format does allow for a primitive type of run length encoding, it is not required and support is almost non-existent). TIFF images do support different compression modes, from lossless to lossy, but often do not use compression to avoid compression artifacts as they are used in high resolution printing. As soon as I’m done with this post, I’ll add a check to Zoompf for TIFF images.

Another odd finding was that not only were images not cached, they included a Cache-Control: must-revalidate header. This directive forces the browser to always send a conditional request for a resource before using it. Even if USPS.com started using HTTP caching for its resources, it would need to also remove the must-revalidate directive for caching to function properly. The only time you usually must-revalidate is on the image beacons used to report into web analytic packages. They use must-revalidate to ensure that each visit causes a request to the analytics’s tracking script to record the visit. I don’t think I’ve ever seen a normal web resource like a website’s logo always forcing conditional requests using a must-revalidate directive.

Know a site we should make a video about? Contact us and you may see a future episode about it. Want see how fast your site is? Try our free assessment.

Something interesting and cool happened with photo sharing service Instagram in the last week. No, I’m not talking about Facebook buying them for $1,000,000,000.00. I’m referring to Instagram co-founder Mike Krieger, and the talk he gave at AirBNB entitled "Scaling Instagram." In his presentation, Mike discusses the technical hurdles faced by going from 0 to 30 million users in 2 years. Very cool indeed.

As all good presentations are, the slides contain minimal text, allowing the speaker to drive the conversation. Unfortunately this reduces the value of the slides when viewed in isolation since the bulk of the presentation’s content was verbal. However there is still a large amount of interesting information and I encourage you to read the slides.

One part that really interested me was when Mike was speaking about early performance and load problems Instagram had. In fact, the very first issue Mike mentions, which happened on their opening weekend, had to do with their favicon.

Instagram’s Favicon Problem

Ahhhh our friend the favicon!

While slide text is sparse, it appears that the favicon for the Instagram website was not present on opening weekend. This caused the underline Django controller to generate a bunch of errors when trying to services incoming favicon requests and that was bad, as we’ll see in a second.

There are a couple of reasons that a favicon can result in a 404. First of all, you can explicitly define the location of a favicon using a <link> tag. It is possible that the <link> pointed at a non-existent favicon, resulting in a 404. Another option is not having a <link> tag at all. By default, all modern desktop browsers will make a request to /favicon.ico if no <link> tag is present which explicitly defines the path to the favicon.

So what’s the problem? 404′s are bad for a few reasons. First of all, the user’s browser has made a request and waited for a response only to get nothing. Usually the body of 404 response is larger than the resource being requested. This is certainly true for favicons which are only a few kilobytes in size. Next, web servers and application frameworks can be configured to respond to 404′s in different performance-impactful ways. At a most basic level, they might log information specific to the 404 to help a developer resolve the problem. Some applications, like WordPress, try to be helpful when to request a page they does not exist. These application will try and find the page you might have meant, and redirects you to it. Suddenly a simple typo in a URL is causing extra disk I/O, database queries, and computation, all of which adversely affect response times and increase server load.

In fact, it caused Instagram such a problem that Mike states:

"Lesson #1: Don’t forget your favicon."

But optimizing your website’s favicon goes beyond just making sure you don’t forget it. Below are 5 recommendations for optimizing your favicon, plus two bonus tips, to optimize your website’s favicon.

Tip 1: Make sure it exists.

Since we know that browsers are always going to try and request a favicon, Rule #1 for favicons is to make sure it exists. Whether you include a <link> or not, the browser is making a request for it. Even if you are able to remove any disk I/O or database queries or additional computation by your application when a 404 occurs, the size of your 404 page is going to be larger than the size of your favicon. This is adding insult to injury, since the client has to spend more time downloading more data, all of which is completely useless.

Tip 2: Make sure it is compatible.

Internet Explorer, even modern versions like IE9IE8, only supports favicons using the ICO file format (Update: IE9 supports ICOs and PNGs as favicons, but not GIFs). This sucks, since the ICO format is essentially a uncompressed BMP with a more primitive header.

Instagram doesn’t do this properly. They have a file, /favicon.ico, but it’s actually a 683 byte GIF image as shown in the screen shot below:

Since its a GIF we can see that IE does not render Instagram’s favicon:

While a GIF version of Instagram’s favicon is smaller than the ICO, it does not work properly on any version of Internet Explorer, including IE9. (And if it was a PNG image, it won’t work on anything up to and including IE8). Since over 50% of the browser market is still IE, it’s silly to waste time sending bytes to an IE client which it can’t do anything with.

One possible solution is to always use a <link> tag to point to a favicon. When requesting an HTML page, the application could look at the User-Agent string. Based on the browser type it could write out a <link> tag pointing at the GIF version of the favicon for non-IE browsers and a <link> to ICO version of the favicon for IE browsers. It is critical that this sniffing occurs for HTML pages and not when serving the actual favicon. As we have seen, using factors other than URL to determine the content to serve has an enormous negative impact on caching.

Tip 3: Make sure it is cacheable.

Speaking of caching, like all your images you should also cache your favicon. Unless you are using <link> tags to reference the favicon, you cannot version your URLs to allow for far-future caching. This is yet another reason why you should be using <link> tags for favicons instead of simply placing favicon.ico in the web root. In this case, consider caching for a few days to a few months, based on how often you radically alter your website’s design.

Tip 4: Make sure it is the right size.

Favicons are almost always shown at 16 pixels by 16 pixels in the browser’s address bar or browser tab. It does not make sense to use larger dimensions for a favicon.

Using the proper dimensions is not enough. The ICO file format allows for multiple copies of an image with different dimensions to exist in a single ICO file. This is because the ICO file was originally designed to store icon images for Windows and needed to support different resolutions and sizes. These larger dimensions are unused by modern web browsers and merely increase the size of the favicon.

Detecting improperly sized favicons is pretty easy. A 16×16 pixel, 256 color ICO file with no extraneous sizes should be around 1400 bytes. If your ICO is larger than that, you are probably doing something wrong.

You can use the file command on Mac/Linux/Unix/Cygwin systems to identify ICO files containing multiple images as shown in screen shot below.

When opening multiple image ICO file in an editor like Photoshop or GIMP, the different sizes appear as separate layers as shown in the screen shot below. This allows you to remove the extraneous images and create an ICO file with a single 16×16 pixel image.

Tip 5: Make sure you use compression.

If you converted Instagram’s 683 byte GIF into a properly formatted ICO file, it would be 1406 bytes. That sounds bad, but this is not an apples-to-apples comparison. As I mentioned in my Lose The Wait: HTTP Compression post, ICO is not a natively compressed image format. This mean you can and should serve your favicon using HTTP compression. Using GZip, the 1406 byte ICO file compressed to 826 bytes. This is only about 20% larger than Instagram’s GIF-based favicon and it works across all browsers.

Bonus Tip 1: Try using a 4-bit favicon.

ICO images can have a range of color depths, from 8-bit up to 16-bit or 24-bit color. Typically favicons are 8-bit color, allowing for up to 256 distinct colors. Considering that a favicon is only 16×16 pixels, each pixel could literally be a different color!

As I wrote about in Choosing PNG8 Candidate Images and presented last year at Velocity 2011, images with smaller dimensions can have fewer colors without being noticeable to users. Reducing a 16×16 pixel, 8-bit, 256 color icon to a 16×16 pixel, 4-bit, 16 colors can produce a much smaller file with little visual distortion. Obviously the results are subjective and this works better with some favicons than others.

Below is Google’s favicon:

On the left is an 8-bit, 256 color version and on the right is a 4-bit, 16 color version. There is virtually no visible difference. The 4-bit version is resulting ICO is only 318 bytes. Applying HTTP compression takes this down to 256 bytes, a savings of 71%.

Let’s try something more complex, like Instagram’s favicon.

On the left is a 8-bit, 256 color version. On the right is a 4-bit, 16 color version. You can see some slight visual distortion, but the resulting 4-bit ICO is only 318 bytes. Applying HTTP compression takes this down to 262 bytes. That means a 4-bit HTTP compressed ICO-based Instagram favicon is 2.5 times smaller than their current GIF-based favicon! As an added benefit it works across all browsers!

As you can see, for simple favicons, converting to a 4-bit icon can achieve a large amount of savings with little visual sacrifice. This is why major sites like Yahoo’s and Wikipedia all use 4-bit, 16 color favicons.

Creating a 4-bit favicon is easy, but it various from tool to tool. Look for a way to reduce the image’s color depth. As shown in the screen shot below, GIMP provides an option for 4-bit ICO when saving the image.

Bonus Tip 2: Try inlining the favicon.

One of the downsides of favicons is that you cannot prevent their use. That was the original problem that Mike from Instagram was talking about. You cannot avoid using a favicon by not including a a <link> tag reference; all browsers will send a request for /favicon.ico if a <link> reference is not present. So the browser is going to request a favicon whether you want it to or not!

Or does it?

You can embed the favicon into the HTML document using a <link> tag and a data URI. Since it’s a data URI, you have to Base64 encode the ICO file, but then it gets gzipped since it is inside of HTML document (you using HTTP compression correctly right?). The 4-bit, 16 color, gzipped Instagram favicon is only 262 bytes. If you Base64 the Instagram favicon and gzip it, it is only 285 bytes in size. And you get the advantage of not having to make an HTTP request!

Sounds great right?

Well, as mention over on the WebPageTest Forums, there can be a few problems. First of all, only IE8 and above supports data URIs, so you are going to need to do some server-side sniffing to only embedded it for compatible browsers. Also, many browsers try to be intelligent and only load the favicon after all the other page resources have been downloaded. Embedding the favicon could potentially delay page rendering. Another downside is that you need to embed the favicon into every page, adding 285 bytes each time.

Whether this makes sense is going to vary from site to site and the browsers they use. If the majority of your visitors are new users with an empty cache, and only visit a few pages, then embedding a favicon may make your site faster. Experiment and see!

Conclusions

When the co-founder of a company that went from 0 to 30 million users in less than 2 years and which was just acquired by Facebook for $1 billion dollars mentions favicons as his very first lesson in performance scaling, you know they are important! Just remember this list of optimizations for your website’s favicon:

1. Make sure it exists.
2. Make sure it is compatible.
3. Make sure it is cacheable.
4. Make sure it is the right size.
5. Make sure to serve it using compression.

Additionally you can try:

6. Using 4-bit color depth icons.
7. Inlining the favicon using data URIs.

Want to see what performance problems your website has? Unoptimzied Favicons and Favicon served without Compression are just 2 of the nearly 400 performance issues Zoompf detects when testing your web applications. You can get a free performance scan of you website now and take a look at our Zoompf WPO product at Zoompf.com today!

Our regular video series How Fast Is…? examines real world websites and details the cause of their performances issues as well as what should be done to solve them. After all, the best way to learn about front-end web performance is to see what other people are doing right and doing wrong. In this edition of How Fast Is…? we analyze online retailer Orbitz.

Why Orbitz.com?

I wanted to look at Orbitz for a few reasons. The first is they are entirely an online business. All of their revenue comes from their website. This means they should be keenly aware of the impact of web performance on their bottom line. Orbitz is also large enough to have resources, in the form of both people and money, to spend on optimizing its website. Finally, Orbitz’s website integrates with flight information and reservation systems, hotel adn car rental agencies, and other 3rd party systems. I wanted to see how this might impact Orbitz’s web performance.

Optimization Across the Entire Site

Usually in Zoompf’s How Fast Is…? videos, I only examine a few pages of a website. This time, I crawled and analyzed nearly 1000 pages and compared their performance to the performance of the Orbitz.com home page. In this video I discuss how the rest of the Orbitz site suffers from many performance problems which do not exist on the main page. This leads me to conclude that Orbitz’s frontend performance process is immature. They either do not have the tools or the processes in place to test large portions of their website for frontend performance best practices. When they do recognize a performance problem and optimize it, it is applied haphazardly and not uniformly across their entire website. Most likely this means that their process to optimize content is not formalized or automated, and is applied ad hoc when their is time available, or as issues are noticed.

Know a site we should make a video about? Contact us and you may see a future episode about it. Want see how fast your site is? Try our free assessment.