Stoyan has a good blog post today as part of his Performance Advent series about required reading for web developers. He covered some great books. All three of the three books that have been published are currently sitting on my bookshelf and you should buy them immediately if you don’t already own them. I thought I’d share a few more books that I have read that contain more web performance tips and tricks that I have not seen in the books he recommended. Some of helpful some are not. Having written a book myself on Ajax Security I know exactly how difficult it is to create a meaningful and lasting book of substance. All of these authors deserve respect, even if the book no longer is beneficial today. For each of these 4 books I have included my overview and opinion of the book, the key performance tips and ideas it contains, and my recommendation.

Web Caching. By Duane Wessels (O’Reilly, 2001)

Duane Wessels is the perfect choice to write what is the definitive guide to web caching as he is the creator of the Squid Caching Proxy. While this book is targeted more at IT operational folks (specifically people who install, configure, monitor, and maintain web proxies) it provides excellent background into how caching proxies work and are deployed and what they will and will not cache. It also has, without a doubt, the best explanation about Cache-Control directives I have ever read. It explains what the directives mean, how they interact with each other, and how caching proxies and the browser cache act on those directives. Think you know what “no-cache” does? You are wrong.

Key Performance Information

This book has tidbits here and there that will help front-end performance such as: Using Cache-Control correctly. Adding support for stale resources. What will proxies not cache even if it’s allowed (URL’s with query strings, CGI-bin directories, etc). When is caching used but pointless (varying on cookies, host, etc). How can you improve your hit/miss ratio.

Verdict

BUY! Good background, worth the cost of the book for the exhaustive explanation of caching directives alone. A dozen or so front-end performance tidbits scattered throughout. Find a cheap used copy.

JavaScript: The Good Parts. By Douglas Crockford (O’Reilly 2008)

Written by JSON creator Douglas Crockford, JavaScript: The Good Parts provides a detailed analysis of JavaScript as a programming language and explorers what features of the language aid and what features hinder the creation of beautiful code and why. While targeted at JavaScript developers Chapter 10 and Appendixes A, B, and C provide a wealth of performance advice.

Key Performance Information

Half a dozen JavaScript performance tips mixed in throughout such as: Controlling scope chains of variables, dynamic compilation of code at runtime, avoiding type coercion, loop construction, regular expression performance.

Verdict

BUY! Will open your eyes about the elegance of JavaScript. If you like computer science and algorithms you will love this book. If you are only interested in the performance tips you’ll be disappointed if you pay full price for such a small book. Buy it used in that case.

Web Performance Tuning. By Patrick Killelea (O’Reilly 2002)

Originally written in 1998, this 2nd edition was released in 2002 with seemingly minimal updating. I really wanted to like this book. It is well written with tons of data tables, charts, and graphs. Unfortunately nearly the entirety of the book serves better as a reference manual and contains little and poorly actionable performance advice for web developers. For example, the chapter on “Security” is about SSL. (I will punch the next person in the face who equates web security with SSL and firewalls). This chapter contains some nice graphs of the performance of an obsolete Netscape web server. After all of that the “advice” is to “consider buying an SSL accelerator card.” What about performance of different algorithms? Or how to optimize SSL negotiation? Or the importance of keeping SSL connections open? Nothing (though I’ll be writing a blog post about optimizing SSL performance soon). The book also contains very outdated filler chapters such as choosing a modem, choosing a client and server OS, choosing client and server hardware, and an overview of non-HTTP network protocols.

That is not to say this is a bad book. There are some very enjoyable parts. I found the information about the chain of syscalls Apache makes to process an HTTP request and serve the response to be utterly fascinating. Chapter 19 is only chapter truly applicable to front-end performance. You should know everything in the chapter already but it is interesting largely because the advice it contains predates the current front-end performance movement by a good 7 years.

Key Performance Information

All but a very few bits of performance advice is obsolete and focuses entirely on the back-end. The main nuggets were things like: Use short filenames to save space. Minimize the use of symbolic links on the server. Turn off reverse DNS lookup for log files. Turn off mod_status. Set height and width HTML attributes to avoid repainting/re-rendering.

Verdict

Do Not Buy. This is no longer a useful book about web performance and based on the number of filler chapters I doubt its value when it was published. It is an enjoyable book if you are interested in learing more about how back-end web hardware functions. If so I suggest you find a used copy as the information this book contains is so out of date it’s not worth anywhere near its cover price. I purchased it for $2.77 from Amazon and was happy.

Building Scalable Web Sites. By Cal Henderson (O’Reilly 2006)

Cal is the lead developer of Flickr so he knows a thing or three about building complex web applications that have to performance for millions of users. Don’t pigeon-hole this book as a back-end hardware book. It is a holistic book that covers a lot of ground in just 320 pages. This book is a guide to the development processes and practices, as well as architectural and back-end design of web sites that can be maintained and scaled to immense levels of traffic. Yes there is information about load balancers and database clustering. But there is also information about coding practices: Using source code, branching, supporting international characters, abstracting away translations, abstracting/modulizing your code for easy updating, fail over, A/B testing of new features, and failover. Think of it as a modern version of Web Performance Tuning with current and proper information and no filler.

Key Performance Information

No specific advice per say. Instead this book is about how the design and building of web applications that are easy to maintain, expand and extended, and quickly replace based on the growth of your user base. It will change the way to build web applications.

Verdict

BUY! An excellent survey of the processes needed to build and grow truly scalable applications. Its information on building asynchronous remote systems is worth the price alone. I am using this as my bible as I design the web front-end to Zoompf’s scanning engine. I highly recommend this book to both web developers and IT operations.

Conclusions

There are some obvious must have web performance books available today. However there are additional books that provide insight into the tricks, tips, and processes needed to build high performance web applications that are not published elsewhere. Hopefully this post should help you build out your library of web performance books.

Did I miss one? Please comment below and tell me what other books you recommend that can contain good advice to improve website performance.

Not defining a character set or where you do define it can cause poor performance for your website’s visitors. In this post we will discuss character sets and how best to define them to avoid web performance problems.

At their core, HTML documents are just a series of bytes. The character set (or charset) for an HTML document tells your web browser how it should process those bytes to construct characters. The browser then interprets those characters to render the web page. The 2 most common ways to tell the web browser what charset to use for an HTML page are by specifying it in the HTTP Content-Type header or by using a <META> tag to emulate an HTTP Content-Type header. When the web content author is the same person as the web server administrator it is possible to directly configure the web server to use the appropriate charset for the appropriate URLs. In this world of virtual hosts, Content Management Systems, and blogs this is rarely the case anymore. As such more and more web developers are using <META> tags to define the charset for HTML documents.

This leads to a Chicken-and-the-Egg problem. The HTML document contains text which tells the browser how to read the document. Hmmm. So how does the browser read the document without a charset? While it varies with browser and version, most assume a Latin alphabet charset like US-ASCII, Latin-1, or ISO-8859-1. The browser then reads the HTML document using this charset scanning for charset information. At this point one of three things happens:

1. There is no <META> tag with charset information.
2. There is <META> tag with a charset and it’s what the browser guessed.
3. There is a <META> tag with a charset, but it’s a different charset than the browser guessed.

If there is no charset information the browser is in an odd position. At this point most browsers attempt some type of charset detection. With several years of web security experience believe me when I tell you that in theory this is an awesome idea but in practice this is a horrible idea. Web browsers or servers trying to “fix” broken data is the root a number of nasty web security vulnerabilities (such as UTF-7 XSS attacks and various other injection evasions). Regardless, no charset information of any kind forces the browser to do more processing which can produce a very small performance hit at best and a hacked website at worse.

If there is a <META> tag whose charset is the same as what the browser guessed there is no issue. Nothing else needs to occur.

If there is a <META> tag and it specifies a charset different than the assumed charset the browser has a problem. It has already interpreted some amount of the HTML document but it was the wrong charset. That information is all bad. The document needs to be reprocessed using the correct charset. So right now at best you are talking about a small performance penalty as the browser has to reparse the beginning of the HTML document.

But it can get worse! This is because browsers don’t scan the entire HTML document looking for a charset. They want to start rendering content! If they don’t see a charset defined “near the top” of the HTML document they start rendering content and executing JavaScript using the assumed charset. (“Near the top” varies from browser to browser which we will discuss in a minute). But once the browser gets going interpreting and executing content and then finds a <META> tag with charset information it’s in a real bind. Because now it has already been executing code, and requesting other resources, and render content using the wrong charset! Those URLs could be wrong, that JavaScript could have syntax errors, or the CSS rules could be misspelled all because the browser read them using the wrong charset information.

“Near the top” for Firefox 3.5 means within the first 2048 bytes. If Firefox does not detect charset information in the first 2048 of an HTML document (and no charset was defined in the HTTP headers) it starts rendering the page and executing script using an assumed charset (I did not investigate other browsers). Consider this example web page adapted from a Simon Pieters test case. It contains some JavaScript, whitespace, and starting just after 2048 bytes, a <META> tag defining the charset. In Firefox the JavaScript and pop an alert box showing a Euro sign. After 2048 bytes there is a <META> tag changing the charset from the assumed Latin-1. Firefox has to reprocess and re-render the page which will execute the JavaScript again with a Cyrillic character appearing in the alert box this time.

It is also interesting what the browser does if it has already made a request. If Firefox has already requested a URL and then detects a new charset the URL must be re-requested. Consider this example page. Here JavaScript make a request to a nonexistent image from www.google.com (we include the alert box to create a delay in thus simple test case to ensure Firefox has already started fetching the resource). The URL contains a character changes based on the charset so it must be re-requested. Using an HTTP proxy we see the browser made 2 requests to 2 different URLs (with URL encoding to encode the characters being sent)

Note: it appears that Firefox does not try to re-request a URL if the change in the charset did not affected the change the meaning of the URL. If you modify the 2nd example to request “abc.gif” it does not appear that Firefox fetches this twice. More testing is needed here.

Want to see what performance problems you have? An appropriately placed <META> charset tag is just one of the 200+ performance issues Zoompf detects while assessing your web applications. You can sign up for a free mini web performance assessment at Zoompf.com today!

This post is a follow up to the previous post “Rezipping Web Resources for Fun and Profit.” In that article, we showed that many common web files, such as MS Office documents, Silverlight applications, Java Applets, and more are really just Zip files with a special structure of files inside. By rezipping a file (unzipping the contents and rezipping those contents using a higher compression level) web developers can reduce size of those files by 5-30%!

An obvious, but less useful expansion of rezipping is to extend it to other compression types, namely GZip compressed files or BZip2 compressed files. We can use 7-zip’s command line version 7za to accomplish this. It looks something like this:

This approach can be extended to BZip2 using “-tbzip2″ switch. I collected a few samples of GZip archives and using rezipping was able to reduce their size by an average of 5.03% as shown in the table below.

Using rezipping on GZip or BZip2 archives is unfortunately less useful and beneficial than on Zip files. This is because so many files that served or downloaded on the web use Zip files as a wrapper. Finding ways to optimize Zip files lets you optimize a dozen other file types on the web. These files are either directly loaded and executed by the browser (like Silverlight or Applets) or are very common downloadable content like documents or presentations. However I know of no web content that uses a GZip file or BZip2 file as a wrapper file. While downloadable programs, source code, or other archives might use GZip or BZip2 you will not find any widely deployed document or content format that uses these as the wrapper file. This limits the usefulness of rezipping GZip or BZip2 archives.

As mention in the last post, one positive note is that while no widely deployed web files use GZip as a wrapper, many files contain raw GZip or DEFLATE streams. Flash files use GZip to compress the contents of the SWF tags. PDF’s uses DEFLATE to compress text streams. This means with a little parsing and some glue code proven tools like 7-zip should be able to be used to reduce the size of other files that are very common on the web today!

One large area of web performance optimization is reducing the size of your content. Most people know about obvious techniques like HTTP compression, minifying, or removing extra data from images. However there is one size-reduction technique that does not seem to be common knowledge for most web performance junkies: Rezipping.

Let us start with a little background. Zip archives consist of multiple compressed files that are package together into a single file. Zip archives are compressed using the DEFLATE compression algorithm. Deflate supports different compression levels from 1-9. These compression levels provides a trade-off between CPU and memory resources used to create the Zip file and the size of the resulting Zip file. Using a higher compression level consumes more resources but you end up with a smaller file. Most Zip programs tend to create Zip archives using a compression level of 5 or 7. While this can be a good trade off as the file is created quickly and is reasonable compressed it will not produce the smallest file possible.

Now all that is well and good. But why should frontend web developers care about Zip file optimization? Simple: Many of the most common files on the Internet are actually Zip files. By creating methods to make smaller Zip files we are actually optimizing multiple different types of web files. Optimizing these files will reduce bandwidth consumption and server load while improving page load times.

These “Files that don’t end in .zip but really are Zip Files” use the Zip file format as kind of a wrapper to collect all the bits and pieces that really make up the file and store them in a single compressed unit. For example, Silverlight applications have a XAP file extension. However Silverlight applications are just a Zip file containing compiled byte code, resources like images and sounds, and other configuration. Java Applets contained in JAR files are Zip files. All of the Microsoft Office’s OOXML documents (DOCX, XLSX, PPTX, etc) are Zip files. All of OpenOffice.org’s ODF documents (ODT, ODP, ODS, etc) are Zip flies. You can rename any of these types of files to “.zip” and open them with any Zip program.

Since all of these common web files are simply Zip files we can optimize them to improve web performance and operational costs. This is where Rezipping comes in. Rezipping is process of recompressing a Zip file to create a smaller file. The process is simple: you take any Zip file, unzip the contents, and then rezip the content at a higher compression level. To accomplish this, I am using the command line version of 7zip. 7zip’s implementation of the DEFLATE compressor is generally considered to compress files better than other Zip programs by 5% to 10%. The process looks like this:

Silverlight

On average rezipping reduces a Silverlight application by 12.32%. This is quite good given that XAP files can contain many binary files like images or sounds that will not be recompressed. Some files created from Visual Studio saw an improvement or more than 25%! Also notice that “ImageGallery_v25″ is the Silverlight application used by Bing to change Bing’s background image. This heavily served file could be slimmed by nearly 4% simply be rezipping the XAP file!

Microsoft Excel Documents

On average rezipping Excel files saves about 25%. This makes sense as most Excel spreadsheets contain predominately text and not uncompressable binary data.

Microsoft PowerPoint Documents

On average rezipping PowerPoint files saves about 9%. This can vary widely depending on the number of images that are contained inside the PPTX file as images are not recompressed (more on that in another article).

Microsoft Word Documents

On average rezipping Word documents saves about 12%.

Conclusions

Always Use Rezipping! Stop sending bytes down the pipe you don’t have to! The savings you receive from ReZipping is driven by the contents of the Zip file. Files with a large number of binary objects that will not be compressed (like images) will have a lower improvement. Also note that higher compression levels increase the time and memory to compress data. but they do not increase the time it takes to decompress data. This is because all the work is in finding out what can be reduced during compression, not in recreating the original data during decompression. There is no reason not to use rezipping.

By rezipping your files you can reduce the size of your content. This reduces bandwidth consumption and server load while improving page load times! There is more work to be done. There are a number of web flies that contain raw Deflate streams like Flash files, WOFF font files, SVGZ, and more. All of these could be redeflated using a compression level of 9 and make smaller, faster files. Stay tuned as we investigate this more.

(This is the second article in a series of articles about performance questions you should ask when choosing a hosting provider. The first article in the series is here)

Last time we covered the most important question you should ask a hosting provider: What control do I have over the web server. This time we will be showcasing another important question to ask a hosting provider:

“What Access Do You Provide to Web Server Logs?”

The main reason you want access to log files from the web server is to learn how visitors are accessing your content. This will reveal a wealth of knowledge about the raw traffic patterns of your web application and expose various performance issues and limitations. Often these performance issues will not be detected by page-based performance tools like Yahoo’s YSlow or Google’s Page Speed.

Web server logs come in many different formats. Usually they are large text files where every request is logged on its own line. Several pieces of data about each request are logged in different fields on the line separated by commas. Typically information that is logged for each request is:

• URL requested.
• IP address of the visitor.
• Date and time request was received.
• The program or browser used to request the URL. This is called the User-Agent.
• The Referring webpage (if any).
• HTTP version used to request the page.
• Status code of the response.
• Size of the body of the response.

Log files are a very granular view of your web traffic. Sometimes it can be difficult to see the forest through the trees. For example, what pages did user XYZ visit, in what order, and how long did the user stay on each page? It is usually very difficult to get this information from logs alone because web server logs only track users by a specific IP address. To provide a larger view and answer questions like those listed above web developers use web analytics packages like Omniture, Hitbox, or Google Analytics. Web analytics packages uses cookies and JavaScript to gather detailed information about your visitors, the capabilities of their browsers, and their actions through your web application. Web analytics packages are simple to add to a website. Typically all that is involved is inserting a block of JavaScript at the end of each HTML page. This is very easy to do on templated or dynamically generated websites. So if web analytics provides you with “bigger picture” and richer data than web server logs that begs the question:

Are Web Analytics Reports Good Enough?

Actually, no. Web analytics reports are not good enough. Web analytics data abstracts away the raw traffic of your web application and which can hide several important problems. Web analytics packages only track visitor requests and activity for HTML files that are served with a 200 status code. Out of the box, here are things that most web analytics packages do not track:

• All requests to non-HTML resources.
  • Images
  • JavaScript
  • Style sheets
  • Feeds (RSS, Atom)
  • RIA files (HTC, Flash, Silverlight, Java, etc)
  • Access files (robots.txt, sitemaps.xml, crossdomain.xml, etc)
  • Documents (PDF, Office docs, Zip files)
  • Other resources (Fonts, Cursors)
• Most error pages (404, 5xx, etc).
• Conditional requests that return “304 Not Modified.”
• Requests from non browser User-Agents (spiders, mash-ups, etc).
• Users who have JavaScript disabled for accessibility or (more commonly) security reasons.

This valuable information is completely missed if you are only using web analytics data to understand your traffic. Consider the valuable questions you can answer with web server logs:

• Where are your redirects? Which can be removed to decrease page load time?
• What web resources are using the most bandwidth? This is calculated by simply adding up all the body sizes that are returned for a resource and sorting. Can you reduce the size of these files somehow using compression, minification, or by removing meta data?
• What are the most requested resources on your website? Can you use caching or other methods to minimize the number of times the resource is requested? If you cannot cache those files because they are dynamically generated can you add programming logic to use a Last-Modified header to reduce bandwidth? Can you remove resources like external JavaScript or CSS files that are referenced by not actually used by that web page?
• How often does your static content actually change? This is calculated by counting the 304s for a resource. Perhaps you can use a longer Expires time on your content.
• How often are search engine crawlers visiting your site? Are there crawlers that are missing? You should submit your website to their indexes.
• Are crawlers finding your high value content? Perhaps you should be using or modify your sitemap.
• Do crawlers request a large amount of low value content? Do crawlers “get stuck” on part of your website? Perhaps you need to fix your robots.txt.
• Which web resources are you still getting a lot of requests for that no longer exist? You should use a redirect that points to the correct content.

Real Life Examples

At Zoompf we have detected and solved numerous performance issues just by examining a client’s web server log files. Here are some of our more interesting stories:

• A social networking client used their robots.txt file to prevent crawlers from indexing content from new, untrusted members that could be content spammers. As such their robots.txt was 250 kilobytes! Because the file was updated so often all of the search crawlers would request it multiple times a day. These factors resulted in the client using 5 gigs of bandwidth a month just to serve its robots.txt file! Turning on HTTP compression for text files reduced this by 70%. The client is currently implementing the use of <META> tags and rel=”nofollow” attributes to limit search engine indexing for the web pages of untrusted users. This will result in even higher performance savings.
• A software client found that by far their most popular pages were for their product documentation. These pages were constructed using a PHP templating system. However these files never changed once that version of the product shipped. The client moved to pre-rendering the web pages to static HTML files and using a far future Expires header on the HTML files. This drastically improved performance and reduced bandwidth consumption and server load.
• An ecommerce client discovered crawlers were requested all the available colors for each item they sold. For example, the crawlers would visit /item1/, /item1/color/red/, /item1/color/blue/, and so on. By creating a robots.txt rule to prevent crawlers from requesting every color for every item the client reduced their bandwidth by nearly 80% while still having their important content indexed.
• A client discovered that their most important content was not getting indexed. A developer had copied a code snippet from the Internet into the top of their template to solve a CSS problem they were having. Unfortunately this code snippet also included a <META> tag to telling search engines not to index the web page.
• A client learned that its logo had not changed for over 4 years and it was also an uncompressed BMP file even though the logo had a JPEG extension. They change the logo to be a proper image format for web and increased the logo’s Expires time.
• A client discovered that no one had every requested their iPhone application image. They removed the <LINK> tag and reduced the size of all of their HTML pages.
• A client discovered their favicon.ico file was consuming huge amounts of bandwidth. This is because it contained multiple versions of the same icon at different dimensions. Removing all but the 16 by 16 pixel version from the ICO file reduced file size by 97%.

Typically Log Access

If you have your own server access to the web server logs is usually unrestricted. However in most shared hosting environments you will not have direct access. Typical web server log options you have are:

• The raw Apache, IIS, or NCSA log files in a directory outside of your web root that you can access using FTP or sFTP. This is the ideal case.
• The raw Apache, IIS, or NCSA log files placed directly in your web root. While this provides you with the raw logs anyone on the internet can also access your log files. This is a security risk as log files can often contain sensitive data like credentials or “hidden” areas of your web application. Talk with your hosting provider about moving the location of the web logs.
• An option through a web-based website administration system like cPANEL that lets you download the raw log file.
• An option or interface in the web admin system that lets you view or download a specially formatted version of the logs.

If you cannot access the raw log files don’t panic. As long as the log file contains the follow information you will have all the data you need:

• URL requested
• Date and time request was received
• The program or browser used to request the URL. This is called the User-Agent.
• Status code of the response
• Size of the body of the response

Another question to ask hosting providers is not only “what information is in the log file” but also “how much time does the log file cover?” You can imagine in large sharing hosting environments how log files can quickly go to hundreds of megabytes for potentially thousands of customers. Hosting providers often limit the log file in different ways including:

• Record only a week of traffic and replace the log with a new empty file every week.
• Limit the total size of the log file. Each new entry removes an entry from the start of the log
• Provide a night copy of the log file for all the traffic of the site received that day. These copies are usually removed after a certain about of time.

If you do have a time window make sure grab a copy of the log file. Some interfaces like cPANEL offer a scheduling services that can email you the log file or place them in a special location that you can then download. You can schedule an FTP download or use wget or curl to download these log files.

Processing Log Files

Depending on how much log data you have, you might want to concatenate your log files together until you have a big enough sample. At Zoompf we suggest collecting a sample between 500,000-1,000,000 requests, or a week’s worth of web traffic, depending on which is larger. Programs like awstats are very helpful for processing and provide reports with your most popular and least popular files, largest files in terms of bandwidth, and other data already broken out. Directly processing the logs yourself always you to discovered more detailed data and not as hard as you would thing. Some basic regular expressions can make it very easy to gather metrics like “show all of the 304s, 404s, 500s, etc.”

Remember, examining your web logs is a key technique to discovering and solving performance problems with your web applications. Those pretty graphs from Google Analytics or other web analytics data is simply not good enough to detect performance issues and bottlenecks. You need access to the information about all the requests the web server is processing. Make sure you ask your hosting provider how you can access the raw web server log files. Find out how much web traffic data the logs contain and how you can easily collect this data so you can analysis. If your hosting provider does not provide this you should consider that a deal breaker and find another provider.

Hosting a web application can be annoying and time consuming. There is the cost of the hardware. There is the time configuring, administering, and patching the operating system, web server, and other software. There is the security risk of exposing a machine onto the Internet. So it’s no surprise that many people and companies use a 3rd party hosting provider to host their web application and manage the infrastructure. Choosing a hosting provider should not be made lightly. You no longer have full control over the machine running your web application. For those interested in creating high performance web applications you must ensure that you don’t give up control over the features that you need to make your web application run as fast as possible.

This is the first in a series articles of performance questions you should ask a hosting provider. While hosting providers do offer dedicated hosting (where your application runs on a single machine all by itself) the vast majority of people choose shared hosting environments. While we will be references hosting services that use the Apache web server all of the advice in this series is applicable to Windows hosting as well.

Without a doubt the first and most important question you should ask a hosting provider is:

“What Control Do I Have Over Web Server Configuration?”

This questions is critical. Many of the easiest and most impactful performance improvements you can make to your web application, such as HTTP compression and caching, are configured at the web server level. You should start off by asking what modules are installed already. The Apache modules most relevant to performance are:

• mod_deflate (for Apache 2) or mod_gzip (for Apache 1) – This module enables HTTP compression.
• mod_expires – This module enables HTTP caching.
• mod_rewrite – This module enables on-the-fly URL rewriting which is very helpful when maintaining and updating resources while using far future caching.

All of these modules are installed with the typical default installation of Apache. While this depends on the platform and the distribution they are almost always present by default. Sometimes web hosting companies will compile their own version of Apache from source to maximize performance for their particular server machines. Often they will remove modules to save space and time. If you find a hosting provider like this explain to them that you would like these modules installed. Tell them this is a reasonable request as these modules are part of the default installation of Apache. You should be able to convince them to turn these modules on for you. If not, this is a deal breaker and you should not use that hosting provider. The vast majority of hosting providers offer these modules even at the lowest pricing tiers.

Even if the hosting provider offers these modules, you should ask them for a list of all available modules as well as their policy is for enabling new modules. While mod_deflate, mod_expires, and mod_rewrite and the most helpful modules from a performance point of view there might be other modules, such as mod_cband or mod_bw, that you might want to use for performance reasons.

Once you know what you can configure on the web server your next question should be “how do I configure it?” In most shared hosting environments you will not have access to the main Apache configuration file httpd.conf but usually can control the web server through the use of .htaccess files. This is the best solution since it allows you to directly configure the web server. You simply edit the .htaccess file in the root directory for your web application and upload it to the hosting provider.

Some hosting providers supply you with a web interface to control web server configuration typically through a web administration system like CPanel. If this is the case ask to see examples of the interface. It could be simply a web form that allows to you edit a raw .htaccess file. It could be a more structured web interface with check boxes to turn on modules or forms to add new rules. Be very wary of any type of web-based server configuration. The interface will limit what you are able to configure. If a web interface is available ask if you can still manually upload your own .htaccess file to control the web server. If you cannot do this your ability to configure the web server will be severely limited. If the web interface does not provide the functionality you need you should not use that hosting provider. In general you should not use hosting providers that only offer web-based server configuration.

Bad Idea: Hacking Around Limits

Some developers like to point out that you can use server side application logic to compress content or implement caching for static resources like images or JavaScript files or CSS files. This means you don’t have to have access to the web server to configure things like HTTP compression or caching. Unfortunately this actually hurts performance more than it helps! With this method, PHP (or some other application logic layer) is invoked for all requests. Remember that the vast majority of requests are for static content and do not hit the application layer. The overhead of invoking PHP dozens if not hundreds of times for a page load removes any performance benefit of compressing or caching. We will explore this method more in a future post. For now you should completely avoid it. Never use application code to hack around the blatant shortcomings of a hosting provider.

Remember when choosing a hosting provider the single most important performance question you can ask is “how do I configure the web server?” In our next post we will explore more performance questions you should ask when choosing your hosting provider.

Since launching Zoompf last Friday I’ve performed dozens of free web performance scans. A few users reported to me what they thought was a bug. Zoompf was reporting that certain JavaScript files could be further minified. The issue was that these websites used already minified versions of well known JavaScript frameworks like jQuery . These frameworks had a JavaScript comment at the top of the file that included the copyright information and licensing information. Zoompf uses JSMin under the covers to determine which JavaScript files or inline JavaScript blocks can be minified. JSMin would see this comment and remove it. While this is a certainly a performance improvement it is also a copyright infringement violation! That’s not good. We need to solve this.

This is actually a broader problem then just Zoompf. It exists with all of the dozens of ports of JSMin out there as well as many other minifiers. It turns out YUI compressor does not have this issue because YUI supports so-called “important comments.” Important comments are special code comments that will not be removed by the minifier. Ajax libraries like jQuery and SWFObject use them around their copyright comments so important comment aware minifiers will leave them intact. They look like this:

/*!
    Somthing very important
*/

Important comments are an excellent idea. They provide a way to minify files while retaining copyright and licensing information. All minifiers should support this! So how can we achieve that? We must solve 2 challenges:

1. Add support for important comments to existing JavaScript minifers
2. Get JavaScript frameworks to start using important comments appropriately

Since JSMin is the canonical example, I went ahead and added support for important comments to the JavaScript port of JSMin written by Franck Marcia. You can download the updated version of JSMin with Important Comments support or download the JSMin Important Comments patch. (For tips on apply the patch read this article).

The change was made to the next() function, which is supposed to return the next character to be inserted into the minified output. This is the code that detects // and /* */ based comments and silently consumes them. I simply modified the code to check if the comment is important and if so return the entire comment to be written to the minified output stream instead of a single character. The main part of the patch is shown here:

case '*':
//this is a comment. What kind?
get();
if(peek() == '!') {
	//important comment
	var d = '/*!';
	for (;;) {
		c = get();
		switch (c) {
			case '*':
				if (peek() == '/') {
					get();
					return d+'*/';
				}
				break;
			case EOF:
				throw 'Error: Unterminated comment.';
			default:
				d+=c;
		}
	}
} else {
	//unimportant comment	gobbling here ...
}

This solution is easily added to any port of JSMin written in a weakly typed language. This is because the next() function is supposed to return only a single character. This patch makes it return a string containing the important comment block. In JSMin ports to JavaScript or PHP this is not a problem. In JSMin ports written in strongly typed languages like C, C#, or Java, the next() function returns an integer.

Now JSMin is like a Swiss-made watch. It is as elegant as it is compact. It is a good example of how state machines can be small and powerful. Unfortunately, like a Swiss watch, you cannot easily add a new feature without a significant amount of work. I toyed with a few very small patches. I tried modifying the next() function to simply not longer detect or silently consume comments that were important. This caused important comments to be are processed as if they were JavaScript source code. Not bad, but it minifies the comment’s content and could puke on things in the comment like unmatched quotes or by thinking a slash is an unclosed Regex literal. I’m going to continue to try and patch JSMin in a way that works with both weakly typed and strongly typed languages that retains the elegance of Douglas Crockford’s design. Stay tuned. For now the weakly typed solution is available.

I would say that minifiers that support important comments should not convert a /*! … */ comment to a /* … */ comment. This would allow minifiers to run on text that could have already been minified without removing anything important. The extra character per important comment cost is well worth the ability to chain together multiple tools without having to worry about losing anything.

The second challenge is getting JavaScript frameworks to start using important comments appropriately. Notice I said appropriately. We don’t want to just go adding a ! to the first comment or anything. Some frameworks, such as Mootools, include all sorts of content in there initial comment that should not be included in a minified version with important content. Ideally frameworks should use a important comment block the way SWFObject did. A simple, single, important comment that contains the name of the library, a URL to the framework’s web page, the license it uses, and a link to the license. Yahoo’s User Interface Library is another example of an appropriate comment block, though they are not using important comment syntax for their copyright and licensing declaration.

Know any other JavaScript minifiers that support important comments? Working on a JavaScript Framework and want to add important comments? Drop me a message and share all about it.

Phreaknic was a blast this past weekend! I’ve been attending and speaking there for 8 years now and while Halloween certainly cut into attendance this year there were some awesome presentations. Famulus‘s talk complete with pictures and video of his quest to build a functional Bussard fusion reactor was amazing. Adrian Crenshaw gave a great overview of the various anonymizing networks and darknets out there including some of the work Matt Wood and I did on Veiled. Finally Azureus‘s Tyler Pitchford gave a highly entertaining talk on reverse engineering. His best line? “And now you are on the NOP sled! Wheeeee!”

I spoke last on Friday night and presented on web performance. As usually I ran long but I was getting some excellent questions from the audience. You can download the slides from my presentation, Optimizing Web Performance, on Zoompf.com.

The business case for web performance is obvious. Faster apps increase revenue. Using resources more efficiently reduces operational costs. The push to the cloud and the down economy are just amplifying the importance of web performance for today’s web. So it is not a surprise that the performance testing market is huge.

But there is a gap in the performance testing market when it comes to performance testing of modern web applications. Talk to anyone about web performance and they start talking about the usual suspects:

• Refactoring, optimizing, JITing, caching application code and data
• Database tuning, queries, store procedures, indexes, denormalizing tables
• Reverse proxies, memcached, Varnish, load balancers, SSL accelerators, etc.

But recent research has found generating dynamic content accounts for typically less than 10% of page load times. The vast majority of page load time is spent downloading, parsing, and rendering all the components that make up a modern application. It is on the front end, and not on the back end, where optimizations can be made that drop seconds off load times. JavaScript code, CSS, the inner workings of browsers, HTTP voodoo. I am a thought leader in exactly this space.

The majority of widely known web performance optimization practices today focus on the application tier or the database tier. Traditional performance testings tools do no front end optimization testing. And yet the front end has the biggest impact on web application performance in modern applications. Do you see the disconnect yet?

This is why Zoompf exists. There is an enormous opportunity to make real improvements to the web as we know it. Zoompf will be leading that charge.