Rich internet applications (RIA), using Ajax, Flash, or other methods, have always been a problem for search engines.  That’s because search bots don’t have the ability to access content interactively the way humans do. In 2009, Google introduced a proposal for indexing Ajax.  We’ve got clients using lots of Ajax, so naturally, we’ve been watching closely to see what sites are adopting Google’s specification, and how it’s working out for them.

If you need the background on Google’s Ajax crawling specification, here it is in a few bullets:

• In October of 2009, Google proposed a method for developing Ajax-based sites so that the content in those sites would be crawlable.
• Vanessa Fox provided some helpful interpretation on Search Engine Land: “Google May Be Crawling AJAX Now – How To Best Take Advantage Of It.“
• For those who are seriously considering implementation, you can read Google’s full specification.

Public Service Announcement: Because we have a few clients who are considering this specification, we reached out to Google and were pleasantly surprised that they were responsive to our questions regarding implementation. Google’s eagerness to encourage web-wide adoption of the specification means that they may be more open than you might expect in terms of communicating with early adopters. If you are seriously considering implementing Google’s Ajax crawling specification, there is no better time to reach out to Google directly with your technical questions.

Here are some examples of sites using Google’s Ajax indexing specification, and a look at their current indexing status in the Google search engine:

Holiday Inn

The site Holidayinn.com appears to have implemented Google’s Ajax crawling specification (we’ll call it Google-Ajax for short) to a limited extent.

An example of the Google-Ajax protocol can be seen on this page:

http://www.holidayinn.com/hotels/us/en/san-francisco/sfocc/hoteldetail

The Google-Ajax links can be seen by hovering over the links labeled: Rooms, Amenities, Dining and so on.  “Amenities” links to /hoteldetail#!Amenities , “Dining” links to /hoteldetail#!Dining, and so on.

In each of these URLs, the fully rendered javascript can be seen by replacing “#!” with “_escaped_fragment_=”, as seen here:
http://www.holidayinn.com/hotels/us/en/san-francisco/sfocc/hoteldetail?_escaped_fragment_=Amenities — this is the fully generated page that Google uses for indexing purposes.

You can see the “Dining” page in Google’s search results, here:

At this writing, Holiday Inn’s Google-Ajax pages have been live for weeks, but these pages are frustratingly hard to find in Google’s index.We do not know whether the incomplete indexing we’ve observed is because of a glitch in implementation, or if Google’s robot simply hasn’t gotten around to visiting all of these sub-pages.

Facebook

Facebook appears to have fully implemented Google-Ajax.  After some digging, we were able to find examples of Google-Ajax URLs indexed in Google, as seen here:

Facebook is an interesting case study, because in addition to implementing Google’s crawling specification, the site also has alternate content that is browsable without javascript (we call this “degraded” content). After several weeks of watching Google’s index, we are finding that the degraded pages are far more prevalent in Google’s index than the Google-Ajax pages. It’s redundant to have fully degraded content and Google-Ajax in place, so we wonder:  is Facebook planning to keep this redundancy for the sake of Yahoo! and Bing? (There is no official word yet on whether Yahoo! and Bing will support this specification.) Or is Facebook holding off on getting rid of its degraded content until it feels more confident about Google-Ajax? Or are other factors at play?

Google Itself

An excellent example of a successful implementation of the protocol can be found at Google’s GWT showcase site:

http://gwt.google.com/samples/Showcase/Showcase.html

Here are some pages within the showcase:

http://gwt.google.com/samples/Showcase/Showcase.html#!CwCheckBox
http://gwt.google.com/samples/Showcase/Showcase.html#!CwFileUpload
http://gwt.google.com/samples/Showcase/Showcase.html#!CwBasicPopup

A Google site search within the Showcase section shows 154 pages have been indexed from within the Showcase – including pages that are clearly within the Ajax experience:

It’s readily apparent from looking at Google’s cached versions of these pages that Google is indexing the content on each individual page within the Ajax interactive experience on the site. We can conclude that Google web search is indexing Google-Ajax pages just like any other webpage.  We would call this one a total success, which is hardly a surprise, since it’s Google’s own implementation.

It does not appear that Yahoo or Bing have indexed any of these pages.

Bookwhack.com

Bookwhack.com appears to have fully implemented Google’s Ajax Indexing Specification.  Unlike the Facebook implementation, this site relies fully on the Ajax crawling specification to generate crawlable text. We’ll be watching this site closely.