Q: Hello! I am a consultant in web analytics, and I'm looking for a tool that simply counts how many pages the website has, so I can calculate the inclusion ratio, or the percentage of pages indexed in the search engine. It seems like such a basic thing, yet I've been literally searching for hours and can't find what I'm looking for. Can you help?

A: What a great question. Although we don't know of a tool that is designed to do specifically what you're describing, we suggest you try an XML Sitemap generator. These tools are designed to create, basically, a list of all the pages on a site, so you could get the total number as a fringe benefit.

For a small site under 500 pages, you could use the online version at http://www.xml-sitemaps.com/ . For larger sites, you can pay for the standalone software available on that site, or review other sitemap generators at: http://code.google.com/sm_thirdparty.html

Another option would be an application such as SiteCrawler, designed to download and crawl an entire website.

Be warned: any problems that Google is encountering while trying to index your website (such as pages accessible only through javascript form submittals) may also be encountered by these other software programs. Getting an accurate number using any of these tools will be almost impossible. You might be better off using a trend as your metric - rather than total percentage indexed, you can report on the (+) or (-) change in number of pages indexed over time.

***
posted 7.1.2008

Like what you see here? You may wish to:
> Read more Ask The Experts answers!
> Buy our book, Search Engine Optimization: An Hour a Day
> Learn about our SEO consulting services