I seem to be on a roll lately in finding Web pages that are not indexed by Google and are only found by one search engine. Yesterday I was exploring LibraryThing, a social networking and book cataloging site. I became curious as to how well the search engines covered the personal pages that people create in a social networking site like LibraryThing. So I took a look at two user profiles, grabbed a unique-looking phrase from the profile page and checked to see which, if any, of the main Web search engines could find it. Each page was found by only one search engine, but it was not the same one!
You can watch the brief screencast to see the winners in this example.
If the embedded version above does not work, try it direct at YouTube. Don't want to wade through it? Ask and Live were the two winners yesterday, although now when I try the search at Live again, it does not find the page. (I'm glad I recorded the search yesterday.) Each found one of the profiles, but not the other.
I also tried the same searches at Clusty and Dogpile for a quick comparison of those meta search engines. One of the problems with such meta search engines is that they sometimes work and sometimes do not. For these two searches, Clusty found the pages from Ask and Live. Dogpile found the one from Live but not the one from Ask. Unfortunately, as seen at the end of the screencast, Dogpile had extra hits that were all from an ad database. For the first profile page, Dogpile displayed seven ads and did not find the hit from Ask (even though it reputedly searches Ask). I just checked again, and Dogpile only has six ads for that search now but still fails to find the Web page that Ask itself can find.