AlltheWeb Database Dies! (But Lives at Lycos)

The AlltheWeb site is still up and looks quite similar, but Yahoo! has changed the underlying database and removed many of the great advanced features that helped make AlltheWeb such a great search engine. The advanced search page has lost the following options:

• FTP database is gone
• Field searches (drop down) for 'in the URL,' 'in the host name,' and 'in the link to URL'
• Boolean operators must now be in UPPER CASE (and NOT replaces andnot)
• Afrikaans, Basque, Byelorussian, Faeroese, Frisian, Galician, Indonesian, Latin, Malay, Serbian, Swahili, Ukrainian, Vietnamese, and Welsh language limits but gained Farsi
• Media type inclusions
• IP range limit
• Flash indexing and limit
• Size limit

In addition, we have lost access to the indexing of the full text of large Web pages and PDF files. Yahoo!'s database stops indexing at 500KB. While not unexpected, this is a major loss for many searchers. This will likely be a progressive roll-out, so some users may not see the change quite yet.

One small bit of good news is that for the moment at least, Lycos continues to provide access to the old AlltheWeb database (the FAST Web Search database). How much longer this will continue is unknown, and Lycos lacks most of the options on their advanced search form that AlltheWeb offered, but at least the database is still accessible a bit longer.