You are sending a link to...
General Search Engines and the Invisible Web
I have said several times that  lawyers should not limit their internet searches to Google, but should try other general search engines such as www.ask.com.   But general search engines simply cannot find all the goodies on the Internet.  Search engines are constantly expanding their search areas, but some parts of the web remain “invisible”  to general search engines.

In 2001 Chris Sherman and Gary Price wrote a book called “The Invisible Web” and defined the Invisible Web like this: 

In a nutshell, the Invisible Web consists of material that general-purpose search engines either cannot or, perhaps more importantly, will not include in their collections of Web pages (called indexes or indices).  The Invisible Web contains vast amounts of authoritative and current information that’s accessible to you, using your Web browser or add-on utility software – but you have to know where to find it ahead of time, since you simply cannot locate it using a search engine like Hotbot or Lycos.

Many things have changed since 2001 (for example, Lycos is now powered by ask.com), but the Invisible Web remains invisible.  Gary and Chris discuss the details of the Invisible Web in much more detail, but, in general, think about it this way.  First, remember that when you use a search engine, you are not really searching the web; you are searching the database of web pages that that particular search engine’s crawler has gathered.  When the crawler goes out into the web to create its database, it is stopped – just like you and I are – when it gets to a place where it has to make a decision.  For example, the crawler is stopped when a website asks for a password to continue the search.  Without the password, the crawler cannot reach those pages, so those pages are not included in the search engine’s database.  There are also non-password decisions, such as when a website for a store asks you to choose a location before proceeding.  The search engine may not be able to make that decision and therefore cannot include all the pages from that website in its database.  The technology of search engines changes and gets better every day, so they can reach more and more pages.  We just hope that they never are able to reach those things that we think are password-protected. 

Anyway, all this to emphasize that you cannot find everything on the web with a general search engine or even a combination of search engines.  You need to get into the Invisible Web, but that requires using many specialized databases and search engines.  The Online Education Database has put together an excellent list of those resources and you can check it out here.  Bookmark the page and remember to go back to it when you need some specialized search tools. 

Thanks to Wisblawg for the Online Education Database link