Who works with whom in the US search engine market. Interrelations, between search engines and directories

Who works with whom in the US search engine market. Interrelations between search engines and directories

Search engines and directories interrelate with other search engines. They share their database contents to cover more web pages and they partner with pay per click search engines to supplement their revenue

It's essential to know which search engines and directories work together. If you want to plan an effective search engine optimization strategy, you must know which search engines cooperate and who gets results from whom.

How Search Engines Work

1. Who Works With Whom in the US Search Engines Market

2. Where Search Engines Get Results

3. Search Engines: Table of Features

4. How Search Engine Database Works

Who works with whom in the US search engine market?

This table tells you which search engines and directories get their search results from whom and will help you in your future search engine marketing efforts: if a search engine doesn't list your web site, you can try to submit to the partner engine which provides secondary or directory results.

- For the world major search engines list see: World Major Search Engines List

- For the Australian search engines list: Australian Search Engines List

Where search engines get results

AllTheWeb	Gets search results from Yahoo! and paid results from Overture.
AltaVista	Mergered its database with Yahoo! but also gets directory results from the Open Directory Project, and paid results from Overture.
AOL Search	Gets normal search results and paid results from Google, as well as directory results from the Open Directory Project.
Ask Jeeves	Has its own database but gets additional results from Teoma and paid results from Google.
Google	Has its own database but gets directory results from the Open Directory Project.
HotBot	Gets search results from Inktomi, directory results from the Open Directory Project and paid results from Google. HotBot can also display unfiltered search results from Ask Jeeves, Google and Lycos.
Inktomi	Mergered its database with Yahoo!
iWon	Gets search results Google, directory results from the Open Directory Project and paid results from Google.
LookSmart	Has its own database but gets additional results from Inktomi.
Lycos	Gets search results from FAST Search, directory results from the Open Directory Project and paid results from Google.
MSN Search	Gets search results from Inktomi and LookSmart. MSN Search is currently working on its own database.
Netscape Search	Gets search results from Google, directory results from the Open Directory project and paid results from Google.
Overture	Has its own database but gets additional results from Inktomi.
Teoma	Has its own database but gets directory results from the Open Directory Project.
Yahoo	Has its own human-compiled database but gets addition results from Inktomi and paid results from Overture.

As of 1 January 2004. Note that partnerships and alliances change regularly in the search engine world.

Google has one of the largest databases of Web pages, including many other types of web documents (e.g., PDFs, Word or Excel documents, PowerPoints). Despite the presence of many advertisements and considerable clutter from blog sites and newsgroups, Google's popularity ranking often makes pages worth looking at rise near the top of search results. Google alone is not sufficient, however. Less than half the searchable web is fully searchable in Google. Overlap studies show that about half of the pages in any search engine database exist only in that database. Getting a second opinion is therefore often worthwhile. For a second opinion, we recommend Teoma, Vivisimo (a meta-search engine that indirectly searches three huge search engine databases), or AlltheWeb. See also: How do search engines works?

Back to Top

Search engines: table of features

Search Engine	Google http://google.com/	Teoma http://www.teoma.com/	AlltheWeb Advanced Type alltheweb and click Advanced Search.	Alta Vista Advanced Type http://www.altavista.com/ then click Advanced Search
Links to help	Google help pages	Teoma help pages	AlltheWeb help pages	AltaVista help pages
Size, type Size varies frequently and widely. See tests and more charts.	HUGE. Over 2 billion. Claims over 3 billion but about 1 billion are not fully indexed (i.e., cannot be full-text word searched). Unindexed pages are retrieved if your search matches their titles or match other pages linking to them. Biggest in tests.	LARGE. Claims to have 1 billion fully indexed, searchable pages, and 1 billion more partially indexed. Strives to become #1 in size.	HUGE. Over 3 billion fully indexed, searchable pages. Sometimes ties for first in tests. Advanced Search worth mastering.	LARGE, but smaller than Google or AllTheWeb. See tests. Use the Advanced Search.
Noteworthy features and limitations	Popularity ranking using PageRankT. Limit of 10 words per search, excluding OR. Indexes the first 101KB of a Web page, and 120KB of PDF's.	Subject-Specific PopularityT ranking. Suggests terms within results to refine Suggests pages within results with many links.	No stop words. URL Investigator to find out about a page. Conversion of weights and measures.	Full Boolean searching and powerful Searching within results using SORT BY box in Advanced Search. Basic search provides distracting commercial, paid, and directory entries.
Phrase searching (term definition)	Yes. Use " ". Searches common "stop words" if in phrases in quotes.	Yes. Use " ". Searches common "stop words" if in phrases in quotes.	Yes. Use " "	Yes. Use " "
Boolean logic (term definition)	Partial. AND assumed between words. Capitalize OR. - excludes. No ( ) or nesting. In Advanced Search, partial Boolean available in boxes.	Partial. AND assumed between words. Capitalize OR. - excludes. No ( ) or nesting.	If Boolean expression is selected in Advanced Search, accepts AND, OR, ANDNOT, and ( ).	AND, OR, AND NOT, NEAR (within 10 words). In Advanced Search, or capitalized in Basic Search.
+Requires/ -Excludes (term definition)	- excludes + will allow you to retrieve "stop words" (e.g., +in)	- excludes + will allow you to retrieve "stop words" (e.g., +in)	In top box, - excludes	Available only in Basic Search. We recommend Boolean logic in Advanced Search.
Sub-Searching (term definition)	Sort of . At bottom of results page, click "Search within results" and enter more terms. Adds terms.	Sort of . Add terms. REFINE pastes suggested sub-topics within results.	Sort of. At bottom of search results. Terms entered will be added to terms previously searched.	Yes. Use Sorted by box under Boolean search box. Sorts and filters search results.
Results Ranking (term definition)	Based on page popularity measured in links to it from other pages: high rank if a lot of other pages link to it. Fuzzy AND also invoked. Matching and ranking based on "cached" version of pages that may not be the most recent version.	Based on Subject-Specific PopularityT, links to a page by related pages. More info.	Automatic Fuzzy AND. Also seems to use "importance" and links to pages. In Advanced Search, SHOULD INCLUDE gives higher priority to word or phrase in box. Each box read as a phrase. In Boolean Search, rank:word is supposed to rank by that term.	By the terms you specify in Sorted by box under Boolean search box. Relevancy ranked if left blank.
Field limiting (term definition)	link: site: allintitle: intitle: allinurl: inurl: Advanced Search boxes for most of these. Offers Uncle Sam for US federal pages and other special searches.	intitle: inurl: site: geoloc: Explanations, limitations.	In Advanced Search, can search within: text, title, link name, url, link to the url (Explanation of these distinctions.) and filter by: domain terms. Also offers commands similar to Google as Special Features.	title: url: link: host: domain: anchor: text: image: applet: Definitions.
Truncation (term definition)	No. Search variant endings and synonyms separately, separating with OR (capitalized): airline OR airlines	No. Search variant endings and synonyms separately, separating with OR (capitalized): airline OR airlines	No. Enclose variants in ( ) in top box to create OR search. (airline airlines)	Yes. Use *.
Case sensitivity (term definition)	No.	No.	No.	Yes. Upper case retrieves matching upper case. Lower case retrieves lower or upper case. Also accent and character sensitive.
Language	Yes. Major Romanized and non-Romanized languages in Advanced Search.	Yes. Major Romanized languages. Use lang:	Yes. Major Romanized and non-Romanized languages. Allows you to specify matching character sets. Read Help and Customize.	Yes, extensive list includes major Romanized and non-Romanized languages.
Limit by age of documents	In Advanced Search.	In Advanced Search.	In Advanced Search.	Yes, in Advanced search.
Translation	Yes, in Translate this page link following some pages. To English from major European languages.	No.	No.	Yes, to and from English and other languages. Click on Translate following result.

You may also wish to consult "What Makes a Search Engine Good?" - a table (PDF file) summarizing the factors I use to evaluate search engines overall.

Back to Top

How search engines database works

Search Engines for the general web (like all those listed above) do not really search the World Wide Web directly. Each one searches a database of the full text of web pages selected from the billions of web pages out there residing on servers. When you search the web using a search engine, you are always searching a somewhat stale copy of the real web page. When you click on links provided in a search engine's search results, you retrieve from the server the current version of the page.

Search engine databases are selected and built by computer robot programs called spiders. Although it is said they "crawl" the web in their hunt for pages to include, in truth they stay in one place. They find the pages for potential inclusion by following the links in the pages they already have in their database (i.e., already "know about"). They cannot think or type a URL or use judgment to "decide" to go look something up and see what's on the web about it. (Computers are getting more sophisticated all the time, but they are still brainless.)

If a web page is never linked to in any other page, search engine spiders cannot find it. The only way a brand new page - one that no other page has ever linked to - can get into a search engine is for its URL to be sent by some human to the search engine companies as a request that the new page be included. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program identifies the text, links, and other content in the page and stores it in the search engine database's files so that the database can be searched by keyword and whatever more advanced approaches are offered, and the page will be found if your search matches its content.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Pages that are excluded are referred to as the "Invisible Web" -- what you don't see in search engine results. The Invisible Web is estimated to be two to three or more times bigger than the visible web. For more information about the Invisible Web and how to find and use the web "hidden" in it, please see our web page on this topic.

As of 1 January 2004.

Back to Top

Home • Site Map • Contact Us • Price List • Webmaster • Company Profile • Links • Add a Link