All major search engines have billions of web pages indexed, and they may generate thousands,
or even millions, of search results for many queries. This is true.
Here is the proof
*:
Let's search for
constitution using the major search engines. This keyword should produce
many, many results. Indeed:
- Google claims to have 119,000,000 results.
- Yahoo! claims to have 88,600,000 results.
- MSN claims to have 13,983,163 results.
Let's browse and see if they really can display this many search results.
Google.
In the beginning, all goes well:
- ...
- Page #10: Results 91 - 100 of about 119,000,000 for constitution
- ...
- Page #55: Results 541 - 550 of about 119,000,000 for constitution
- ...
But very soon we come to a problem.
Page #84 seems to be the last,
although at the top of the page it says:
Results 831 - 834 of about 119,000,000 for constitution.
Well, it's not really the end. At the bottom of the page we can see a note:
"In order to show you the most relevant results, we have omitted some entries very similar to the 834 already displayed.
If you like, you can repeat the search with the omitted results included."
OK. If you really need all search results, most likely you will need those omitted, too; so let's repeat the search.
Now Google says that it has about 107,000,000 results for
constitution.
It is 12,000,000 less than it told us it had found just a couple of minutes ago. What happened to those
12 million search results? Actually nothing. You may have noticed that Google always adds
"of about"
before the number of search results.
At Google, they have so many indexed web pages that it is difficult for them to count the number of search results
for every query every time someone uses Google. Instead, they provide us with an estimation.
We can only guess how they calculate this estimation, but it depends on many parameters; you may see
different numbers for the same queries over time. One of the parameters seems to be whether similar results need
to be included. This teaches us a lesson that we should treat big
numbers
provided by search engines not as absolute values but as
estimations only.
Let's return to the search results. This time the last page is
Page #100 (
Results 991 - 999 of about 107,000,000 for constitution).
And there is no links to get any more search results.
You might think that using the Google API you can get more search results. Wrong.
The Google API has an explicit limitation of
1,000 results per query, too.
What about
Yahoo? You will obtain the same result.
The last page is
Page #100
(
991 - 1000 of about 88,400,000 for constitution) and it displays a message similar to Google's note:
"In order to show you the most relevant results, we have omitted some entries very similar to the ones already displayed.
If you like, you can repeat the search with the omitted results included."
But even with all the results included, the last page is still
Page #100 (
991 - 1000 of about 88,400,000 for constitution).
And the Yahoo! Web Search API also has a limitation of
1,000 results per query.
Now let's move on to
MSN. No surprise here.
Page #100
(
constitution. Page 100 of 13,983,163 results) displays the last of the search results with
1,000 results total. Not bad, by the way. Not long ago they had a limit of only 250 results per query.
Don't be fooled by their user interface. The 100th page contains links to pages #101, #102, #103, and #104,
but when you click them, you still get the same results shown on page #100.
We cannot speak for other search engines, but the obvious answer is that
this helps them reduce operational expenses while maintaining a reasonable response time for the millions
of requests they receive daily.
Search engines are geared to average web surfers with average needs, and 1,000 results per query
is more than enough for them. Most surveys say that the majority of users only look at the first page of search results,
and only an insignificant number of searchers go farther than the third page of search results.
If you are not an average web surfer and your needs are more sophisticated,
you'll need a specialized solution.
FirstStop WebSearch? aggregates search results from multiple search engines whose search results don't
fully overlap. In this way you can get
. The more
independent search sources you use, the more search results you will get. Moreover, FirstStop WebSearch?
gathers the results much faster than any human can do it. It then compiles them in a single list,
automatically removing any duplicates. This large volume of search results
from
within FirstStop WebSearch? or exported to popular office applications.
