Objectively Comparing Competing Search Engines?

aendeuryu asks: "My default search engine of choice is, like most of you I assume, Google. That said, some complaints about Google over the years do seem to have some merit -- basically, that sometimes the indices aren't always updated, that it's too easy to manipulate via googlebombing or legislation, and that maybe too many of its featured services never get out of beta stage. Maybe the fact that Google has gone so long without significant competition is enough to make one at least begin to ask questions about it possibly becoming stagnant. Personally, I'm so used to doing things the Google way (and achieving acceptable results quickly) that I'm not really interested in switching -- case in point, all the above links referenced were quickly found via Google. However, what am I missing out on by not giving (for example) Yahoo search a shot? Or, more to the point, how would one go about trying to effectively and objectively compare competing search engines? In what areas have people found Google to have become obsolete for their purposes? Have less ignorant people than myself figured out ways to test a competing search engine's efficacy for themselves?"

dogpile.com

[Bad-JuJu-Man]

personally I prefer dogpile. I like the organization of results much better.

Alternates

[hambonewilkins]

Here's my own experience.

Yahoo search is okay, not as nice as google, but a good second.

Alltheweb.com has found things google hasn't, but in general I rarely use it.

I rarely use MSN because it was awful all the times I tried it. Same for Altavista.

In general, if I'm searching for something I'll use google first and then Yahoo and Alltheweb to catch anything that google may have missed.

I use Google for almost everything

[Snarfangel]

...but I have to admit the AltaVista search engine for pictures is pretty nice. I use that when I want to search for pictures of a particular size for wallpaper.

I quite like Google.

[Morlark]

Now sure enough Google has its faults, but I do still use it as my primary search engine. I do dislike Google never-ending cookies, so I've blocked them, and my Google bookmark contains all my preferences. I've not really noticed any problems with Googles indices not being updates (except in the silly image search, and I don't really use that for any serious purposes). Having said that, I also do find Yahoo to be a very acceptable alternative. I should probably try it out more so as to see how they compare in greater detail.

One of the great things about search...

[PornMaster]

is that if you can't find what you're looking for with one engine, try another. I don't see what's so hard about that. Any synthetic benchmark will be just that. It's not like you have to change your voter registration to another party in order to vote in a primary -- you put in another URL!

To help you out, I'll even get you started with a few clickable links...

Yahoo Search [yahoo.com]
MSN Search [msn.com]
Ask Jeeves [ask.com]

Hope that helps. Good luck.

Precision and Recall

[xyzzy]

Unfortunately, comparing search engines is a nearly impossible task, since they probably aren't indexing the same data.

When you measure a search technology, the values you typically look for are precision and recall. precision says "of the X results you gave me, how many of them are relevant". recall says "in the world, there were Y possible pages you could have found, but you gave me X of them".

you can't measure recall for a public search engine, but you can measure precision. Take a set of sample queries, and some users. Have them perform the queries, and go through the first ~100 pages and give them a "thumbs up" (relevant) or "thumbs down" (not relevant).

Your overall score will measure precision: if at N=100, all 100 were relevant, that's 1.0. if only 50 were judged relevant, precision is 0.5.

You can estimate recall by judging say 1,000 documents (phew). Then sample precision at N=10, 100, 500, etc, assuming that is an "exhaustive" list of documents in the world.

Re:Questions

[TheViffer]

It will cost ya.

Google Answers [google.com]

Re:Try them out yourself

[Anonymous Coward]

it may be flmaebait but the parent has a point. This is a dumb question.

Search Engine Watch

[br0ck]

These types of issues are discussed ad infinitum at SEW [searchenginewatch.com].. particularly in the forums.

Re:Alternates

[krgallagher]

" Here's my own experience."

Here is my alternative. It is called Copernic Agent [copernic.com]. It is a desktop application that searches multiple search engines returns the results sorted by relevance. It will then let you further refine your search by searcheing aginst the actual pages in the result list. There is a free version that is crippleware. I bought the personal version and it was my favorite tool for searching job sites when I was unemployed.

search.yahoo.com

[popo]

Many people don't realize that Yahoo! has a scaled down (Google like) search interface which is actually pretty sweet: http://search.yahoo.com

Lately my Google results have been so Google bombed that I've been going back and forth between the two. I can't say for sure yet, but I may be in the middle of a bit of a personal transition.

Depending on what you're searching for, Google is often so front-loaded with dead-end advertiser links that its results aren't really worth much. Although it has to be said, it depends what type of a search user you are, and what types of things you're looking for.

Google is still the king of advanced search.

Re:Dont bother

[TyfStar]

I will agree that 99% of the time, google is perfect for what I need. Stick a few words together, add an extra one, and VOILA, you have what you're looking for.

The times that I have had problems is when I am not exactly sure what I am looking for in a few quick words. I can put it together in a question, such as "What is my house in Utah worth?" or "Why are flamingos pink?".

in those cases, I usually do ask.com That will get me going on a few pages, at which point I will know more clearly what I'm looking for (Utah "Real Estate") and can google from there on out.

MSN's sandbox test searchpage

[standsolid]

I personally think Microsoft's sandbox search engine front-end is pretty nifty.

Too bad the search results aren't nearly as up to par as google's results (in my opinion)

http://start.com/1 [start.com]

Simple Method

[Salamander]

I've stuck with Google for a while, but I used to do surveys pretty often. My approach was to start preparing a couple of days in advance, by keeping notes about things I was searching for. Then I'd take three or four of them, usually the ones that I'd had the most trouble refining, and try them out on a bunch of search engines. For each, I'd keep track of how many searches I had to do and how many junk pages I had to get through before I could get to something useful on that subject. It usually became clear pretty quickly which search engines were allowing me to make efficient use of my time and which were wasting my time.

Another thing you might want to do is check out some of the newer "clustering" or "concept map" search engines such as Vivisimo or Kartoo, to see whether they suit your searching style better. They're really quite different from the search engines we've gotten used to, so the metrics I just described don't quite work for them. That doesn't mean they're better or worse - just different.

Re:I quite like Google.

[neil.pearce]

if you visit Google's No Country Redirect [google.com] page, it'll set a cookie to stop you being redirected to your national page when visiting google.com
Not sure why you end up at different fr/dk/... domains though

Re:Why Google works

[jerw134]

1. Check out MSN Search [msn.com]. Maybe not quite as simple as Google's, but it's far from complex. It also loads very quickly.
2. MSN Search has no graphical ads.
3. MSN Search separates the paid results just as clearly as Google does.

So, when was the last time you looked at MSN Search? Last year?

Teoma used to be good...

[Newer Guy]

Teoma (teoma.com) used to be good, but lately I find it's not giveing me a smany choices as google, nor are they as good.

Surprisingly, I still use Ask Jeeves (www.ask.com) for things - and find it finds things that Google has completely missed!

I guess you have to use a combination of several to really find everything you want - though Google by far is the best one.

I base it on bot/spider visits

[KlomDark]

Looking through the logs for my website, I see Googlebot visiting nearly every day, followed (recently) by MSNBot. (Actually, in raw count, I'm seeing that MSNbot has just recently surpassed the number of requests as Googlebot. Would need to do some in-depth analysis to see if those are requests for the same thing over & over, but in raw requests...) I pretty much never see anything from Yahoo cataloging my site.

What's weird I'm noticing is that I don't see anything from something like a Yahoo bot at http://klomdark.servebeer.com:443/analog/report.ht ml#browsum [servebeer.com], but Yahoo is giving more traffic (http://klomdark.servebeer.com:443/analog/report.h tml#refsite [servebeer.com]) than MSN.

Google still leads however. I wonder where Yahoo is getting it's data, unless it's from a crawl previous to fall 2003, as I'm not tracking logs from that far back. Strange.

Re:Dont bother

[iamhassi]

duh, google it [google.com]... answer [64.233.179.104] found on the first page:
"Flamingos are not born pink. They are white at birth. However, a substance -- called carotenoids -- in the foods they eat produce the bright pink color.

Flamingos would lose their shading if they could not eat carotenoid-filled foods like plankton, shrimp, or -- as handlers at the Philadelphia Zoo have found -- carrots. "

Search Engine Watch

[Guidii]

Or, more to the point, how would one go about trying to effectively and objectively compare competing search engines?

For some time now, Search Engine Watch [searchenginewatch.com] has provided a good editorial and comparison on various search engines. They focus on marketing topics, but also tend to talk a lot about the underlying technology, etc.

A recent roundup of engines is at http://searchenginewatch.com/links/article.php/215 6221 [searchenginewatch.com].

Re:Questions

[rsadelle]

While I don't doubt that BrainBoost works, heroin addict dublin into Google gets me "There are 13,000 heroin addicts in Dublin" in the first page of results.

Re:Dont bother

[Jaycatt]

Oh, don't get me started about Experts-exchange. Most of the time someone there is asking the exact question I'm trying to find the answer to. However, once you've paid (and I did, once, because the person I was working for was under time pressure) you inevitably discover the "answer" is something you're already tried. Just try getting your money back. Of course, after registering, you can always ask your own question. Didn't help me, though, since I couldn't wait a week (or however long it takes) for the answer. After about an hour of extra searching, I ended up finding what I was looking for. Its was just too bad I paid for Experts-exchange first.

Re:Questions

[amokrun]

That kind of engines are indeed nice. Still, they have their own oddities. For laughs, I tried to ask the system whether moon is made of cheese.

It so turns out that moon is indeed made of cheese!

"is moon made of cheese?"
"The Moon is Made of Cheese"

I guess it still takes some time before that kind search engines become more popular than the traditional ones.

Re:Yahoo seems lazy

[Anonymous Coward]

I don't use a9 much, but it seems like google with a different skin. I swear sometimes they're snarfing google's results and storing them. Not that this is all bad, since Google's results tend to be some of the best, but it's still eerie.

A9 does use Google for the main results. Their play is in customizing the interface for various types of information.

Re:Alternates

[MMatessa]

To find pages that Google misses, I use jux2.com [jux2.com].

Re:Questions

[funny-jack]

Interestingly, you also get the same result if you actually do a Google search for your original question: "How many heroin addicts are there in Dublin?" [google.com] In the summary of the first result: "... There are an estimated 13000 heroin users in the Dublin area. ..." I'll give you the benefit of the doubt here and assume that, as you have supposed, these results have changed since you did your searches.

I have actually found searching for a plain english question to work in a number of other instances, as well. Not always, but sometimes.

Re:Alternates

[XorNand]

While your list is a good start, Phil Bradley has an excellent, and extremely comprehensive, analysis [philb.com] of how to use different search engines to find different things. Definitely bookmark-worthy.

Re:Dont bother

[tepples]

What be really sweet, would be a way to EXCLUDE certain sites.

For each site that you want to exclude, add a term along the lines of -site:overstock.com to your query.

Re:Dont bother

[aminorex]

> it will be posted on slashdot

Or at least mentioned in the comments: vivisimo.com

Re:Dont bother

[meetchanakya]

I tried this on Ask Jeeves ... Web Results www.straightdope.com/columns/010518.html www.ford-trucks.com/dcforum/DCForumID64/45.html www.honoluluzoo.org/american_flamingo.htm www.seaworld.org/infobooks/Flamingos/home.html www.thewildones.org/Animals/flamingo.html www.yahooligans.com/content/ask_earl/20030611.html www.expage.com/doyoulikepinkflamingoes All top six results have what you are looking for :)

Re:Dont bother

[CvD]

Um... yeah, you can pay, but you can also scroll down about 2 pages of advertisements and a listing of catagories and find the answers lodged beneath. Unless there are yet more answers to these questions?

For example: a question about Java [experts-exchange.com]. The question first, then the SIGN UP! bla bla, then a bunch of catagories, but if you scroll down further, you'll find answers to the questions, including the 'accepted answer' and such.

Hope this is useful to someone.

Cheers

Re:Dont bother

[Hank the Lion]

When you are looking for a phrase, instead of loose words, enclose them in quotes on Google.
Googling for "The Who" gave me mostly relevant results.

Re:Dont bother

[joeykiller]

If you had read the entire thread, you'd see that the topic was searches phrased as questions - "Why are flamingos pink", etc. Someone pointed out that ask.com was best at answering these kinds of questions, while someone else proved that Google was best.

I merely tried to point out that in some cases -- such as this -- searches phrased-as questions can return no relevant answers at all on Google.

Another thing: I may have composed my search in an idiotic fashion. But don't you think most people are idiots, in the way that they ask naïve, corteous questions to Google and expect an answer? Very few people actually reads Google's search tips before searching.

Re:Dont bother

[Peer]

Experts-exchange.com does require registration to see the answers, but the google-cache always contains the full page with the answers.

Re:Major reason Yahoo is better...

[nmg196]

-1, Wrong
This is absolute rubbish. Google DOES crawl dynamic pages quite happily. It's crawled all of my sites with no problem.

Neither (no) search engine crawls dynamic sites where there are no links to the dynamic content (eg where you HAVE to search using keywords to find the content) but Google and Yahoo are happy to index any dynamic page which is directly linked to even if it has lots of parameters in the URL. Google has indexed 15000 dynamic pages on a directory site of mine quite happily.