Objectively Comparing Competing Search Engines?

aendeuryu asks: "My default search engine of choice is, like most of you I assume, Google. That said, some complaints about Google over the years do seem to have some merit -- basically, that sometimes the indices aren't always updated, that it's too easy to manipulate via googlebombing or legislation, and that maybe too many of its featured services never get out of beta stage. Maybe the fact that Google has gone so long without significant competition is enough to make one at least begin to ask questions about it possibly becoming stagnant. Personally, I'm so used to doing things the Google way (and achieving acceptable results quickly) that I'm not really interested in switching -- case in point, all the above links referenced were quickly found via Google. However, what am I missing out on by not giving (for example) Yahoo search a shot? Or, more to the point, how would one go about trying to effectively and objectively compare competing search engines? In what areas have people found Google to have become obsolete for their purposes? Have less ignorant people than myself figured out ways to test a competing search engine's efficacy for themselves?"

Dont bother

[nb caffeine]

If you know how to use google to achieve your results, whats the issue? If a better search comes along, im sure it will be posted on slashdot (twice), so you dont need to worry about missing out.

Subjective

[fembots]

I think you have said it already, Google is good for returning acceptable results quickly, but acceptability is something very subjective.

Even by comparing keyword search side by side, one can still consider a worse result better, but who's to judge except the user?

I kept using Yahoo until it's not giving me results that I think are good enough, then I switched to Google, and I'll keep using Google until it's not returning good enough result.

I hate to say it, but...

[Raindance]

I hate to say it, but I think your quest to directly compare search engines "objectively" is pretty problematic.

Frankly, I think you're on the right track when you ask, "What am I missing out on by not giving Yahoo search a shot?"

Likewise, I think you're on the wrong track when you go on, "Or, more to the point, how would one go about trying to effectively and objectively compare competing search engines?"

Comparing the results of searches is necessarily subjective. Only that first question has a real answer.
RD

Re:Appalling

[adjwilli]

Every bittorent user has blood on his (or hers) hands.

Is this person serious?

Re:I use Google for almost everything

[Tribbin]

Try deviantart.com for wallpapers. I can spend hours just clicking along these art-galleries.

Any algorithm can be gamed

[Ars-Fartsica]

This is the dilemma for any centralized algorithm, as soon as you are number one you are exploited, thus relatively increasing the utility of as-of-yet unexploited competitors.

Wikipedia

[chiapetofborg]

I love wikipedia. I basically use it as my default search. Unless I think that the question I have is non encyclopedic. acronymfinder for acronyms, babelfish for translations, imdb for movies, and well, for everything else I use google. It has integrated everything else I need. Yes it is subjectable to googlebombing and similar ilk (I should know, I work for a SEO company), but its *way* easier to "hack" Yahoo, MSN, Altavista and others. Googleboming is much harder (and therefore more reliable) than the others.

Google is not Microsoft, and therefore is good.

[IGnatius T Foobar]

Yes, it's really that simple. We need a few big, strong, non-Microsoft companies out there keeping Microsoft from becoming even more all-encompassing. It is good to back non-Microsoft technologies whenever we can. It's best to back totally open technologies, but non-evil corporations like Google are a good second choice.

Remember this -- never forget this -- once Microsoft takes monopo-ownership of something, it's nearly impossible for anyone to take it away from them. Google's strong lead in search (and increasingly in other Internet services as well) helps to keep things at least a little balanced. Imagine a future in which Microsoft owns search and webmail as well? Sooner or later everything would be IE-only, and eventually Windows-only, and Microsoft will have completed its goal of effectively taking ownership of the Internet.

A good policy to go with is to simply always go with the strongest non-Microsoft choice available when choosing any product or service.

Punctuation

[Daetrin]

If in the process of comparison you find a search engine that can actually handle punctuation please let me know. Altavista used to be able to do it, but sometime in the last few years in the process of "competing" with google they dropped the feature.

I _used_ to go to altavista everytime i had a search that involved specific punctuation, usually some kind of coding question. Now i just get frustrated with google while trying to find some related term i can add in that will give me the results i want.

Why Google works

[onyxruby]

OK, here it, why Google has become king.

1. Simple interface, quickly loads.
2. No graphical Ads
3. Paid results are clearly ads and seperated from real results.

That's it, that's why Google is king. Until Yahoo, MSN search, Ask Jeeves and the like get those three points, they will continue to be second fiddle.

One way to test

[mcguyver]

I usually test search engines by typing in popular keywords that spammers generally go after, ex:

phentermine
home loans
poker
mesothelioma
viagra
miserable failure

Then look at the sites that rank at the top. It's very easy to tell which search engines are more succeptible to manipulation. A quick look at the backlinks for sites favorably ranking in those competitive keywords tells you how that SE is doing.

Here's my opinion on the race between Google, Yahoo & MSN. Google has more sites that are authorities in the top results and Google penalizes over optimization however extreme examples of over optimization continue to show up in Google. Yahoo is a moderate success and does a fair job of filtering out spammy sites as well as authorities like wikipedia - wikipedia will always rise to the top in G but not in Y - and this is good for Y because you get more variety. MSN does an average job of filtering out blog spam but new sites are too favorably ranked and this is because MSN is new and has no recorded history of URLs. My personal preference is to use G simply because it loads the fastest in my browser... Maybe it's also worth pointing out that my company has several URLs ranked favorably in the terms listed above - looking at the change in rankings over time certainly helps give insight into which SE is better. MSN & Y are by far easier to manipulate than G but G gives the most traffic.

Re:Dont bother - why? Parallel to OS Wars

[drhamad]

I know how to use Windows to achieve the necessary results better than the Mac or Linux. Does that means I should never try to use the Mac or Linux? Does that mean that I won't achieve better results if I learn to properly use the Mac or Linux?

Needs to be more complex though

[Sycraft-fu]

Because relivance is more complex. There's a number of additonal considerations:

1) HOW relivant is a page, and is that page more highly ranked? It doesn't do me any good to have 99 slightly relivant results and 1 highly relivant result, if that one is at the end. So you have to measure how relivant the page is, and how high it appears in teh search and weight that.

2) The ability to find the correct page. Sometimes it's not that you are looking for general inforamtion on a topic, there's a specific page you want. However you don't know the URL or how to get there. Maybe you saw it once and have a vague memory, maybe you just heard about it, whatever. In this case, it's a question of how quickly the engine gets you the correct answer, both in terms of how high it's ranked, and how many search variations you have to try.

3) Along those lines, the ability to deal with degraded input. Sometimes it's as simple as a spelling error, but sometimes it's the searcher misunderstanding their own question. They don't know precisely what they want. Maybe because they only have a vague idea, maybe because the term they remember for it isn't quite right, whatever. So how well can teh search engine figure out what they really want and find that?

So there's lots of things like that to consider as well when you are using a general purpose web search enigne. Really only personal experience can tell you if one works well for you at finding what you want.

Re:Questions

[Yurian]

An excellent question-answering engine is BrainBoost. [brainboost.com]. It's currently slow, but damn impressive sometimes.

The other day I needed to know, for obscure reasons, the number of heroin addicts in Dublin. This is the kind of info that you know is probably on the web, but is going to be hard to find with Google.

I used BrainBoost - "How many heroin addicts are there in Dublin?" [brainboost.com], and, bam, first line of the result - "There are 13,000 heroin addicts in Dublin."

That's damn impressive. Out of curiosity I tried to see if I could find the same info with Google - it was fairly tough. Took three or four searches, eventually resorting to

"there are * heroin" Dublin

which is a fairly specialized search that average users probably wouldn't be able to construct. The BrainBoost search, on the other hand, was completely natural, my granma could have done it.

So, thumbs up for BrainBoost for question answering.

Still, it's not the kind of thing you'll want every day. For day-to-day search, Google is the tool, but BB is worth a look.

Re:Precision and Recall

[sasami]

It's also important to notice that you need both precision and recall, because the two degenerate cases are useless: you can get 100% recall if you just return every page in the world (and then your precision is zero). Or you can have 100% precision if you just return a single relevant page (then your recall is, roughly, zero).

--
Dum de dum.

Re:Dont bother

[ShieldW0lf]

Thing about google that really pisses me off is that for a lot of the things i search on, it pops lots of sites that require pay registration and doesn't make it clear. I'm not going to pay for these sites when there are so many free resources available to find my answers, but I have to waste my time visiting them over and over. Experts-exchange [experts-exchange.com] in particular really pisses me off. I don't know if there's something better out there, but I'm actively looking for one, because using google just wastes too much of my time.

I know everyone loves google, and I use it too, but I find that where it used to be an efficient way to find information, it's becoming less and less so as time goes on because of this sort of crap. As far as I'm concerned, if I need to pay to access the information, google should not be indexing that information and putting up links to the sign up page for me to waste my time with when the answer is already freely available elsewhere and that freely available source is in their index. If I wanted to use pay sites to provide my answers, I wouldn't be using google in the first place, would I?

Re:Dont bother - why? Parallel to OS Wars

[Anonymous Coward]

if ((performance_increase * time_spent * value_of_work) > (new_interface_learning_curve * your_learning_speed * amount_lost_work * value_of_your_time) {

learn_new_interface();

} else return null;
}

In other words, depending on how much you do with it and how long you spend using it, if the performance increase is greater than the amount of lost time, then go for it. Otherwise don't bother.

Re:Dont bother

[anagama]

• overstock.com

Sites like this come up all to frequently, even in google. What be really sweet, would be a way to EXCLUDE certain sites. Maybe it's already possible ... I'm sick to death of the cruft that shows up in a search.