Building a Search Engine Using Open Technology? 42
cybrthng asks: "Mozdex.com is my attempt at building a search engine capable of indexing the entire web. Our goal is to provide a completely transparent system utilizing open technologies such as Nutch, Lucene and other systems to provide a search facility that is more scientific and 'protocol' vs the current propriety and almost 'faith based' search engine results and methods of getting listed. What do you look for out of a search engine? What would you look for out of this project? Should large commercial entities be the only way we find information and resources on the net? BTW, our beta index currently has about 50 million pages and we hope it shows what can be done using Open Source systems available today. We are seeking input on starting a developer & input community as well as getting concepts and ideas out and about, so we value your ideas and what you hope to see out of this project."
how is this different? (Score:3, Insightful)
Is it different only because it runs on open source software? Hell google does that successfully already.
Re:how is this different? (Score:4, Interesting)
Re:how is this different? (Score:3, Informative)
ALthough Mozdex appears to be of good will, notice that the GPL does not force them to distribute changes to GPLed code as long as they're the only ones using the code. THe GPL would only be effective if they would try to distribute changed binaries, but they do not distribute anything other than HTML web content. This could become a major headache with the GP
Re:how is this different? (Score:1, Funny)
Step 1: Create P2P search engine technology.
Step 2:
Step 3:
St
As a webmaster (Score:2, Insightful)
Re:As a webmaster (Score:2, Informative)
Yes, Nutch is designed to be a good bot and follow the normal rules, but just like any open source project, it could potentially be used badly by someone.
More information can be found on the Nutch Webmaster Information Page [nutch.org].
Open source search engine? (Score:4, Funny)
Re:Open source search engine? (Score:2)
Yeah, but so will everyone else. And it's a zero sum game.
Re:Open source search engine? (Score:4, Insightful)
Re:Open source search engine? (Score:1)
Subject/Topic based filters (Score:2)
Searching for "Jaguar" the fighter bomber as opposed to "Jaguar" the comic book character.
But then, I'd want you get into natural language processing to determine what the real "topic" was that I meant. Of course I'm assuming a free form field. I'd like to just be able to put in "Jaguar the bomber" or "aeronautical: jaguar" or "plane jaguar" or even "plain jaguar" and have it do a Googlesque "Did you mean 'plane Jaguar'".
Hmm, and a fun API so you could
Re:Subject/Topic based filters (Score:2)
Re:Subject/Topic based filters (Score:4, Interesting)
I run a search site that only indexes a few hundred other sites and around 170,000 files (today). What the files contain doesn't matter here. What's significant is that the data, while being (usually) plain ascii text, is not in any human language. If you saw it and didn't know the subject area, you wouldn't be able to make sense of it. It's very useful to a few thousand users, and of no interest whatsoever to anyone else.
One thing that could be feasible with an open-source search project is to discuss ways in which specialized search engine like mine can be incorporated. The data that I index can be related to several other kinds of online data that are in turn indexed by others. But my code doesn't make the connection, and neither do the search engines for the related types of data.
This strikes me as a significant problem that the big guys can't much work on (yet). And, like "orphan" drugs, they probably won't ever find it worthwhile to work on most kinds of data that only exist in a few thousand files.
But if we could define a way to interface search engines so that they can recognize each other and refer queries to each other, then these specialized data formats could be usefully searched and indexed.
Sounds worthwhile to me. I wonder if I could find someone to pay me a salary while I worked on it?
Re:Subject/Topic based filters (Score:2)
Re:Subject/Topic based filters (Score:2)
In general, I'd think that to solve this problem, you wouldn't want to look too closely at specific examples, other than to become convinced that each specialty really is going to need its own parser and syntax analyzer.
A good traditional examp
Re:Subject/Topic based filters (Score:3, Insightful)
Re:Subject/Topic based filters (Score:2)
Re:I look for... (Score:1, Informative)
There may be parallels in encryption... (Score:2)
I think of the Dow and other financial indices and believe that the proprietary model may be the only successful way to provide useful, reliable information.
Then I look at encryption, and I see how the algorithms, being public, can be vetted without compromising the security of the communication through a proprietary, secret key.
I suspect t
Rejection penalty (Score:2)
results of course (Score:4, Insightful)
The only thing that matters is results. Is the answer that I need in the first three or four results? If you can do that, you win. If you can't, don't bother.
I'm skeptical about how realistic it is to develop an open source search engine. Wikipedia [wikipedia.org], although cool, has large gaps in content, and only a few months ago was begging [slashdot.org] for donations to survive. I'm betting that a Google sized operation would be even more resource intensive.
Re:results of course (Score:3, Insightful)
Well, Wikipedia did get almost $30K in donations that time and is still getting lots of donations from what I gather, and could easily get a lot more whenever it wanted because lots of people LOVE that project, so that part is successful.
As for the larg gaps in content, it is being worked on everyday. That's
What I like to see the focus (Score:1)
More explaining on "explain" would be useful (Score:3, Interesting)
What I have found REALLY interesting about MozDex is the "explain" button which I assume provides some insights into why MozDex decided to rank that web URL as whatever ... but the information as currently presented isn't understandable and/or explained.
For instance, I was interested where a Google Compute [powder2glass.com] web page came up and was actually quite surprised that a MozDex Search shows it as #1. [mozdex.com] So I click on the explain button and I get a page with a buncha numbers ... but nowhere on this page (or anywhere on the MozDex site) can I find an explanation for what they heck they mean.
Since your claim-to-fame is open source/search, I think adding information on the internal algorithms would help you out. Keep up the good work - interesting stuff! ;-)
alek
P.S. Minor typo in the Corporate Info link from your FAQ [mozdex.com]
what i look for in a search engine.. (Score:1)
- results not easily manipulated (eg. http://your-search-keywords.com/your-keywords/key
- fast discovery of new or updated sites
- features such as caching, view as X, spam reporting
Mozdex using Nutch sponsored by Overture which is (Score:5, Informative)
While browsing the Mozdex site, I learned they are using Nutch, an open source search enigine. So I started browsing the Nutch site. On their site I found out that they are sponsored by Overture Research ... The name seemed familiar. Clicking on the link I arrived at http://labs.yahoo.com [yahoo.com].
Apparantly Yahoo is rather interested in this project. Browsing the Yahoo Labs site I found this page [yahoo.com](which is also the third hit when googling for nutch): "Welcome to the Yahoo! Research Labs implementation of the Nutch open source search engine (www.nutch.org). This search engine is intended as a demonstration platform for a number of search related technologies that we are working on and is specifically not intended to provide a full and comprehensive search experience for the average user. If you do a search here, please do not be surprised or offended if your favorite site is not in the result set for your query.
With this in mind, please feel free to test drive the technology. Happy Nutch-ing.
A very quick test shows that the 50 million pages counting index of mozdex is indeed still far to small to really find something. The ranking system will also need some tweaking, but this is also clearly stated on the nutch site: "Nutch has not yet been tuned for quality. There are ten or twenty knobs that we can twiddle to adjust the ranking formula. We are developing software to do this tuning automatically, but the current code just contains guesses. With a little tuning we should be able to get results that are competitive with those of major search engines.".
Although it is currently not possible to do any real comparison due to the big difference in the number of indexed pages, it sure is nice to see both the Nutch project and the Mozdex project. I hope that both of these project will receive enough funding (and hardware) to continue, and maybe we'll see another /. post when they hit the 5 billion page count and we will be able to do a massive comparison ... and all change from googling to nutching or mozdexing!
One to watch
Filter commercial sites (Score:4, Insightful)
Searching on Google for things like reviews of mp3 players has become a nightmare these days. Any useful sites are drowned out in a noise of pricerunner/dealtime/kelkoo/shopping.yahoo/etc and other sites that are simply affiliate sites for Amazon etc.
Re:Filter commercial sites -MOD UP IF YOU WILL (Score:1, Insightful)
MP3 player review -buy -dealtime -pricegrabber -kelkoo -shopping.com -amazon.com -nextag.com -bizrate.com -moreover.com -celeb -porn -free -coupon -pimprig.com
I have it set to a hotkey for all the -'s and boom relatively valid search results.
A gift from a marketroid to you techies.
Re:Filter commercial sites -MOD UP IF YOU WILL (Score:1)
httrack and grep... (Score:1, Funny)
Download the internet with httrack and search it with grep.
In over your heads? (Score:3, Insightful)
Writing the search code itself is not too hard (you still need a PhD in data structures and algorithms, but those can be found), the real hard part is the amount of bandwidth and CPU power that is required.
A different name (Score:4, Interesting)
You need a name that is as easy to pronounce as google. As friendly sounding would be good as well.
You're "competing" on a number of different areas with google, including the name ofcourse.
The first thing that came to my mind when I read the name was: "Typical for geeks who are good at the technical side of things, but are bad at marketing and the human interface/psychology side".
Results Baby (Score:2)
My two tests were 1: "Hattrick" which is an online soccer management game, and is great. Google it and up it comes with some handy links to some sites about it. Using this engine, I got a bunch of crap. It may have been pages that linked to hattrick, but I didn't check.
2: "Buyer Agent Boulder Colorado" - Exclusive buyer agents are the preferred way of buyi
Easy! (Score:2, Funny)
Completely Open Source!
It's broken (Score:2)
Back to the drawing board.