Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
The Internet Technology

Understanding Search Engines? 49

Posted by Cliff from the a-primer-on-the-technology-behind-them dept.
An anonymous reader asks: "I guess by now we can be fairly certain that search engines are here to stay, and hence I'm trying to understand how the technology works. I'm not so much looking for a particular 'best' technology or implementation, but rather an overview of the different approaches and their trade-offs. Something that would teach me: which approach works in a distributed vs a centralized infrastructure; how different algorithms will perform on complete search words vs arbitrary sub-strings; or how mass storage (hard disk vs. solid state) affects implementation choices. For most mature technologies there is a host of 'overview' books and papers for my questions -- but I couldn't find anything on search engines. Where should I look? Are there any good books or papers?"
This discussion has been archived. No new comments can be posted.

Understanding Search Engines? More

Understanding Search Engines?

Comments Filter:

  • Start off by reading up aboutPage Rank (Score:5, Informative)

    by xmas2003 ( 739875 ) * on Saturday February 04, 2006 @07:35PM (#14643548) Homepage
    Google has a summary here [google.com] ... but start with the original Brin and Page paper - The Anatomy of a Large-Scale Hypertextual Web Search Engine. [stanford.edu]

    Same basic concepts apply today ... although they probably didn't anticipate the rise of Black Hat SEO [watching-paint-dry.com] which attempts to "beat" the algorithms.

  • Learn math (Score:5, Insightful)

    by 2.7182 ( 819680 ) on Saturday February 04, 2006 @07:40PM (#14643572)
    SIAM Review had a survey article on different methods recently. You need to know linear algebra, combinatorics and probability
    • IANAM (I am not a mathematician) but I recall the terms 'set theory' and 'boundary theory' being used by mathematics researchers when they were talking about search engines and ranking / grouping the results.

  • Google founder's links to publications (Score:5, Informative)

    by zoeblade ( 600058 ) on Saturday February 04, 2006 @07:45PM (#14643590) Homepage

    One of the founders [stanford.edu] of Google still has links to various publications (in PostScript format) about search engines, if that helps.

  • Class (Score:3, Insightful)

    by addaon ( 41825 ) <addaon+slashdot&gmail,com> on Saturday February 04, 2006 @07:45PM (#14643591)
    Take a class on information retrieval from your local university.

  • Easy Start (Score:3, Informative)

    by MikeFM ( 12491 ) on Saturday February 04, 2006 @07:49PM (#14643609) Homepage Journal
    Try the web. I have a short intro to search engines [kavlon.org] on my website. Many others exist. The basics aren't hard and are very effective.
    • I have a short intro to search engines on my website.
      That's an introduction to search engine optimization, not an introduction to search engine technology. The submitter seems to be interested in the technology and algorithms behind finding pages that match a particular term, not finding out how to maximize the position of their website in indexes.
      • Good point although I think you can infer the one from the other. I actually am fairly good at SEO and a lot of that is due simply to understanding how the Internet and search engines work. I've always had a thing for studying ways of indexing and searching data as well as things like AI so it's not as mysterious a field for me as it is for a lot of people.

        • Good point although I think you can infer the one from the other. I actually am fairly good at SEO and a lot of that is due simply to understanding how the Internet and search engines work. I've always had a thing for studying ways of indexing and searching data as well as things like AI so it's not as mysterious a field for me as it is for a lot of people.

          SEO can give you an insight into how particular search engines order the results they return. But it doesn't give you any idea how those search engi

          • I was thinking more along methods of indexing data and search algorithms etc. As for what kind of db to use and stuff I guess really that depends on what kind of search you're trying to do. A normal db is good enough for many problems but not good at many other problems. Is sort of like asking what makes a good program.

  • Managing Gigabytes (Score:5, Informative)

    by cariaso1 ( 674515 ) on Saturday February 04, 2006 @08:03PM (#14643658) Homepage
    Managing Gigbytes author site [mu.oz.au] Amazon [amazon.com]

    is a spectacular book on most of the underlying technologies. Although I've only read the first edition, I don't recall it talking about spidering/webcrawling. Instead it starts with building a simple index, and builds through all the refinements (ie stemming, etc) until you've built a serious workhorse for mining text documents. Its definitely at the core of what a search engine does,

  • One interesting (and perhaps unexpected) place (Score:4, Interesting)

    by Snarfangel ( 203258 ) on Saturday February 04, 2006 @08:04PM (#14643662) Homepage
    Look up voting methods, with keywords like Kemeny, Condorcet, and Borda. A lot of search engine algorithms are like vote aggregation methods, where each site "votes" for other sites it has links to. There is quite a bit of stuff on spam page filtering and the like as well.

  • Comment removed (Score:4, Interesting)

    by account_deleted ( 4530225 ) on Saturday February 04, 2006 @08:14PM (#14643688)
    Comment removed based on user account deletion

    • Re:Related Question: (Score:3, Informative)

      by Denyer ( 717613 )
      Write in the language of the users you expect to use your site, and look at the server logs to see what terms people were using when they found your site.

      If you sell poultry medication, for example, there's no point in only labelling products as being for cockerels if your visitors are more likely to use rooster as a keyword. You might also want to refrain from putting "cock pills" in your meta tags...

      Other than that, write semantically valid code (header tags, etc) and don't put large blocks of navigat
    • Although meta tags are largely thought to be a waste of time these days (from a search engine p.o.v.) you might still consider using "description". This is used by 'Google' and probably others to replace the snipet of your page with an actual description.

      It depends on what you want to achive tho. GameSpy always uses "GameSpy is the most complete source for [insert game here] trailers, screenshots, cheats, walkthroughs, release dates, previews, reviews, ..." for every game it covers even when it's clearly no

    • Re:Related Question: (Score:3, Insightful)

      by Flwyd ( 607088 )
      Look at your page with Lynx.

      Web crawlers can't see the text in your images and weird HTML constructions can make it hard to parse the text back out. If your page content can be clearly expressed in plain text there's a good chance a search engine will know what you're talking about.

      As an added bonus, if a web crawler can read your pages so can blind users.
  • From the link I just clicked on I saw:

    Ask Slashdot: Understanding Search Engines? 8 of 7 comments

    It took a second glance to notice the subtle error in the wording.
    Bug in slash?
  • Well hell, guess everybody can go home. Nothing more to search for here, it is all figured out.

  • Look at Open Source projects (Score:5, Informative)

    by X ( 1235 ) <x@xman.org> on Saturday February 04, 2006 @09:19PM (#14643842) Homepage Journal
    So, aside from reading books on Information Retrieval and Data Mining, the other easily available reference are open source search engines. In particular, look at the Nutch project [apache.org], which is actually a pretty high quality search engine implementation. Even better: start contributing to the project.

  • some info... (Score:3, Informative)

    by Pavel Stratil ( 950257 ) on Saturday February 04, 2006 @11:11PM (#14644157) Homepage Journal
    If you really want to go behind the theory, you will want to start here [psu.edu]. But be prepared to have some really good skills in math, statistics, computer sciences and system administration to understand the articles as they are not intended for general public.

    A brief intro of how classical search engines work goes as follows:
    Grabbing: A crawler visits pages which it considers important, downloads them and parses them
    Analysing: The document receives an identification string and is stored in a reversed index, which is simply a database table with culomns such as "word", "document", "possition", "importance". The "word" culomn is indexed and used for searching.
    Searching: Say that you search for the phrase "ask slashdot". The search engine searches the lines with the terms "ask" and "slashdot", looks into the "document" cell and selects only those documents that both terms occure in. Then it looks into the "possition" cell which carries all the possitions of the searched word in each document and discards all the documents that do not have successive "ask" and "slashdot" terms possitions. The resulting documents are then sorted according to the importance cells of the searched terms.

    This is how basically all search engine works. The only major difference is usually only in the math used to compute the imprtance. There are also some major optimisations done to speed up the responses. To discuss this would take too long. So if you have any questions feel free to ask. Currently I am part of a team developing a large scale search engine, so you have a chance to get some hot info here :)
  • There is a website [searchenginewatch.com] dedicated to search engines

  • I guess by now we can be fairly certain that search engines are here to stay...

    I'm not so sure I agree with you. What do people really search for nowadays (OK, other than "sex"/"porn")? I know where to go for my news, weather, sports, tech news, political discussion, coding tips, dining reviews, etc. The last time I *really* searched for something was a couple months back. I search within websites quite often, but I do not use large search engines like Google, Yahoo!, MSN, etc. more than a few times a year

    • I use google thoughtout the day every day for my job. Then again, I'm doing web development for a living...

      • Well yeah, in that case you would use it quite often :-)

        I was talking more about the average Joe looking for sports scores or news. I think most people know where to go for their information, and search engines are just a last resort when they absolutely can't find what they need on the sites they know. I think Ruppert Murdoc said something to that effect. Although I think Murdoc is wrong about nearly everything else, I have to agree with him on that small point.
  • I found Mining the Web [iitb.ac.in] useful. It's written by academics, so you'll have to put in a little brain work translating it into implementable patterns, but it gave me a good jump start when I took on a new client that does a lot of crawling and searching.
  • You might want to study "v7ndotcom elursrebmem" - the latest Search Engine competition. Just type it in your favorite search engine. If you enter it in Google, you'd even get strange ads.

    Some are joining to get prizes from the competition, like v7ndotcom elursrebmem: Blogging for Charity [yugatech.com]
  • You might want to check out the paper Web Search for a Planet: The Google Cluster Architecture [google.com]. It is 4-5 generations old, but provides some interesting information about Google's previous cluster setup.
  • There are a lot of papers on the underlying theories, but there is very little out there that will actually tell you how and when to implement them. Pagerank has really nothing to do with building a search engine, it is just one measurement that goes into determining relevancy. And it isn't applied the way most people think. I would say the best book on search engines is "mining the web" by Soumen Chakrabarti, but it doesn't really talk about implementations. But that's why information retrieval experts get

  • For print resources I would suggest:

    Understanding Search Engines [amazon.com] by Michael Berry and Murray Browne

    as well as

    Modern Information Retrieval [berkeley.edu] by Ricardo Baeza-Yates and Berthier Ribeiro-Neto

    For online resources I would of course direct you to the work of our Search Focus R&D Group [greenbuilt-research.com]

Slashdot Top Deals

We warn the reader in advance that the proof presented here depends on a clever but highly unmotivated trick. -- Howard Anton, "Elementary Linear Algebra"

Close