Indexing Dynamic Sites For Search Engines? 7
Moeses asks: "I am working on a Web site that uses the Altavista search engine software. The latest version of the site has moved most of the data from static pages to dynamic pages. This causes some issues to arise, but I've developed work arounds for most of them, such as generating pages with URL's that contain all the query string information to index the whole database and code to handle situations where a user searches for something that can't be displayed because of some state information specific to that users session, but there are still enough issues that I can't index all the states of the files that I need. Building a custom search engine for the database isn't within the budget of this project. What are you others doing to index and search your dynamic sites?"
Re:Building Dynamic Pages With Search Engines in M (Score:1)
Solution for local searching (Score:2)
If you have a more sophisticated search engine that can deal with item tagging (for metadata like keywords, creation dates, authorship, description, title, etc.), all the better. Create your text files with the appropriate tags and metadata pulled from your database and get that indexed too, and when displaying search results you can parse it back out of the text file or straight from the database if you want. Verity's engine is very nice for this.
go around! (Score:3)
the problem is of course the pages end up getting old. no problem, add a little "this is an archived version of this page, please click here [silicongod.com] for the newest version" message. rerun the script when necessary.
i did this and was able to submit all my dynamic pages to altavista. what i also did was add an additional little "prev | next" link at the bottom, so a spider could start at one page and follow links to the end. i went further and created a hallway page [silicongod.com] to submit to altavista.
also, the pages are flat so they tend to load faster than dynamic ones.
check out the page i submitted to AV [silicongod.com], and old archived page [silicongod.com] (contains the links prev|next links @ bottom, or the live homepage [silicongod.com]
NEWS: cloning, genome, privacy, surveillance, and more! [silicongod.com]
Here is my signature (Score:1)
Building Dynamic Pages With Search Engines in Mind (Score:2)
mod_rewrite ! (Score:5)
An example:
you have a script called news.php and an news index id (news.php?id=42 i.e.).
You could map that to
news_id42.html with
RewriteEngine on
RewriteRule
in your
Voila ! your dynamic content looks exactly like a static html page.
Anoter one is to fool searchengines that the script is an directory:
foobar.php/param1/param2/
Works perfectly fine
(don't remember which
regards,
Michael
Samba Information HQ
Constantly index in background. (Score:1)
dB@dblock.org [dblock.org]