Researching the Slashdot Effect? 41
leonia asks: "At Columbia University, we are investigating methods and software to automatically deal with sudden, unpredictable load spikes ala the 'Slashdot effect'. We are looking for web sites that have been the 'victim' of this effect who would be willing to share anonymized web logs with us. If you know of a site that went under after it was mentioned on Slashdot or similar site or are the proud owner of such a site, we'd love to talk to you."
What? No link? (Score:5, Funny)
Hah! (Score:5, Funny)
Re:Hah! (Score:1)
Well, (Score:3, Interesting)
Re:Well, (Score:1)
Forget Ellen Feiss (Score:2)
Agenda (Score:5, Funny)
Step 4 (Score:2)
Re:Step 4 (Score:2)
just post a link to Columbia U next time! (Score:2, Informative)
naive Question? (Score:4, Interesting)
Re:naive Question? (Score:3, Insightful)
Re:naive Question? (Score:4, Insightful)
Re:naive Question? (Score:5, Informative)
Slashdot gets slashdotted regularly. They just know how to handle it.
Ever notice how sometimes, the front page doesn't have your login information? Or the post count doesn't get updated? You're being served a static page for one reason or another.
The simple fact of the matter is that most of the 'database driven' sites, which is where you get the 'mysql socket error' or 'too many sql server connections' or whatever problems, don't need to be database driven. You don't write a static article, put it in a database, then hit that database every bloody time somebody wants to read it. Put a copy in a variable, display that, refresh the variable every five minutes. Or even every minute. Or even ever ten seconds. You'll still never get more than six connections a minute, at that rate, and the information is just as current and fresh.
Re:naive Question? (Score:1)
Doesn't mean they couldn't cache those things ahead of time, but most sites don't think of that.
Luke Codehacker with a page on his home server about his nifty new software that ends up linked from Slashdot...ouch! Even if he isn't pulling from a database, watch the poor guy's DSL get obliterated!
That said, there are solutions to fix the Slashdotting problem. My own server hosts a site which was linked to from Slashdot about a month ago, and we saw the corresponding traffic spike. But it didn't kill the site or the link. Why?
I use all sorts of bandwidth-shaping tools and rules at both the kernel and Apache levels. Normally, the site in question is bandwidth-limited to 128kbit/sec of the overall link. When a whole huge spike of hits from the same referrer domain came in, the rules adjusted; it freed up another 128kbit/sec of the link for that site, and only put people from that referrer into that pool. As a result, normal site access wasn't impeded.
Now, if everything on the site was generated on-the-fly from a database, I probably would've had a lot more problems.
We can help (Score:5, Funny)
A simple way to research the Slashdot effect (Score:4, Interesting)
(2) Submit an 'Ask Slashdot' asking people to visit a link to the above servers
(3) Wait
(4) Study logs
Re:A simple way to research the Slashdot effect (Score:2, Funny)
(6) PROFIT!!!
Looking for people who have been slashdotted? (Score:5, Insightful)
sign up to be slashdotted! (Score:2, Funny)
1- the server is reduced to a blob of melted plasic and silicon, or
2- CIS revokes my network connection for life.
Now all I need is a suitable article to submit.
Drupal's Throttle (Score:4, Interesting)
It generates static files (similar to caches) when access is too high. You can check drupal's cvs [drupal.org] (drupal -> modules -> throttle) or go straight to it. [drupal.org]
Ps: Some links may contain whitespaces, cut, paste and edit...
Re:Drupal's Throttle (Score:3, Informative)
I have written another module, called "filecache" which does what you have described. That is, it simply caches pages to the filesystem. If Slashdot results in MySQL choking, the site will continue to function thanks to the static cache files, automatically resuming dynamic generation after the rush is past and MySQL is able to respond to queries again. The module and patches to Drupal can be viewed here [drupal.org]. I'm adding some more features to this module, but the core is done and seems to work quite nicely. (Additionally, with pages cached to the file-system I'm seeing a significant performance boost)
Drupal [drupal.org] is an excellent engine, very modular in design and easy to work with. More, it's backed by a team of friendly and helpful developers. I made the switch after performance issues with PHP-Nuke, and I've not looked back. And it just keeps getting better...
-Jeremy [kerneltrap.org]
Sept 11, 2001 (Score:5, Insightful)
That said, people have written about the same thing, researching the Slashdot effect on their own sites, and posted logs here on Slashdot. I'm too lazy right now, but you may want to search the archives for some info.
Re:Sept 11, 2001 (Score:2)
Re:Sept 11, 2001 (Score:1)
Petswarehouse.com (Score:1)
Google sued as PetsWarehouse Lawsuit Continues. [slashdot.org]
Remember, it is a bad idea to type the following:
while true; do wget -O
and do not email pet@Petswarehouse.com [mailto]
Two Stories down (Score:2, Funny)
Two? (Score:1)
I got one for ya (Score:1)
The site receiving the poinding (www.thefreeworld.net [thefreeworld.net]) is down so you could check with them after the dust settles and find out what it was like...
Simple procedure (Score:2)
2 Search for the string (Score:5, Informative)
3 If the post below contains a link, record where it leads
4 GOTO 1 for mor data points
Tor
Thank You, Thank You (Score:1)
Have you tried a...
Beowulf cluster?
*rimshot*
Google is your friend (Score:3, Informative)
Note: make sure you check Google's archive of the pages for those that aren't static.
Can you see it? (Score:3, Interesting)
Re: (Score:1)
some hardware/and or gameing sites sites (Score:1)
neowin.net
icrontic.com
over
hope this helps.
Easy data collection... (Score:1)
Slashdot effect Analyzed (Score:2)
This analysis is based on the data gathered after the story 'Donating Time To Goodwill Projects?' [slashdot.org] was published at the Slashdot.org site [slashdot.org] on 24th October 2002 around 13:30 GMT -05:00 (US East coast time). What I wanted to find out is: what is the slashdot effect really like, what are visitors like.