Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
The Internet Upgrades

Load Balancing Heavy Websites on Current Tech? 63

Posted by Cliff from the more-hits-thru-heavier-iron dept.
squared99 asks: "I have just delved into some research on a set up for very high traffic websites. I'm particularly interested in how many webservers would be needed at minimum and the type of technology powering them. Slashdot seemed like a good sample site to check out, so I went to Slashdot's technology FAQ to get a starting point. This setup seems to be from 2000, is most likely a bit out of date, and I'm assuming the same number of webservers would not be needed with current server technology. What would experts in the Slashdot community recommend as a required setup to handle Slashdot-like volumes, if they had to do it today using more current hardware? How many webservers could it be reduced to, while maintaining enough redundancy to keep serving pages, even under the heaviest of loads?"
This discussion has been archived. No new comments can be posted.

Load Balancing Heavy Websites on Current Tech? More

Load Balancing Heavy Websites on Current Tech?

Comments Filter:
  • Anything fairly new should be alright, I think the big problem is your pipe size. I mean if you have 30,000 new connections and only 300 kbs, its not goign to transfer data very well.

  • One Word (Score:2, Informative)

    by cuntzilla ( 885978 )
    Akamai [akamai.com]

  • Prime Example: wikipedia (Score:5, Informative)

    by dyftm ( 880762 ) on Thursday May 26, 2005 @06:38PM (#12649680)
    http://meta.wikimedia.org/wiki/Wikimedia_servers [wikimedia.org] This is how they do it.

    • Re:Prime Example: wikipedia (Score:1, Insightful)

      by Anonymous Coward
      Since I can't reach the wikipedia server around 2 out of three times I wouldn't call this a successfull example
    • 72 servers and it still runs slower than any other website of its popularity.

      • Re:Prime Example: wikipedia (Score:4, Insightful)

        by FooAtWFU ( 699187 ) on Thursday May 26, 2005 @11:07PM (#12651414) Homepage
        Well, few sites of that popularity are quite as 'read-write'. When you have people submitting edits to articles every second, things get a little trickier.

  • Test, test, test... (Score:4, Funny)

    by PaulBu ( 473180 ) on Thursday May 26, 2005 @06:39PM (#12649689) Homepage
    ... and you've just missed your greatest opportunity for this by not providing a link to your website! ;-)

    Paul B.
    • That's cause I'm still scoping the architecture :) But I'll be sure to come back when its up. As has been said in other slashdot posts, I'll include references to Natalie Portman, SCO, and the RIAA. that ought to do it.
  • Many sites are moving towards utility based hosting or virtualized setups. The problem with high capacity sites is that you often end up having to purchase enough servers to deal with peak time, but don't need the servers during off hours. Utility based hosting services charge you for what you _use_ and allow you to scale as needed. Savvis (http://www.savvis.net/ [savvis.net] I know offers a utility hosting platform based on Inkra, 3Par and blade servers. IBM has a similar setup.

  • Take a look at livejournal's setup (Score:3, Informative)

    by Artega VH ( 739847 ) on Thursday May 26, 2005 @06:45PM (#12649721) Journal
    Akamai [akamai.com] for static content and take a look at livejournal's [danga.com] setup [zawodny.com] for dynamic content (master-master replication based on mysql).

    Other people are much more qualified than I to answer the number of servers questions though.
    • Please god, don't *ever* duplicate Livejournal's setup. It's a horrible, nasty hack and anyone who uses Livejournal will tell you that it doesn't work very well either. Although it's gotten better in the last year or so. But that's way, way, way more computing power than they should need to run that site. It's mostly a sign of a system that expanded without any real future-proof planning at all, which isn't really their fault, but if you have the opportunity to think it over and actually plan things, please

  • RTFB (Score:1, Informative)

    by Anonymous Coward
    RTF Server Load Balancing [oreilly.com] by Tony Bourke. After reading that book you will at least know what you need to look for. Also, you can outsource your load balancing if that is optimal for your needs using something like the Akamai's servers (Microsoft.com uses Akamai, Netcraft confirms [netcraft.com]).

  • Pound (Score:3, Interesting)

    by slashflood ( 697891 ) <<flow> <at> <howflow.com>> on Thursday May 26, 2005 @06:58PM (#12649829) Homepage Journal
    Take Pound [apsis.ch], a few web server machines, a database server and a NFS server (no Coda [cmu.edu], AFS [psc.edu] or GFS [redhat.com] needed in most cases) and you should be set. This is a setup that I installed for a high traffic website and it is very stable.
  • I would start off by examining Slashdot's updated technology guide [slashdot.org].

    From those, you will get an idea of the type and scope of technolgy the slash teams use to maintain one of the world's most popular sites.

    Granted, your team is not as stilled as the crack techs at /. central, but the specs on that page will get you pointed in the right direction.

  • I've worked on both the Windows and Linux development sides of a shop that receives about a million hits a day on each side. On both sides, the bottlenecks where always the database.

    Both sides used pretty much the same setup for webservers... 4 load balanced webservers with hyperthreading at around 3ghz (at the moment... always around 75% of the fastest processors out there to save money). These are sitting in datacenters with multiple 10gbps connections, and each has a hotswapable copy of the entire sys
    • This should be qualified with the database can be the bottleneck. Depending on how often the data is updated he may be able to cache a large amount of the data on the webservers. A .NET example would be caching a commonly hit data table or object collection even for just a minute or 2 can reduce a large amount of database traffic.
    • First of all, any person that talks about 'hits per day' isn't someone who works with high traffic websites. To folks like us, it's all about page views, uniques and sometimes impressions.

      we have 4 admins for the mysql servers.
      What the hell for?

      each has a hotswapable copy of the entire system running at another data center.
      with proper failover, this is pointless and wasteful.

      Cron static pages off the database when possible
      I couldn't agree more.

      I'm doing on average of 250k page views per day. My
      • the main reason for the redundant system is in the even Level3 loses connectivity. We provide billing services for 40,000 websites, and we can't ever have the system go down for more that 5 minutes. Our customer support department gets swamped in a hurry.

        The 4 database admins are for the 7 different clusters, and these systems do more than provide for webpages, they do spidering for site changes, password management, etc, etc.

        so I suppose this kind of a system is beyond the scope of the question asked.
      • >>First of all, any person that talks about 'hits per day' isn't someone who works with high traffic websites. To folks like us, it's all about page views, unique and sometimes impressions

        Sad part about your statement is, that due to lack of industry standardization people confuse all the terms.

        I myself prefer to use the terms that were published back in 2000 by a company called net genesis. written by Matt cutler . title of the work is called e-metrics - business metrics for the new economy ... her
  • We use Foundry ServerIrons. We have two of them set up in an active/standby configuration. We've got approximately 35 servers (5 services) load balanced between the SI's, and average traffic over 50Mb/s just to those services. The SI's are very robust, and I'm quite pleased we got them.
  • I run quizilla.com, a pseudo-entertainment site that does 60-70 million pages a month, at least 2/3rds being dynamic database backed.

    The site faq has the grity details [quizilla.com], but basically everything is running on 8 web servers with a cluster of 4 database servers. Mod_perl is used for the most highly trafficed pages, though some less used pages are still static CGIs.

    For the way I have it set up, this farm has reached it's limit with the web servers getting pegged pretty constantly during peak hours, and the d
    • And not just logic code either, but stuff like SQL makes a massive difference if your app is DB heavy.

      In some applications there may be a valid argument for denormalization to reduce load. In others, the SQL was cobbled together by someone without the adequate experience and it's pounding away at the database (one such occurrence is when using subselects with MySQL 4.1.x.. it can prove significantly faster to split out subselects and pass them into another DB call using regular code.. since MySQL does not

  • Your question cannot be answered (Score:4, Insightful)

    by Guspaz ( 556486 ) on Thursday May 26, 2005 @07:45PM (#12650166)
    It is impossible to answer your question unless you define "heavy" traffic.

    Some people might consider a hundred thousand pageviews per day to be heavy. Others might consider a million pageviews per day to be heavy.

    From experience a hundred thousand for a reasonable application can be handled on one server. A million would probably require 2 to 4.
    • Hmm... would your experience be MySQL-based? :P

      Carnage Blender serves about 1M pages a day off a single (dual 2.4GHz xeon; 4 GB ram) machine. Those are database-backed pages, with a lot more updates than most read-only-ish sites.

      Powered by postgresql. And a lot of tuning.
      • Hey man, I love Carnage Blender!

        I had to stop playing because it was too addictive, but you've got a cool thing there.
      • My experience was indeed MySQL based. But it was a very low-end server.

        When I was worrying about such things, it was cheaper to get 4 low-end servers than one high-end as you described. Of course it's a bitch to manage multiple servers instead of just one.

        And of course 100k pageviews per day (Which eventually grew to 300k on a different server) simply didn't justify anything more than a low-end server.
    • It is impossible to answer your question unless you define "heavy" traffic.

      Amen to that.

      Step one is to figure out what you mean by heavy traffic. Slashdot is probably at a couple million pageviews per day, and Alexa tells us that there are nearly 1500 sites bigger. A top-10 site will get circa 1000x what Slashdot gets.

      In step two, figure out what kind of traffic you're dealing with. Most of Slashdot's page views are probably just hits on the front page or current article by guests, so they can be heavil
      • Just want to add my support for this post. Until you've answered the first three questions, the solution can vary from a single box with an appropriately-sized pipe to a full N-tier load-balanced architecture. I was going to add something as well, but now I've forgotten it. :-P

  • Ultramonkey + LVS-Kiss + Mon (Score:3, Informative)

    by Plake ( 568139 ) <rlclark@gmail.com> on Thursday May 26, 2005 @07:48PM (#12650184) Homepage

    At my work we use Ultramonkey [ultramonkey.org] with LVS-kiss [linpro.no] and Mon [kernel.org].

    Our hardware infrastructure includes 2 load-balancers running in a failover system with 3 web servers in the backend (1.8ghz, 512ram, 40gig hdd, 100mbps network) systems. That hosts over 60 million page views a month, it also supports real-time failover. For monitoring there are tools out there that use MRTG/RRD for cluster statistics.

  • Obvious answer... (Score:3, Informative)

    by ebrandsberg ( 75344 ) on Thursday May 26, 2005 @07:57PM (#12650253)
    Check out http://www.netscaler.com/>. The companies behind the top 10 websites on the internet have, maybe you should too.

    Disclaimer, I work for Netscaler, but the customers we have gained should help in your decision.
  • It's somewhat dated but the FUD busting response to the Mindcraft fiasco has all the formulas on how to figure out what you hardware you need for your pipe. You only need to plug in current processor specs to see what you need. I could only find it in the archives: http://web.archive.org/web/20040409223206/http://c s.alfred.edu/~lansdoct/mstest.html [archive.org]
    • You only need to plug in current processor specs to see what you need.

      That's probably not true. The processor is only one component in a system, and it's often not the bottleneck. Also, since then there have been substantial changes in web servers, kernels, and all sorts of hardware that goes around the processor. Further, that page only talks about static content; it doesn't tell you anything about the dynamic content.

      Ordering hardware based on theoretical calculations from formulas in six-year-old arti
  • As previously mentioned, Pound is a wicked, lean load balancer/HA arbitrator that runs well on Linux.

    -psy
  • Outsource to geocities /ducks
  • So Slashdot uses an Arrowpoint Content Switch from circa 2000, but except with the name change to Cisco and some technology updates, the same basic lineup still makes Cisco's portfolio.

    Some examples here. [cisco.com] The examples are heavy on Corporate speak, but you were asking about a large Web/Content architecture, right?

  • This page [big-boards.com] shows the server specifications for some of the busiest message boards on the internet, along with what software they use. You'll see the configurations are quite disparate, from the 90+ servers serving the anime fans at Gaia Online [gaiaonline.com] on 100% open source software, to the sometimes single server hosting some of the other top 50 forums.
  • Hey, it's one method.
  • I've worked with the Cisco CSS, Foundry ServerIron and F5 solutions and I fell in love when I got the F5s. I haven't given the other two another chance since I switched.

    But before I switched, I got demos from all three players and put them in a head-to-head contest. I would suggest doing the same. In a lab setting, we couldn't hit the devices hard enough to pick a clear winner based on performance. When I looked at administration and features, the F5 pulled to the front.

    The GUI is clear and concise


  • One mistake that I see lots of people make is use a PC-based load-balancer. A hardware device (Foundry ServerIron, Nortel Alteon, Cisco CSM, etc) is well worth the money (especially if you get it on ebay).
  • AnandTech.com [anandtech.com] has had a few articles on their site setup, e.g.: There are a few on that site about database server performance, too...
  • Our web site [skyblog.com] serves about 3 millions pages a day on two gigabit links. The balancing rules are quite complex since the content is splitted on multiple servers and SANs. We use software load balancers: Zeus ZXTM [zeus.com] on Gentoo Linux. The nice thing about software load balancers is that you can easily replace the hardware if it fails. Having a spare PC is way cheaper than a spare load balancer. We are very pleased with ZXTM so far. Very reliable, fast, and very flexible. It uses a PHP-like scripting language to
  • The answer to this question depends entirely on how heavy each request you serve is. If you are just throwing together some PHP code that pulls information from a few databases, possibly updating some others, on every hit, it could require quite a large number of machines to handle the load. If you are clever, effectively making the results static pages, it may take very few systems.

    A good starting place is to just measure it by testing how long it takes to serve a page like what you are expecting to be

Slashdot Top Deals

Math is like love -- a simple idea but it can get complicated. -- R. Drabek

Close