Are Web Pages Getting Larger? 67
An anonymous reader asks: "I work for a large multinational in a remote part of world. Our connectivity to the outside world (the Internet as well as company communications) is all done via a single E1 line - that's 2Mbps. Thousands of users. The company keeps access pretty well screwed down for security reasons, and the fact that our link to the outside world costs almost $300K/year! Our growing problem is Internet traffic. While policing of non-business use is very active, Internet traffic continues to grow. I'm becoming convinced that one of our problems is that average web page size is growing. As more of the world enjoys broadband access, I think web developers have less reason to limit the size of their web pages. Large images, flash animations and other size-increasing content seem increasingly common. Am I right? Can anyone point to a recent study that would support my theory, and help me convince my management that we just plain need more bandwidth?"
the answer is... (Score:4, Interesting)
I think the answer to your question lies within the technology itself, and the obvious answer is "yes", web pages are getting larger. Consider that:
So, yes, the web universe is "expanding" in very nearly every dimension. To your specific question, will you need to petition for more bandwidth? Undoubtedly. And, I can't imagine it isn't doable at today's rates. It sounds like a balky bureaucracy, not a question of need. Good luck.
I think maybe the better question to ask, is what has happened to the general psyche of the average employee, and how do you address it? If I had to guess (see, I'm not proving anything with this post!) I'd guess the technology has easily stepped up to the task of underpinning the network use but people still have not learned how to modulate and attenuate the siren that is the internet. (Maybe that would help decelerate your need to upgrade and expand bandwidth.)
Re:the answer is... (Score:2)
the number of web sites has grown exponentially
I think this is the biggest factor causing more and more bandwidth usage, and your best angle for convinciong management to buy more. It is true that extra computing power lets people get away with writing bigger web pages, but also the number of web sites is quickly growing, as is the number of people who use them and the number of things they use them for. I'm sure there are hundreds of studies and surveys that show this. It's only a small exageration to
Re:the answer is... (Score:2, Insightful)
Re:the answer is... (Score:1)
Re:the answer is... (Score:2)
If you're browsing Slashdot, for instance, you know that there aren't going to be any graphics on the page that you really need to see. It's all text, the only graphics are ads. So wasted bandwidth, essentially.
However if you were reading some howto article or news story, that referenced a graphic, then you could click on it and download.
It's not very hard; you just wait until you either need to see something because it's referenced in the text, or because the page doesn't make
Caching. (Score:5, Interesting)
Re:Caching. (Score:4, Interesting)
Re:Caching. (Score:2)
Re:Caching. (Score:1)
We set up a squid cache on an old workstation. We were pulling about 10GB/month with our 2mbs cable modem. Not a huge amount, but after installing the cache, and running our 75 users through it, we took it down to 3GB a month. Just part of being good Net citizens.
Re:Caching. (Score:2)
We set up a squid cache on an old workstation. We were pulling about 10GB/month with our 2mbs cable modem. Not a huge amount, but after installing the cache, and running our 75 users through it, we took it down to 3GB a month. Just part of being good Net citizens.
I would hope they already have a local proxy server if they are serving "thousands" of users with a 2 megabit link. It would probably be a small expense to set up a remote proxy server at a colo facility somewhere and point their local proxy t
How hard is it to set up? (Score:2)
There are a few places I know of that are really dying for web cache servers, because even without looking at the logs I can tell that they are probably spending GB a month downloading the same banner ads and Google splashscreen to every user on the network. It's a situation where there are hundreds of users and they're all basic
Yes (Score:1)
Or it could be the dreaded Skype sucking up all your bandwidth as it has done at my company.
Re:Yes (Score:2)
And then there is lazyness as well...
one word... (Score:3, Funny)
Re:one word... (Score:1)
Re:one word... (Score:2)
Often, but not always (Score:4, Informative)
An E1 costs $300k/yr? (Score:2)
But yes, I wouldn't be surprised if web pages are getting larger. On average, web pages are a lot more complex than in the past, with lots of links and sections and borders done with lots of little images. There are more ways to post images now, probably less care in compressing them, rather than compress them with care to not ruin the
Re:An E1 costs $300k/yr? (Score:3, Insightful)
A T1 out in, say, Montana in the US? NOT cheap.
It all depends where you are.
Re:An E1 costs $300k/yr? (Score:1)
Re:An E1 costs $300k/yr? (Score:1)
Not nice. ;-)
Re:An E1 costs $300k/yr? (Score:2)
Anyone else remember this?
E1 offers more bandwidth than a T1 (Score:1)
There's no need to guess when you can check. A quick gander at wikipedia [wikipedia.org] or google define [google.co.uk] says that an E1 offers 2.048 Mbit/s of bandwidth whereas a T1 offers 1.544 Mbit/s. However your cost point is taken since an E1 obviously isn't 50 times the speed T1...
Re:An E1 costs $300k/yr? (Score:3, Interesting)
Compressors, TCP (packet shaping) optimisers, proxy caches, DNS/email caching, webvertising blocks, QoS and agressive firewall rules are pretty much a given for any kind of expensive satellite connection. On the luser end, to really make use of the web they can set their browsers to not
Don't Need to Buy More Bandwidth (Score:3, Insightful)
You need to change policy, not spend more money. Change the cache settings on clients. Insert caching proxy servers. Make sure mail, DNS, etc. is local. Et cetera. You should find a solution that does not have a linear (at best) relationship with the number of users.
Web pages are getting larger. It might be what is causing your increase in utilization, but to me it's hard to believe (although if your users are viewing a lot of embedded videos, that's another story). And, if it's hard for me to believe, it's probably going to be hard for your PHB to believe, too.
Re:Don't Need to Buy More Bandwidth (Score:2)
But $300K/yr for 2Mbits? They should just get a satellite Internet connection and pay $100/mo for a server on terra firma at a data center somewhere.
I can't think of a scenario where you could justify that kind of cost. That's $12.50 per kilobit per month. A multiplex of V.34 modems would be about 30 times cheaper.
The report you are looking for should be called... (Score:5, Informative)
With the broadband market now including a minimum of 25% of home users, and up to maybe 40%, though I haven't looked at those numbers in some time, would be a contributing factor to the fact that yes, web pages are getting bigger.
One way to see proof of this is using the wayback machine.
http://www.waybackmachine.org/ [waybackmachine.org]
I took a quick sampling of the NYTimes homepage, and noticed that the number has increased by a few kilobytes per year, from 56K in 2001, to 67K in 2003, to 83K in 2005. That's not even counting images. They've added more ad banners since the old days. If you google search, I'm sure you will find stuff.
Ad banners have increased in size, and complexity over time. Streaming content, is another addition, as well as more services running over the network.
You probably have a number of contributing factors happening to your bandwidth, in addition to web pages.
- Unless you have an internal instant messenging environment, you may have many ppl chatting away on services having to use your bandwidth.
- Email for personal use. Jokes, funny attachments, and worms clogging up things.
Here are a couple of suggestions to try and improve traffic:
- block services that shouldn't be run at the office like streaming music content.
- block websites that you see can have an impact on traffic, that you believe users should not be visiting. ie: quicktime movies.
- block your daytrading slacker coworkers.
- block ad servers entirely! this should drastically improve your situation, and be the easiest to implement.
- switch to an internal instant messenging service, if you haven't done so already.
- disable unnecessary services.
- ensure that you have an internet policy that prohibits the users from using their work companies for personal use.
- cache often used content.
More content? Heh. (Score:2)
I've found some informative and interesting personal websites, including nuwen.net and a fair bit about amateur linguistics. Other than that, if you want more content, the best place to look is universities.
It's not more content, generally, just more media.
Re:The report you are looking for should be called (Score:3)
Re:The report you are looking for should be called (Score:1)
A quick way of doing this might be to block all (or most) Flash presentations. It is my experience that any site that heavily employs Flash displays tends to be very thin on content, despite the fact that they usually chew through a fair amount of bandwidth.
wow (Score:2, Insightful)
I mean wtf....are they getting bigger? you mean...you mean as more and more people attach databases? and...and...java? etc...
I mean really...how stupid is this
Re:wow (Score:1)
Look at the title 'Are Web Pages Getting Larger?'.
Pages.
As in (X)HTML + it's images, css/js files, etc.
Cell Phones/Blackberry Woes too! (Score:2)
It isn't just low-economy bandwidth ISP that is suffering this onslaught of dazzling eye-candies of HTML pages.
Blackberries and java-enabled cell phones also face the same problem. But, there is a way as other postings have shown -- web proxy cache.
One can defer uploading of images with a like-sized null (one-dot) image placeholder. These images, if the end-user desires, can be retrieved selectively. This is the BIGGEST bandwidth saver. Other is filtering of ad-contents.
Otherwise, it
trend seems to be up, but lately... (Score:5, Informative)
html size (doesn't include images/dependencies)
slashdot.org yahoo.com microsoft.com
1996 - 7k 11k
1997 - 9k -
1998 23k 10k 20k
1999 35k 10k 20k
2000 36k 12k 17k
2001 41k 16k 21k
2002 39k 17k 28k
2003 39k 32k 31k
2004 51k 33k 38k
Today 19k 14k 22k
the trend has certainly been up, but lately big sites' main pages seem to be slimming down, due to CSS as well as a tendency to store style and javascript in separate file
Re:trend seems to be up, but lately... (Score:2)
Re:trend seems to be up, but lately... (Score:2)
Since, then there has been a slight tick back up, but some lessons learned.
Re:trend seems to be up, but lately... (Score:2)
That being said, I wonder if there is any way to get a truly accurate idea of how much content is being transferred per page visit, after all client-side code is executed. Perhaps a fire
Re:trend seems to be up, but lately... (Score:2)
But that would give you the "true transfer" involved in loading a particular webpage.
hmmm (Score:3, Informative)
has some good results
http://www.sims.berkeley.edu/research/projects/ho
has info from 2000 and a link to the same info from 2003
specific internet 2000
http://www.sims.berkeley.edu/research/projects/ho
and 2003
http://www.sims.berkeley.edu/research/projects/ho
it's worth noting, these types of statistics can take a year or more to compile..
more bandwidth = more surfing (Score:2)
It is true that broadband has allowed pages to be bigger than before, but trends in UI design have also resulted in more text and fewer images, and with mod_gzip a mostly text/css site ends up being a lot lighter than its image/table predecessor.
I suspect that if you decreased the quality of the surfing experience by introducing about 1.5 seconds of latency into each web re
Designers don't care about optimizing. (Score:4, Informative)
From a web-designer standpoint, a lot of size can be reduced without altering the content.
Are you serving up nicely formatted HTML with indentations? That's wasteful. Strip whitespace and carriage returns.
Are you using HTML comments? Why? Does the customer really need to see them? Do you need to waste that bandwidth? Delete them or use comments in your server-side scripting language of choice.
Are you using GIF's where PNG's would be smaller? Or PNG's where GIF's would be smaller?
Have you optmized your PNGs [sourceforge.net], JPEGs [freshmeat.net] and GIFs? (I don't remember a GIF optimizer, but there are plenty of non-destructive ones).
A 50x50 JPEG preview of an item does not need embedded comments, thumbnails, or EXIF data.
If you must use animated GIF's, be sure they are optimized and not full-frame!
Are you using pictures of words, when actual stylized text could convey the same message?
Are you using inline JavaScript or CSS, rather than calling it from a cacheable external file?
Are you using PDF, Flash or Java when it's not ABSOLUTELY necessary?
From a user's standpoint, the best solution, short of getting more bandwith: use less bandwidth. Turn off image loading or use a text-based browser. Don't browse the web as much. If you have a choice of sites to use, use the one that is smallest. Use a proxy. blah blah.
Re:Designers don't care about optimizing. (Score:5, Funny)
Re:Designers don't care about optimizing. (Score:2)
Re:Designers don't care about optimizing. (Score:2)
When you are using gzip compression, such optimisation will add up to a grand total of three or four bytes saved per page. Is it really worth the effort?
You know, you could save a few bytes by using correct English instead of spraying extraneous apostrophes everywhere :).
Easy solution (Score:2)
Anyway, I'm just kidding... or am I?
Re:Easy solution (Score:1)
Use MRTG (Score:3, Informative)
Proxy has two uses (Score:1)
Dont ask for more bandwidth (Score:1, Flamebait)
So just asking for $$$ for bandwidth is failing them, in a way. If things arent working out AT ALL right now and you have to fix it, they might pay. But if youre making do with things, they wont slap down another $100K.
So try other ideas like giving them quotas. They'll learn to fit their browsing in the quotas. Cache the pages, try to disable some things like maybe
Re:Dont ask for more bandwidth (Score:1)
Ads (Score:1)
Proxy Adblocker - Outlook2003 Picture downloading (Score:1)
Invest in a proxy adblocker, such as http://www.privoxy.org/ [privoxy.org]. This is easy to justify to management, it really doesn't interfere with legitimate web usage in my experience.
Create a group policy for Outlook that disables the ability to change the automatic picture downloading in Outlook2003.
Adjust you user's Exchange delivery restrictions to something sensible. Does
Content-Encoding: gzip (Score:3, Interesting)
Ticking the box used to crash IIS but these days it actually works, not that you'd notice :
Response Headers - http://www.microsoft.com/ [microsoft.com]
Date: Thu, 08 Dec 2005 10:30:40 GMT
Content-Length: 23186
Content-Type: text/html; charset=utf-8
Cache-Control: private
Server: Microsoft-IIS/6.0
P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
200 OK
Response Headers - http://slashdot.org/ [slashdot.org]
Transfer-Encoding: chunked
Date: Thu, 08 Dec 2005 10:40:11 GMT
Content-Type: text/html; charset=iso-8859-1
Cache-Control: no-cache
Server: Apache/1.3.33 (Unix) mod_gzip/1.3.26.1a mod_perl/1.29
SLASH_LOG_DATA: mainpage
X-Powered-By: Slash 2.005000090
X-Fry: Where's Captain Bender? Off catastrophizing some other planet?
Pragma: no-cache
Vary: User-Agent,Accept-Encoding
Content-Encoding: gzip
200 OK
heh try this (Score:2)
Then:
Option A: Fascist
All outbound nonSSL web traffic (80-83, 8000-8999) goes through transparent proxy.
All outbound SSL webtraffic must be via nontransparent proxy.
All users get their own PCs and IP addresses.
All users don't share PCs where possible.
No other outbound or inbound traffic allowed to client networks.
DNS, mail via the corporate servers.
Then go through the web logs and get the top 10 users and the top 10 sites v
Extranet proxy (Score:2)
However, assuming that has all failed, you can still setup an extranet proxy. The advantage of this is that the machine in your extranet can be run at much lower monthly rates in an area of cheap bandwidth (such as the US), and then COMPRESS the data before it hits your own pipe.
Essentially, use the following steps:
1. Purchase a dedicated server with
Sure, but there are still solutions. (Score:2)
Or you could move to Sweden or south Korea
The Answer Guy says: (Score:1)
A few tips (Score:2)
2) Packet shaping to throttle or block certain protocols (Kill off P2P from your corporate network, for example)
3) Offsite proxying to compress text (HTML, JS, CSS, etc) to save bandwidth. Also might want to think about recompressing JPEG/GIF/etc offsite to save bandwidth at the sacrifice of quality
4) REQUIRE FLASHBLOCK ON ALL COMPUTERS. Seriously, the good thing about FlashBlock is that it doesn't prevent flash, only that flash will only load if you click on them. The benef
Web pages are software, (Score:2)
The other problem is that the value of software is inversely proportional to the quantity of its output. So in that respect, while Web pages are getting bigger they are simultaneously becoming less useful.