x86 Commodity-Hardware Router? 102
neomage86 asks: "I recently had to set up a router for a small company, only five users at any given time, and the needed VPN capabilities are built in. So, instead of using a Cisco or other embedded router, I decided to just install Linux and IPTables on an old 200 MHz PII I had lying around. It's been working fine, and I'm thinking about doing something like this for a much larger network (3000+ users). Does anyone have suggestions on how much I will have to beef up the hardware to provide IP Masquerading for about 1000 users on a T3; provide network-layer filtering of the transmission; and route between 4-5 internal subnets?"
VPN (Score:3, Interesting)
Re:VPN (Score:4, Informative)
Re:VPN (Score:3, Informative)
Even with the higher end router/vpn embedded solutions there seems to be an appreciable slowdown in the other traffics response times and throughput when the VPN is being heavily used - and the hardware acceleration in these systems is liable to "better" than the C3 acceleration.
Does anyone know if the C3 can do h/war
Upgrade? Hell, you're already massively over-spec! (Score:5, Insightful)
No seriously, you're going to swamp your PCI bus if you're doing routing between internal subnets. Goodbye, LAN throughput. Not to mention what merry hell you'll play with the CPU with VPN and firewall rules.
Your solution is great for a small place, or even a large place in a dedicated niche (like only VPN and/or firewall, or monitoring/IDS.) I wouldn't do something that ambitious with PC hardware though.
Wrong Answer (Score:3, Informative)
T3 = ~50mbps
Wrong - you got the division wrong
PCI Bus 127 MBytes = ~1Gbit/sec
T3 = ~45 Mbits/sec
Are you telling me the fastest a PC bus can go is 15 MBits a second ??? I know of Intel class hardware that can keep 100 MByte going over a Gbit NIC. Lets not even go into shipping PCI-X busses and soon to be shipping PCI-Express busses that are significantly higher throughput than this.
Now that we have that problem solved, what you will run across with multiple 100Mbit netwo
Re:Upgrade? Hell, you're already massively over-sp (Score:3, Informative)
PCI-X: more bandwidth! (Score:1)
PCI-X is 64-bit, and with multiple cards, they'll probably be running at 100MHz. Vs. standard PCI at 32-bit, 33MHz, that's 6X the bandwidth, or about 90mbs, more than enough.
Just make sure you get one with enough 100MHz PCI-X slots for all your NICs. Many boards come with, say, 2 100MHz PCI-X, 2 66MHz PCI-X, then some standard slots. (Note that it's 2 slots per bus, and for more slots, the mobo will have multiple
Re:PCI-X: more bandwidth! (Score:1)
But PCI-X at 100MHz is still 6X the bandwidth of standard PCI.
Re:Upgrade? Hell, you're already massively over-sp (Score:2)
As long as he;s not trying to do VPN encryption on it, he probably will. Personal experience tells me a P100 (running FreeBSD, not Linux) can easily firewall a 100Mb network link for a few dozen users, so anything P2 class shouldn't have any trouble at all.
No seriously, you're going to swamp your PCI bus if you're doing routing between internal subnets. Goodbye, LAN throughput. Not to mention what merry hell you'll play with the CPU with VPN and firewall
Re:Upgrade? Hell, you're already massively over-sp (Score:1)
Re:What's good for the customer (Score:5, Insightful)
Hmm... Linux routers and firewall rules are well described on the web. Any "competent network engineer" as you describe him/her is likely able to read...
Buying a service, not a router (Score:3, Informative)
The number of exposures for Linux doesn't particularly bother me, for a box that's being actively maintained. For a generally non-service box you don't even need to be paranoically prompt abo
Re:Buying a service, not a router (Score:3, Insightful)
All software has bugs. All software, particularly that which runs on the edge of the network, must be maintained with patches. All hardware networking solutions of any reasonable complexity like a router or firewall run software. Therefore they too must be patched from time to time.
At least with a Debian box you could put a cron job that automatically apt-get's la
Re:Buying a service, not a router (Score:1)
But appliances have one big advantage in this respect - less. No hard drive, little RAM, well minimized software set. Much of this can be done with Linux as well, but a generic PC makes it all harder to do. (Even if an appliance could be r00ted, it probably doesn't have enough 'spare' resource to do anything useful with, especially without compromising its base function in a use
Re:Buying a service, not a router (Score:1)
Why can I have a machine as intricate as my internal combustion engine car that could likely go 30-40,000 miles without me doing any more than putting gasoline in it. I don't need a team of mechanics constantly following me around. Yet no one can devise a system that can deliver web pages upon request without a team of IT guys monitoring it 24/7.
What's wrong with the analogy?
Re:Buying a service, not a router (Score:2)
2) Your car *does* needs more men to maintain it than a computer system does. It's just designed to wear out equipment/ material at regular intervals so the maintenance can be done periodically instead of intermittently
3) No one is actively working to break-in/destroy your car from a remote location... and those that try to do it locally will succeed. Computers on the network have to be safe from break-in in all cases (but they too are usually vulnerable if thief is at
Re:What's good for the customer (Score:2, Insightful)
A customized linux router solution can be managed via console, ssh or SNMP by any competent network engineer.
Re:What's good for the customer (Score:1)
1. Cisco has plenty of documentation, online and otherwise.
2. No matter whether you run Linux, OpenBSD, or IOS on a Cisco box, if a vulnerability comes up, unless you're a fluent coder, you're not patching it until someone else fixes it. Cisco is generally very good about fixing critical problems.
3. Considering 10 year old Cisco equipment is still in use in many places, I don't think that you have to worry about purchasing "a whole new unit."
4. An army of people who know IOS
Go BSD rather then Linux..... (Score:5, Interesting)
Re:Go BSD rather then Linux..... (Score:2, Interesting)
now that ive hit refresh a few times and have read your comment i might as well add my own $0.02
openbsd with pf is, imho, 50x better (and easier to set up and manage rules for) than anything linux can offer.
Re:Go BSD rather then Linux..... (Score:1)
Re:Go BSD rather than Linux..... (Score:2, Funny)
You will get 0wned much, much faster with Linux than with OpenBSD.
Re:Go BSD rather than Linux..... (Score:1)
Re:Go BSD rather than Linux..... (Score:2)
Re:Go BSD rather then Linux..... (Score:1)
1000+ Users???? (Score:4, Insightful)
Do the math. If your homebrew system goes down, you will be burning the time of 1000+ people ($60,000) per hour. With those kind of numbers it doesn't pay to do it on the cheap. Get a redundant Cisco system with plenty of power backup.
Re:1000+ Users???? (Score:3, Insightful)
Anyways, I have been using netfilter/iptables for on my 30 user, >100mbs network, 6 active NIC's and I've never had a crash that I didn't cause!
Re:1000+ Users???? (Score:1)
Re:1000+ Users???? (Score:1, Informative)
Re:1000+ Users???? (Score:1)
Am I right?
-Peter
Re:1000+ Users???? (Score:2)
Re:1000+ Users???? (Score:1)
Re:1000+ Users???? (Score:2)
Or you could use something like this [linux-ha.org] to provide redundant Linux routers on cheap commodity hardware and spend the money saved on getting more backup power.
All things considered, spend a couple hundred. (Score:5, Informative)
An Athlon-64 or above would be ideal, simply because you'd be able to mount ludicrous amounts of memory on the box, which is pretty much all that could ever matter for a router/firewall app, as Linux can easilly support logging anything you want to a remote boxen.
Realistically though, I've routed 8 T1's at 80+% capacity in both directions among 650 laptops before, including 3 seperated subnets, all routed through one box.
The box was a Celeron (P2) 800Mhz we'd downclocked to 633Mhz (standard practice at my company, downclock everything for live events for stability) and it used around 10% of the CPU at peak once configured correctly.
By 'correctly' I mean having the T1's all coming in on a seperate PCI bus from the actual network cards for the subnets. Specifically, the built-in ethernet turned out to be on a seperate PCI bus from the actual PCI slots in the case. Configuring the box to take advantage of this dropped CPU load from 80+% to ~10%.
So... for a T3 fully loaded? I'd say get a 2.0Ghz machine just for breathing room, and give it at least 2GB of memory, as neither is that expensive and will leave plenty of breathing room for things like IPSec or other fancier options down the road without any problems.
Re:All things considered, spend a couple hundred. (Score:2)
Re:All things considered, spend a couple hundred. (Score:3, Interesting)
The P2 was a typo, and one I appologize for. P3 would be much more accurate, and overlooking the typo is inexcusable as I was simply typing quietly before I hit post, and didn't read the entire post from the beginning before hitting post.
As for the T1's, we didn't use any PCI T1 cards. We used an external 10/100/1000 switch with all 8 T1's plugged into it via normal T1
Re:All things considered, spend a couple hundred. (Score:2, Informative)
Re:All things considered, spend a couple hundred. (Score:2)
While you can do the direct copying from one care to another (I didn't know it could jump PCI buses, but it makes sense), that eliminates all of the packet process
Re:All things considered, spend a couple hundred. (Score:2, Interesting)
The 't1 to 10/100 converters' are just common T1 interface boxes that output ethernet instead of 24 voice/data jacks. Data-only T1 interfaces, essentially. Unfortunately, that was one aspect I had zero to do with, the site provided them and I haven't had a reason to use them since (we usually do satellite T1 links for remote sites, or use sDSL for medium-term fixed emplacements), so other than saying Netopia was branded all over the boxes, I can't help further than a Google searc
Re:All things considered, spend a couple hundred. (Score:1)
Second, that depends on what you define by professional. We can and have been called with travel-time notice only (as in, under 2 hours), and provided a 2Mbit link when a T1 went down.
Pretty? No.
Tidying up for another hour? Yup.
Did it work? Hell yes.
Was the client happy? Yes.
Does anything else matter to me, a field grunt that doesn't deal with marketing or any other aspect of the company except making the tech work on-site? No.
But thanks, I passed along the ty
Re:All things considered, spend a couple hundred. (Score:2)
There's no such thing as a 't1 to 10/100 converter'... you must be talking about something which bridges two ethernets over a t1 line.
Anyway, so you had all these 'converters' plus this linux router plugged into one ethernet switch? I hope you weren't relying on the linux router to enforce any kind of security among these remote networks. Assuming they
Re:All things considered, spend a couple hundred. (Score:1)
The T1's were all used for combined bandwidth, as the event organizers dropped their order for a fractional T3 and got eight T1's at the last moment. We had no say in that aspect. And each T1 was plugged directly from the box-with-a-ca
Re:All things considered, spend a couple hundred. (Score:1)
I have at least two, maybe three slot-1 motherboards from different manufacturers that allow you to change the FSB via the bios. they were around.
Re:All things considered, spend a couple hundred. (Score:2)
I used to overclock it to 475 but it was unstable.
no can do sorry (Score:2, Insightful)
Re:no can do sorry (Score:3, Interesting)
If its 100baseT, 4x12.5MB/s = 50MB/s is easily within the capabilities of a standard 32bit/33MHz PCI bus (100MB/s sustained), at least in terms of transfer rate. Make sure to use a card that has drivers which support polling (aka NAPI on linux).
Re:no can do sorry (Score:2)
That's just the max that is _theoretically possible. The PCI bus (32 bit) is capable of a (again) _theoretical 127MB. Would you stake your job on those numbers? I sure as hell wouldn't. I'd divide all numbers by 5 and you will see a more likely transfer rate plus have room to grow a little. The asker didn't say what type of business it is, but I'd bet at 3000 users a lot of those are transferring some big fi
Re:no can do sorry (Score:2)
4x100Mbit is 4x100MBit.. what in gods name does the number of users have to do with it? If you have 400Mbit/s, is that 400MBit/s "bigger" in some way because its generated by 3000 users instead of, eg, 1000 or 500 or even just 1? It isnt.
That's just the max that is _theoretically possible.
Look, if the machine has a quad fast ethernet card then the max that box will have to route is 4x100Mbit/s. No amount of u
Re:no can do sorry (Score:2)
true...sort of. if those 3000 users aren't doing much other than checking email and browsing the web. If they are doing some serious stuff; which they may not be who knows?; then chances are good that 3000 users means a heck of a lot of traffic. For example, I have an
Re:no can do sorry (Score:2)
There's no sort of about it, sorry.
then chances are good that 3000 users means a heck of a lot of traffic.
Again, no amount of users will be able to get more than 100Mb/s of data through any of those 4 or 5 interfaces. That's 12.5MB/s * 4 * 2 = 100MB/s - absolute worst case, which PCI can do. However, you're unlikely to get 100Mb/s of multi-stream traffic through a 100BaseT network to the box, never mind into this box itself. So that's 100MB/s of bus bandwidth is an absolute max.
I have
Re:no can do sorry (Score:3, Insightful)
You're just not grasping this concept very well, are you? Let me spell it out to you very slowly: the limiting rate here is his T3 connection! No matter what these 3000+ users are doing, they cannot generate more than 45Mbit/sec of traffic because that's the m
Re:no can do sorry (Score:2)
*Sigh* I grasp that concept very nicely, my point is being misunderstood. Not all traffic is going out the T3. Yes I'm aware that you will never go above the 45MB.
Re:no can do sorry (Score:2)
As one of the other posters pointed out, it's not bandwidth that's your problem, it's pps (packets per second). The limiting factor there is going to be how quickly your system can handle
It's do-able but... (Score:3, Informative)
I would suggest getting PIII's instead of PII's though, but check where bottle neck's may be PCI bus, CPU processing packets, NIC not doing so well... etc. Plus if one box is connected to multiple subnets, it can be dhcp and/or dns and/or wins for them (if you do DNS please use the forwarder's option to forward dns requests to an upstream DNS server if possible).
Hardware bottleneck (Score:1)
Go invest in a g
T3 only? (Score:3, Informative)
P4 2.xxGhz (assuming moderate VPN usage)
512MB-1GB RAM depending on how many simultaious connections you're working with. The more connections the more memory eaten up
Hard drives: minimal config.
Motherboard & NIC's: Depending on how much you're 10/100's saturate, you may want to get some 66Mhz 64bit PCI cards instead of regular 33/32's. Eg:
http://www.cisco.com/en/US/products/hw/vpnde
It all depends on how much simultanious traffic you're looking at. You can use the analogy that the PCI bus is a network switch's backplain. 66/64's can transmit a theoretical maximum of 4gbits/sec. so it should be enough for anything you throw at it. 33/32's maximum theoretical is 1gbits/sec. but in reality expect for much less.
Re:T3 only? (Score:1)
If you are looking for cheap 64bit PCI NICs go here [google.com]
Re:T3 only? (Score:2)
The other nice thing about using a floppy instead of a hard drive? Just write protect the floppy when you're done building the firewall. If someone ever "breaks into" your firewall, you
Nice idea, but the hardware won't cope (Score:2, Informative)
If you're inexperienced, try to get everything from one vendor so that getting it all working together is their problem, not yours.
You could do worse than a http://www.nortelnetworks.com/products/01/passport /lan/ [slashdot.org].
Apple Airport Base stations (Score:3, Informative)
The Airport Base Station (original) is a very good "take apart" to learn how to build your own router.They couldn't be more simplistic in design and implementation.
Re:Apple Airport Base stations (Score:2)
What are you people talking about? (Score:2, Informative)
I did a quick test on my home network to make sure. I easily got 97 Mbps using NFS to transfer (multiple simultaneous) files between 2 machines on 100 Mb ethernet. I think that is pretty conclusive evidence that the PCI bus will not be a limit even on a DS3(T3), which only goes 51 Mbps. One of these machines even has the video card on the pci bus.
Anecdotally, w
Re:What are you people talking about? (Score:2)
But even still, a sufficiently fast PC should be able to keep up with a Cisco switch. Optimized code can be 'brute forced' with higher class hardware, yes?
I may be wrong, but most of the answers here are conjecture. I'd love to do some real tests
Re:What are you people talking about? (Score:1)
Re:What are you people talking about? (Score:1)
I'm not doubting you, just really skeptical!
harryk
Re:What are you people talking about? (Score:1)
Original experiment is not scientific!! I am not legally responsible if someone dies because you cited my post that 97 Mbps is possible over the PCI bus.
It would probably have been a better idea to use netcat to dump packets to
Actual response
Using bargain basement NIC's, I can't remember manufacturer or model.
Netgear 8 port switch.
800 MB file
I copied
Re:What are you people talking about? (Score:1)
My real reasoning for questing is because I want to do video streaming on a local network, with the potential for as many as 4 video streams. Currently I'm not able to sustain that kind of bandwidth, but then again, I'm streaming UDP, which I figured would be more effective, seeing as TCP would cause skipping.
Was the three NIC's acting as one, like in line balancing, or where they seperate IP addresses as well?
Still interesting though.
Re:What are you people talking about? (Score:1)
The three NICs in one box are all on seperate networks. One directly to a cable modem, another directly to a WAP, the other to an ethernet. (I live with techie roommates, so we have lot's of boxen).
If all four of these streams are the same, you should definitely look into multicasting. It really sucks over the real internet (r
Don't use Linux for this (Score:5, Insightful)
Use a BSD system, with a real packet filter. FreeBSD gives you the choice of IPFW, IPF, or PF. OpenBSD gives you PF. NetBSD gives you IPF or PF. All of those have much larger / better features sets than IPChains / IPTables, and work a *lot* better in NAT/PAT/MASQ situations. These packet filters are also truly stateful (last time I checked IPTables, it wasn't truly stateful without a bunch of extra patches).
Linux makes an OK home firewall. But I wouldn't use it anywhere near a business.
We use FreeBSD 4.9 on Pentium 166 MHz systems with 128 MB RAM using IPFW to server secondary schools with just under 300 student computers. Haven't had any problems yet with network slowdowns or dropoffs or anything. These are on T1s in the remote schools, and 8 Mbit cable in town.
(I had problems keeping a similar box running Linux and IPTables working on my home wireless T1-equiv link.)
Re:Don't use Linux for this (Score:2)
What do you mean by 'truly stateful'? AFAIK iptables is stateful.
I've got a little diskless P120 that does just fine with DevilLinux on a 1.5Mbit cable connection at home. Even does VPN. Not the fastest for VPN, but I've never seen it not keep up with my non-VPN traffic.
Re:Don't use Linux for this (Score:3)
Linux makes an OK home firewall. But I wouldn't use it anywhere near a business.
I have to question this. Given the few arguments named, it's easy to do so. I built lots of routers with and without VPN (FreeS/WAN and recently Racoon), proxy services (for http, ftp, mail), firewalls doing NAT, VPN and anything else you can imagine. Customers read about a feature on Cisco routers/PIX and they want to have this suddenly. It often does not make much sense, but the customer is king. (I don't connect to T1s di
Re:Don't use Linux for this (Score:2)
Re:Don't use Linux for this (Score:1)
hmmm...this guy didn't have a problem:
Somewhere in 1997 I downloaded and configured the linuxrouter.org projects mini-distro (called LRP). It was based on Debian 'some vegtable' 2.0.36 kernel. It was put on a Zenith Z-select Workstation with a 486 DX with 12 Megs of Ram and two identical 3Com 3c509 ISA parallel taking cards 10Mbit only. From the time it went to production til its retirement it served solid. It has only one failure, the floppy drive. I can say it NEVER failed because of OS. It did act squi
etc (Score:2)
Dont bother (Score:4, Insightful)
Oh. And hire an experienced professional to install it (i don't dobut that you could manage it, though). I wouldn't trust a job of this size to someone who 'did it once at home and it worked'. The enterprise works much differently than your basement.
If you set it up and something goes wrong, you, my friend, are screwed.
Re:Dont bother (Score:1, Troll)
I'm already doing something similar (Score:3, Interesting)
Currently, I'm using Mikrotik RouterOS [mikrotik.com] as a core router. It's at a small ISP -- 400 or so high-speed customers, 3000 dialup customers (400-500 of which are connected during peak times). Standard routing stuff (30 or so internal static routes, big deal). Couple hundred firewall rules (some for stopping Windows worms from spreading, some for general network security, some to help keep the nastier spammers in check). And BGP, taking a full BGP feed from our upstream, plus a couple multihops from places like Cymru's bogons project. And it doubles as a PPTP server so I can securely work from home (in a gesture of supreme irony, I can't get Internet connectivity from the company I work at).
And some other stuff I can't think of right now.
All this is running in a 1U system I got from eRacks [eracks.com] (they make good cheap stuff), except for the hard drive, which I yanked and replaced with a 64MB IDE-flash drive from these guys [wisp-router.com]. Celeron 1.3GHz, 512MB RAM. The system never ever, even during peak times, goes over 10% CPU load.
This isn't quite up to the specs the original author was looking for, mainly because this hardware isn't also doing the T1 stuff. (It's got plain old boring Ethernet to an older Cisco router, to which our four T1s are connected, but the Cisco is basically just a really big media converter.) But given how low the hardware utilization is on this unit, and how underpowered this system is as compared to current hardware, I think it shows that the notion is quite feasible.
Folks are doing this commercially (Score:2)
Also on the thread of interface cards, try Mikrotik [mikrotik.com]. If you're doing wireless, the MiniPCI carrier boards will make your day.
Full disclosure: I'm not related to or affiliated with either of those companies in any way. I've never even bought anything from either of them. I just came across
No Problem (Score:1)
And with bridging you can have two transparent firewalls (no ips) that are redundant, using Spanning Tree Protocol. Pretty cool.
more specs (Score:1)
Re:more specs (Score:1)
And good luck to you.
pc Hardware? (Score:1)