Powerful Linux ISP Router Distribution? 268
fibrewire writes "I'm building a Wireless ISP using commercial grade, low cost equipment. My main stumbling block is that I cannot find a decent open source ISP class routing distribution. Closest thing to even a decent tool is Ubiquiti's AIRControl — but even it doesn't play well with other network monitoring software. I've used Mikrotik's RouterOS for five years, but it just isn't built for what I need. I don't mind paying licensing fees, but $300K for a Cisco Universal Broadband Router is out of my budget. Has anyone seen any good open-source/cheap hardware/software systems that will scale to several thousand users?"
Erm... Requirements? (Score:3, Insightful)
Are you serious, or just killing time? (Score:4, Insightful)
So Cisco makes billions of dollars a year selling some ungodly expensive, ungodly powerful head end router like devices (not even routers in the IP sense) and somehow you suspect a Linux distribution with the same features is going to unpack itself and be everything you want it to be? You need to tell us what the rest of your platform looks like if you expect any answers that go beyond 'any linux distribution can act like a router!'. What subscriber equipment is in use? How much user control do you need (access on/off vs. bandwidth filtering, etc.) Details, details, details.
Mutually exclusive (Score:2, Insightful)
I'm building a Wireless ISP using commercial grade, low cost equipment.
To me, some words in this sentence seem to be mutually exclusive.
To my humble opinion, a good ISP needs to have good reliable equipement. Sometimes, out of the box routers are better because they don't have moving parts and their firmware could be more stable than a full-blown OS (even if it is Linux).
Disclaimer: Not that I don't like Linux, I use it all the time.
Hire someone who knows what they are doing. (Score:1, Insightful)
Sorry to be blunt, but you're asking the wrong question.
The proper question is: How do I find someone qualified to do this for me?
The fact that you are asking on slashdot shows that you are not qualified, and what you're going to get back is a bunch of others, who aren't qualified, suggesting all sorts of half assed hacks to do it which will just result in a utterly shitty service overall.
You could get by with this in the late 90s, but when you're going to compete with cell phone companies, cable companies and standard POTS companies, you probably need to have a bit of a clue.
Few requirements given but... Vyatta? (Score:3, Insightful)
Maybe Vyatta @ http://www.vyatta.org/ [vyatta.org] does what you want. I really don't have any idea what that is from the actual post, tho. You need some routing for thousands of users, and can't afford a Cisco UBR. I'm not sure exactly if you wanted to use the UBR for DOCSIS type support for some reason (a la cable modem) but the fact it'll be wireless leads me to believe it won't be. I'm assuming you don't need a lot of physical ports, just something to manage your VLANS, some routed subnets, a bit of BGP, etc. Maybe XORP is what you want, tho @ http://www.xorp.org/ [xorp.org] so you may want to look there. IHeck, 'm not even sure if you want to take a server with a bunch of PCIe ports and slam multiport switchable fabric cards in there like the ones DSS @ http://www.dssnetworks.com/v3/gigabit_pcie_6468.asp [dssnetworks.com] makes, or do something else. Maybe these links will help, and hopefully there'll be a detailed followup so we can aim at the real target :)
What on earth are you trying to actually do? (Score:5, Insightful)
Routing and ISP's are huge topics- what are you trying to do?
The main problem with routing isn't bandwidth- anyone can pump enough 1500 or 9000 byte frames per second to fill a gigabit pipe. The problem is when you have lots of small packets. At that point, dedicated routing hardware with a high-speed TCAM becomes really important.
What kind of line cards do you need? ADSL? Ethernet? OC12?
What kind of services do you need to run? BGP? OSPF?
What kind of bandwidth are you going to be pushing?
Ebay is your friend. (Score:5, Insightful)
Re:Hire someone who knows what they are doing. (Score:5, Insightful)
The fact that you are asking on slashdot shows that you are not qualified, and what you're going to get back is a bunch of others, who aren't qualified, suggesting all sorts of half assed hacks to do it which will just result in a utterly shitty service overall.
I disagree. The Open Source community has a thousand hidden gems that a person might not have heard about. Proxmox VE for one: virtualization, with a GUI, with live migration, and if 2.0 turns out, with heartbeat and failover (high availability). Most people have never heard of this where I work even though half the place is virtualized with KVM, VMWare, Hyper-V, etc. I would think the Slashdot, with its plethora of experiences, might come up with a little-known or workable solution in an already developed product that you haven't heard of yet.
As others have said... (Score:4, Insightful)
Re:Are you serious, or just killing time? (Score:5, Insightful)
And beyond that, just because a Linux box might support all of the protocols and implementations that Cisco has leveraged in their own products, it does not mean that the Linux box is going to configure itself. A lot of the reason that Cisco makes money is because they provide solutions. The solutions themselves leverage established technologies in many cases (RFCs are in the public domain), but Cisco makes them work together. It's the old discussion about Open Source vendors. They aren't making money selling people Linux because Linux is free. They are making money selling people Linux configured to perform specific tasks, and then selling support to keep the solution functioning and up to date.
Re:Are you serious, or just killing time? (Score:3, Insightful)
The "same features"? You mean like ASICs that forward the data with low latency once the route is established? Yep, Linux is going to somehow magically add those to your computer, and that's one of the reasons people pay the extra money for Cisco over some old P3 tower PC and a CD-ROM with a penguin on it. Another is that they fit nicely in a rack.
The submitter apparently has his own unique idea of what "ISP class" means. Admittedly, this is for a wireless network, so there is already a bit of latency expected and maybe not as much total bandwidth as a wired ISP, but you can never remove latency, only add less. And as you have pointed out, "ISP class" should include things like metrics and controls for users.
Re:Hire someone who knows what they are doing. (Score:3, Insightful)
The problem is, if you ask a Cisco person to do it, you'll get a Cisco solution, even if it isn't the best solution for the task.
Re:Hire someone who knows what they are doing. (Score:2, Insightful)
Re:Mutually exclusive (Score:3, Insightful)
To my humble opinion, a good ISP needs to have good reliable equipement. Sometimes, out of the box routers are better because they don't have moving parts and their firmware could be more stable than a full-blown OS (even if it is Linux).
If not for this reason, why do you suppose the question got asked?
Re:Are you serious, or just killing time? (Score:5, Insightful)
Sheesh. I wish someone would tell that to our clients. My company provides service to (mostly) small businesses, and half of these little five-man operations have some totally over-engineered Cisco gear acting as their network edge because some smartass, self-styled "IT Guy" told them it was the best. Surprise, he vanishes after plugging it in and collecting his fee, and now the client has all these problems with our SIP service and of course they have no idea how to manage their own equipment, and WE end up looking like jerks because our stuff won't work out of the box with whatever equipment the client has.
Could you do the 3 desk operation with a Linux machine and 4 network cards? Sure. In this example, it's cheaper to pick up a cheap hub, than to take even a salvage machine and put 4 network cards in it.
Here, though, I disagree. At the same company I mentioned, when I joined, we were a three-person operation, and we used a Linux machine with two network cards and a switch as our router. It worked great as we scaled up in staff numbers, particularly when tools like ntop and tcpdump existed to let me see when some joker was ruining it for everyone by torrenting the entire internet. If you never plan to expand, then sure, some cheap little router toy from Dlink or Linksys will do fine, but if you intend to grow, may as well do things right the first time than have to re-engineer your network down the road.
Also, a hub? Who the hell uses hubs anymore? I can't even think of a use for them these days other than packet sniffing, and an inexpensive managed switch will let you do that.
Re:Hire someone who knows what they are doing. (Score:2, Insightful)
Have you worked at a cell phone, cable, or standard POTS company lately?
What exactly do you think you are going to get there besides a bunch of unqualified, "half assed hacks to do it which will just result in a utterly shitty service overall."
Besides, many of the folks posting in this thread are probably those same unqualified, half-assed hacks who work at such companies. Corporations don't have any corporate voodoo that makes them special any more than someone working for the government makes them any smarter or able to perform miracles (free healthcare, news cars, and money for everyone, YAY!) Put down the kool-aid and open your eyes.
Re:DD WRT (Score:5, Insightful)
http://www.dd-wrt.com/site/index [dd-wrt.com]
It's Linux on low cost wireless routers.
Yeah, that's just what I'd want my ISP to run as a core router.
Re:Are you serious, or just killing time? (Score:2, Insightful)
Another is that they fit nicely in a rack.
And they provide a packaged solution, that most network engineers recognize and know how to manage, troubleshoot... meaning it will be easier to find/hire people to help manage it, than some custom home-brewed solution?
Lower long-term operational expenses, hardware is darn proven (fewer operational risks than you have buying commodity desktop parts), and you can get a support contract, usually (or opt to save money upfront by finding equipment and replacement parts in the aftermarket).
Many of the low-end routers 26xx are pure software switching. But they can still perform better than Linux, because the OS is designed solely for that purpose, which means performance optimizations too.
Linux is more of a jack of all trades. Forwarding performance and operation as a network device isn't a central design goal in the linux Kernel.
Re:Are you serious, or just killing time? (Score:3, Insightful)
FWIW:
PCI / PCIe x1 are both ~1Gbps max throughput (not counting overhead, that's raw bus speed). All the other PCIe's scale linerly, thus a PCIe x4 is 4Gbps bus speed.
After communications protocol over the bus that speed drops (not sure how much). There are other factors as well but what it all comes down to is PCI or PCIe can really handle only about 500Mbps per link.
-nB
Not quite. One lane of PCIe v1 is 250M/s, double that of PCI. One lane of PCIe v2 is 500M/s, double that of PCIe v1. So, a PCIe v2 4x slot would be able to push around 2G/s, or 16 Gbit/s, which is slightly more than the 500 Mbit/s you state.
Furthermore, given that built in gigabit ethernet ports on any motherboard built in the last 5 years or so are connected via PCIe, and I've never had an issue saturating the whole gigabit, it doesn't make a lot of sense to say that PCIe is limited to half a gigabit.
Re:Mutually exclusive (Score:3, Insightful)
"To my humble opinion, a good ISP needs to have good reliable equipement."
To my humble opinion, a good ISP needs to have good reliable *service*.
Ask i.e. Google to learn the difference.
FreeBSD, BGP, OSPF, pf (Score:3, Insightful)
Seriously, learn to love FreeBSD.
I am assuming that you will be doing a tree style network with a central location providing you bandwidth on a fiber link or T1/T3 etc.
Get a PAIR(at least, add more as necessary) of nice, quad core Dell Poweredge or HP DL series servers. FreeBSD+CARP them giving you as seamless load balancing/fail over as you can realistically get.
at each hub consider either buying commercial wireless routers or build your own. If you build just keep everything fanless as that is where your equipment will fail you.
Use OSPF on branches while being aware of scaling issues and where OSPF isnt ideal, kick in the BGP and you can link your OSPF clusters together giving an extra level on branch redundancy because traffic can hop to another branch if necessary.
OLSR in mesh cells, OSPF on the cells backhaul router linking these cells and providing multiple route options for redundancy, and BGP between groups of cells and between you and other ISPs etc etc.
You dont need to take the Mesh down to the client, only to the neighborhood AP level. The idea of mesh per client creates too many hopps and clients have too much latency. Ideally, you are no more that a 2-4 hops from the backbone, any more and you are going to be adding too much latency from the hops. When a backhaul link goes down and the OSPF saves your butt by routing traffic through a neighboring cell, you are already going to add latency and you dont need that complicated by 6 hops in the neighborhood and 5 more to the backbone (11 hops over wireless is just too many for broadband).
Re:Are you serious, or just killing time? (Score:4, Insightful)
The "same features"? You mean like ASICs that forward the data with low latency once the route is established? Yep, Linux is going to somehow magically add those to your computer, and that's one of the reasons people pay the extra money for Cisco over some old P3 tower PC and a CD-ROM with a penguin on it. Another is that they fit nicely in a rack.
A lot of router equipment is essentially an x86 PC. Add on cards are often just PCI or PCIe cards. You'd be surprised how commodity a lot of that equipment is. At least, for a big part of the mid range stuff.
Granted it's all specially chosen hardware and custom firmwared, plus Cisco IOS is a heavily developed and mature OS specifically written for routing, so you're not going to see anywhere near the same performance with some random Linux whitebox system.
Re:Are you serious, or just killing time? (Score:3, Insightful)
ASICs offload processing power from the cpu as dedicated hardware. But the PC you are running linux on likely has a CPU that is at least 10x as fast as that in the CISCO routers (and use a lot more power) so they can keep up.
This is one of those things that is as much about marketing as reality. There are no shortage of hardware appliance network boxes like BIG-IP LTM/GTM and Bluecoat ProxySG's that cost tens of thousands of dollars and are nothing more than BSD/Linux rack mounted PC's in a fancy case. These devices have no trouble handling enterprise loads (which is about the only place $50k+ pieces of equipment will be found in the racks).
The linux box does consume much more power to accomplish the task than the cisco with its ASICs but the raw power is definitely there. A more significant concern than the processing is the bus speed. I doubt that is a problem if he is concerned about an $800 software license (mentioned in a suggestion earlier) in that kind of budget range he isn't going to have links that could tax the bus.