Powerful Linux ISP Router Distribution? 268
fibrewire writes "I'm building a Wireless ISP using commercial grade, low cost equipment. My main stumbling block is that I cannot find a decent open source ISP class routing distribution. Closest thing to even a decent tool is Ubiquiti's AIRControl — but even it doesn't play well with other network monitoring software. I've used Mikrotik's RouterOS for five years, but it just isn't built for what I need. I don't mind paying licensing fees, but $300K for a Cisco Universal Broadband Router is out of my budget. Has anyone seen any good open-source/cheap hardware/software systems that will scale to several thousand users?"
Re:Just use any Linux distro (Score:5, Informative)
Does it have to be Linux?
Why not try OpenBSD [openbsd.org] and its excellent BGP implementation OpenBGP [openbgp.org]! It powers some pretty hefty businesses and ISPs. [openbgp.org]
-
DD WRT (Score:1, Informative)
http://www.dd-wrt.com/site/index
It's Linux on low cost wireless routers.
Vyatta (Score:3, Informative)
http://www.vyatta.com/about/press_releases.php?id=75
try the beta v6
Re:Just use any Linux distro (Score:2, Informative)
pfSense (Score:2, Informative)
Big Sur Wireless (Score:3, Informative)
There's a small wireless ISP located in the Big Sur area of California that seems to have been up and running for a few years now. Maybe the OP wants to build a system like Big Sur Wireless [bigsurwireless.com]. Their web site includes a lot of details about their homebrew system.
Re:Hire someone who knows what they are doing. (Score:4, Informative)
I think you have a good point, but I don't necessarily agree. First, we don't know what market the submitter plans on operating in or who his clientele are. We don't know what his experience is, how much resources he has, or exactly what level of service he intends to offer. Like the guy who criticized the submitter for refusing to buy a $300k Cisco router, I think you committed a common mistake in thinking that IT is just a series of 1-size-fits-all solutions, and that if you going to use the "right" solution to each problem, you shouldn't bother.
The era of entrepreneurship and hacking things together isn't over, and it probably never will be. Our tools and hacks may become more advanced, but hopefully there will always be people trying out new techniques and business models, testing new start-up technology, and finding different ways of accomplishing the same goals. The answer isn't always to pay an expensive expert or to use established tech.
As for this:
You could get by with this in the late 90s, but when you're going to compete with cell phone companies, cable companies and standard POTS companies, you probably need to have a bit of a clue.
That's true, but neither my phone company nor my cable company provide wireless access where I live. Cell phone companies provide wireless, but it's pretty spotty and slow, and I live in NYC. There are plenty of areas in the US where no service is available except through dialup. Obviously these large companies aren't interested in competing in all markets, so if you come up with a business model and think you can make it work, then I say go for it.
just a thought (Score:2, Informative)
Re:Mutually exclusive (Score:5, Informative)
You can have low-cost commercial grade services run using off-the-shelf hardware.
pfSense [pfsense.org] includes support for CARP, which lets you build high-availablity failover clusters. You can have two (or three or four...) cheap systems and if one dies, just fix/replace it as needed. The backup system(s) automatically take over and nobody would likely even notice the changeover.
When it's cheap, that is much easier to consider.
If you want no moving parts, you can use an ALIX box, Soekris, or perhaps even some atom-based boards. If you want to use server-grade boxes to make yourself feel warm and fuzzy, you can do that too. Supermicro even has a server-class atom board in a 1U rack which runs pfSense very well for us.
Re:Screw Linux (Score:2, Informative)
pfSense 2.0 will solve the multi-wan traffic shaping limitation, and it's in beta right now. As for the multi-wan glitches, I'm not sure when the last time you tried it was, but the outbound load balancer was redone in 1.2.3 and 2.0 will have even more changes as well.
I run an ISP and we use a pfSense CARP cluster in front of our servers and it's worked great for us, but admittedly we are a small ISP. We also use it at more than a dozen customer sites. Everyone loves it.
OBSD or pfSense (Score:2, Informative)
My history is: started on OBSD (due to hardware support, ironically); played w/ FBSD; ended up on pfSense.
My observations:
OBSD is absurdly security conscious... for ISPs especially, this is a good thing.
OBSD tends to have a lot of focus on new network features (pf, carp)
most OBSD features get ported to FBSD... but take time (look into carpdev)
pfSense (built on FBSD) has some overhead vs FBSD raw (obviously), but has *nice* management UI, package support, etc ;)
customizations are easy for pfSense (I added some features to dhcpd a while back)... easier than generating the diff and submitting it
pfSense is more focused on network features (they're working on fixing carpdev for FBSD)
I like pfSense a lot... I use it for routing between ~6 VLANs, IPSec tunnels with another pfSense, PPTP server, *tight* firewall rules (given 6 VLANs).
pfSense 2 will be adding a lot of nice features for businesses (multiple admin accounts, different permission levels, etc)
Vyatta (Score:2, Informative)
What about Vyatta? It's a good router based on linux and you can install it on any old box you want or buy their hardware for it. Even has a cisco like interface if you want.
A suggestion (Score:2, Informative)
RuralLink Ltd (yes, I work for them) does what you want, linux-based wireless network management. Get in touch with us at http://www.rurallink.co.nz/contact-us [rurallink.co.nz]
There's not a lot of info about that side of things on the website, but if you contact us we'll be happy to chat - and don't worry, we're all techs, there's no sales droids here.
Cheers,
Scott.
Step in the right direction (Score:2, Informative)
I guess i'm looking for a scalable ISP-in-a-box solution. And if it doesn't exist, then let's build one. But Proxmox VE looks like it will fit well with managing computer resources between the handful of Dell 2950s slated for Zimbra, FreeIPA (Active Directory for Linux), Nagios, Cacti, and AIRControl. Still looking for a good FreeRADIUS server i can tie into FreeIPA - but i need lots of other stuff than just a router-in-a-box. A balance between smartest / practicality / economical directly translates into cost savings of the end user. Someday i will be able to provide free internet, but for now i am targeting $20-$40 a month for data, voice, video, and multicast TV. Some features of a good OSS router needing attention are:
* PowerPC vs X86 vs GPU - does routing perform better on PowerPC (Mikrotik / Vyatta / Cisco)? would an Nvidia Tesla solution work well?
* Easy to manage large scale routing implementations - speed of deployment, discovery of devices, failover, centrally monitored?
* Weatherproof - power outages, network hiccups, etc. nothing more irritating than going on-site to an AP to reboot in the middle of a storm
For more details about a specific area please ask.
Re:Go 2nd Hand Cisco (Score:2, Informative)
If you are just starting up, I'd suggest a couple of Cisco 3550 layer 3 switches with the IP Services image. They don't have all the features of the big routers, but they can handle a huge amount of traffic. I doubt you could build a linux router that would handle as much traffic for the same price as a 3550.
Re:Are you serious, or just killing time? (Score:2, Informative)
Thus, these guys are setting most of the major network standards, as well as implementing them.
Re:Are you serious, or just killing time? (Score:3, Informative)
Show me the Franken' Catalyst 2950/6500 Sup720 3BXL, Franken Cisco 12006, or Franken Juniper M7i/M320, and then I'll be impressed. Your desktop PC will not contain TCAM or other components required for a minimal level of forwarding performance needed by an ISP.
After all these years, a desktop PC still cannot perform the task of a simple 8 port switch, at nearly the same packet rates as the switch. The packet rates that can occur on an Ethernet network easily overwhelm the desktop PC's limited interrupt capacity and memory I/O bus bottlenecks.
For an Enterprise branch office edge a desktop router is fine. Because Enterprises only buy a limited amount of capacity from an ISP. Also, Enterprise branch offices have only clients, not servers, so they aren't really subject to a DoS (rejecting unwanted packets is half as expensive as fully forwarding normal packets).
Of course, Enterprise server farms never use a firewall at the edge on the path into the servers, unless the periodic unavailability due to DoS attack taking out the firewall is not an issue.
But for an ISP, if you are planning on being a serious ISP, your core business is providing a professional service. Use a well-designed solution, not something you've cobbled together from off-the-shelf parts. You get real value buying gear that performs forwarding in hardware
In the long run, one 24 hour outage or service degradation, can cost more than engineering the network properly, and using good managed pieces.
The fact of the matter is the FrankenPIX was based on the original PIX platform, and Enterprise firewall, that used to be just a PC with some fancy packaging and a proprietary flash card. That platform has been obsolete for many years, and is not suitable for an ISP, anyways.
In case you didn't know, Firewalls like the original PIX can't handle that much traffic, and they are easily DoSed into oblivion by a simple flood.
Anyways, decent gear for service providers these days offloads work to hardware. And runs on a real-time OS that can provide something closer to a service level guarantee than a commodity OS can.
In case you didn't know... Linux is not a real-time OS, and cannot provide timing guarantees a RTOS can.
Generic Linux running on commodity hardware cannot provide proper separation between control plane and forwarding plane.
For certain very important functions, a commodity PC simply can't match the performance of a dedicated ASIC.
You can talk BGP all you want, but you can't reliably forward 30,000 pps through a commodity PC, or push speeds higher than approximately 200megs, due to interrupt contention.
There is also the matter of reliability of the hardware...
Commodity desktop parts are not designed to run 24x7, and they fail frequently. Physical failure in routers is rarer, unless there are environmental issues, or the equipment is old.
Re:Mutually exclusive (Score:1, Informative)
An important point of note here: pfSense is a firewall, not a router. Yes, it has routing functionality, but it is designed to be a firewall and doesn't have support for the kinds of routing functionality that the original poster probably needs.
pfSense, however, rocks as a firewall.
Correct question? (Score:3, Informative)
Alright - I read your question, then a couple responses - but it isn't clear here that you're asking the question correctly. Humor me for a moment, then decide whether you asked the right question.
You have access to the web, with a hardware router behind the modem. That hardware router services both wireless and wired LANs, right?
You want to set up a router behind that router? You still won't be able to monitor traffic going through that hardware router. You need to put your *nix router between the modem and the hardware router, so that you become the gateway for all traffic going to and from the internet.
Of course, that is still not satisfactory if you wish to monitor traffic within the LAN. For that, you want to eliminate the hardware router entirely. Install the hardware to make your *nix router serve the WIFI and the wired LAN, and eliminate that hardware router entirely.
You can only monitor and control traffic that is being gated through your router, so you want it ALL to be routed through your box!
Re:Be more specific! (Score:2, Informative)
The Coachella Valley is the area - all of it. A large area.
A dozen to start but hundreds in the near future - i'm going to provide high bandwidth service for next to nothing. So the routing HAS to work for minimal bucks.
Re:Vyatta has licensing issues (Score:3, Informative)
Re:What on earth are you trying to actually do? (Score:2, Informative)
Tons of multicast video data will eat up 1/4 to half of my last mile bandwidth, followed by voip and data.
I'm trying to balance Access point range to around 1/2 mile without dropping bandwidth, so Ubiquiti AirMAX equipment seems to work in trial runs.
i don't want to drop below 100Mbit lan speeds, rates are fixed so if a customer can't connect they won't kill all the bandwidth for everyone else.
Client's actual throughput will be about 10Mbit down / 2Mbit up + about 45Mbit of Multicast video overhead - 100 clients will share about 50Mbit of bandwidth, if it scales out ok then 300 clients will share 50Mbit
Re:no DD WRT (Score:4, Informative)
Re:DD WRT (Score:3, Informative)
They cut off your network access because of a report of infringement? Are you in the US? Do you think you could mail me at danny@eff.org with more info? We're always interested in the details of these incidents.
Re:Exactly what i wanted to hear! (Score:3, Informative)
Sure, the 2800 and 3800 ISR series can take full tables easily. You can get a 3845 starting at $10k. NM-1T3/E3 module is about $6k. Both the 2800 and 3800 take DDR-266 ECC SDRAM (except the 2801); don't feel the need to pay Cisco's prices for commodity RAM if you really don't want to. The 3845 is recommended to handle up to 2 DS3's. According to people I've asked, you can push a 3845 to 100-150 Mb/s. You can go as low as a 2811 ($2k) and still take full tables, but only at fractional DS3 speeds. I would guesstimate a 2811 is good to 10-20 Mb/s, Cisco recommends it for 4xT1.
Also, consider that some ISP's will include equipment bundles with circuit orders if you haven't already explored that angle.
Re:Mutually exclusive (Score:3, Informative)
These guys:
http://www.applianceshop.eu/ [applianceshop.eu]
Sell embedded systems with monowall/pfsense preloaded.
Extremely easy to use and reliable.
I use a pfsense one at home, no idea how things would scale...