Powerful Linux ISP Router Distribution? 268
fibrewire writes "I'm building a Wireless ISP using commercial grade, low cost equipment. My main stumbling block is that I cannot find a decent open source ISP class routing distribution. Closest thing to even a decent tool is Ubiquiti's AIRControl — but even it doesn't play well with other network monitoring software. I've used Mikrotik's RouterOS for five years, but it just isn't built for what I need. I don't mind paying licensing fees, but $300K for a Cisco Universal Broadband Router is out of my budget. Has anyone seen any good open-source/cheap hardware/software systems that will scale to several thousand users?"
Just use any Linux distro (Score:5, Interesting)
Just pick up your favorite Linux distribution and get back to me with your requirements. I think Linux can easily do what you need almost out of the box. It is only a matter of configuring it. I bet some would recommend looking at OpenBSD or FreeBSD as well.
Either way, you would definitely have a more flexible solution that any canned product will provide you with.
Re:Just use any Linux distro (Score:5, Informative)
Does it have to be Linux?
Why not try OpenBSD [openbsd.org] and its excellent BGP implementation OpenBGP [openbgp.org]! It powers some pretty hefty businesses and ISPs. [openbgp.org]
-
Re: (Score:3, Funny)
The title in the question was asking for a Linux distro.
Anyway, you have proven me right, if you read my OP very carefully, it states:
> I bet some would recommend looking at OpenBSD or FreeBSD as well. ;-))
Re: (Score:2, Informative)
Re: (Score:2)
I agree, OpenBSD seems to have bottomless performance in my installations and the configuration is so easy.
Re: (Score:2)
I have all my wireless IP addresses on a different subnet. The wireless router connects directly into the Linux router with its own interface. Thus it is easy to setup firewall rules specific to the wireless network and to monitor it for bandwidth usage and what not. Then connect your linux router to the ISP link.
To provide even more monitoring and trafic control capabilities from the Linux router, I do not use the DHCP server in the router but instead, wireless machines query the DHCP server on the Linux s
Correct question? (Score:3, Informative)
Alright - I read your question, then a couple responses - but it isn't clear here that you're asking the question correctly. Humor me for a moment, then decide whether you asked the right question.
You have access to the web, with a hardware router behind the modem. That hardware router services both wireless and wired LANs, right?
You want to set up a router behind that router? You still won't be able to monitor traffic going through that hardware router. You need to put your *nix router between the mode
Re: (Score:2)
Definitely PFSense. I prefer the traffic shaping in Linux (can't speak for the traffic shaping in BSD), but PFSense is sufficient in that regard, and excels at everything else. You can't beat the interface for visual presentation and ease of management.
If it absolutely has to be linux though, I love Tomato. It's mostly aimed at less-powerful hardware though, so I'm not sure how much you could scale it up.
Re: (Score:2)
Re: (Score:2)
Tomato is great, really.
On 40$-plastic-boxes.
But if you read the story, he is looking for a commercial grade system for an ISP. Then you have easily 1000 times the throughput of a little Broadcom box.
And its limitation of 2 physical interfaces doesn't help neither.
Re: (Score:3, Funny)
blah, blah, that was just a typo.
Go fuck yourself !
Cheers,
Vyatta (Score:3, Informative)
http://www.vyatta.com/about/press_releases.php?id=75
try the beta v6
Erm... Requirements? (Score:3, Insightful)
Re: (Score:3, Informative)
Screw Linux (Score:2)
Re: (Score:2)
Seconded and my idea of fun is running 50 odd Gentoo based systems around the UK. I probably wont try and screw them though.
For me the multi link routing ie load balancing/failover gateways is the key feature (I have 6 ADSL lines - my office is a bit rural). Add to that a good list of add ons, eg ntop, OpenVPN and IPSEC, WiFi with mesh and captive portal etc etc etc and its a bit of a winner.
Re: (Score:2)
PFsense has been OK for me in a small business environment, but it's nowhere near robust enough for ISP duty. For one, the multiwan implementation has been somewhat troublesome (mostly working, but occasional glitches) and traffic shaping doesn't work at all with multiwan. If you can do your multiwan stuff with an appliance, then perhaps that's not an issue, but my assumption was that you wanted something to act as your "core" using commodity hardware.
Best,
Re: (Score:2, Informative)
pfSense 2.0 will solve the multi-wan traffic shaping limitation, and it's in beta right now. As for the multi-wan glitches, I'm not sure when the last time you tried it was, but the outbound load balancer was redone in 1.2.3 and 2.0 will have even more changes as well.
I run an ISP and we use a pfSense CARP cluster in front of our servers and it's worked great for us, but admittedly we are a small ISP. We also use it at more than a dozen customer sites. Everyone loves it.
Re: (Score:2)
I've been using 1.2.3 since it was released and while some of the problems have gotten better it still isn't nearly stable enough to be a core component of an ISP's mix of gear. Also, 2.0 has been in alpha for ages and only JUST went to beta and has a prominent warning in their support forums about not trusting it for production use.
I've heard things are MUCH better when using it as an inbound loadbalancer, but the outbound stuff is troublesome and doesn't scale well (at least for me).
That said, for a SOHO
Are you serious, or just killing time? (Score:4, Insightful)
So Cisco makes billions of dollars a year selling some ungodly expensive, ungodly powerful head end router like devices (not even routers in the IP sense) and somehow you suspect a Linux distribution with the same features is going to unpack itself and be everything you want it to be? You need to tell us what the rest of your platform looks like if you expect any answers that go beyond 'any linux distribution can act like a router!'. What subscriber equipment is in use? How much user control do you need (access on/off vs. bandwidth filtering, etc.) Details, details, details.
Re: (Score:3, Insightful)
The "same features"? You mean like ASICs that forward the data with low latency once the route is established? Yep, Linux is going to somehow magically add those to your computer, and that's one of the reasons people pay the extra money for Cisco over some old P3 tower PC and a CD-ROM with a penguin on it. Another is that they fit nicely in a rack.
The submitter apparently has his own unique idea of what "ISP class" means. Admittedly, this is for a wireless network, so there is already a bit of latency expe
Re: (Score:2, Insightful)
Another is that they fit nicely in a rack.
And they provide a packaged solution, that most network engineers recognize and know how to manage, troubleshoot... meaning it will be easier to find/hire people to help manage it, than some custom home-brewed solution?
Lower long-term operational expenses, hardware is darn proven (fewer operational risks than you have buying commodity desktop parts), and you can get a support contract, usually (or opt to save money upfront by finding equipment and replaceme
Re:Are you serious, or just killing time? (Score:4, Insightful)
The "same features"? You mean like ASICs that forward the data with low latency once the route is established? Yep, Linux is going to somehow magically add those to your computer, and that's one of the reasons people pay the extra money for Cisco over some old P3 tower PC and a CD-ROM with a penguin on it. Another is that they fit nicely in a rack.
A lot of router equipment is essentially an x86 PC. Add on cards are often just PCI or PCIe cards. You'd be surprised how commodity a lot of that equipment is. At least, for a big part of the mid range stuff.
Granted it's all specially chosen hardware and custom firmwared, plus Cisco IOS is a heavily developed and mature OS specifically written for routing, so you're not going to see anywhere near the same performance with some random Linux whitebox system.
Re: (Score:3, Insightful)
ASICs offload processing power from the cpu as dedicated hardware. But the PC you are running linux on likely has a CPU that is at least 10x as fast as that in the CISCO routers (and use a lot more power) so they can keep up.
This is one of those things that is as much about marketing as reality. There are no shortage of hardware appliance network boxes like BIG-IP LTM/GTM and Bluecoat ProxySG's that cost tens of thousands of dollars and are nothing more than BSD/Linux rack mounted PC's in a fancy case. Thes
Re: (Score:2)
Cisco (and others) make ungodly money because they are perceived as the "best". I won't argue that too much though.
For low end stuff, there are cheaper options. Do you need a Cisco Catalyst to handle 3 desks on a fairly slow DSL line, who aren't doing outrageous sharing between each other? No. Do you have 100 desks, then sure. Could you do the 3 desk operation with a Linux machine and 4 network cards? Sure. In this example, it's cheaper to pick up a cheap hub, than to ta
Re:Are you serious, or just killing time? (Score:5, Insightful)
Sheesh. I wish someone would tell that to our clients. My company provides service to (mostly) small businesses, and half of these little five-man operations have some totally over-engineered Cisco gear acting as their network edge because some smartass, self-styled "IT Guy" told them it was the best. Surprise, he vanishes after plugging it in and collecting his fee, and now the client has all these problems with our SIP service and of course they have no idea how to manage their own equipment, and WE end up looking like jerks because our stuff won't work out of the box with whatever equipment the client has.
Could you do the 3 desk operation with a Linux machine and 4 network cards? Sure. In this example, it's cheaper to pick up a cheap hub, than to take even a salvage machine and put 4 network cards in it.
Here, though, I disagree. At the same company I mentioned, when I joined, we were a three-person operation, and we used a Linux machine with two network cards and a switch as our router. It worked great as we scaled up in staff numbers, particularly when tools like ntop and tcpdump existed to let me see when some joker was ruining it for everyone by torrenting the entire internet. If you never plan to expand, then sure, some cheap little router toy from Dlink or Linksys will do fine, but if you intend to grow, may as well do things right the first time than have to re-engineer your network down the road.
Also, a hub? Who the hell uses hubs anymore? I can't even think of a use for them these days other than packet sniffing, and an inexpensive managed switch will let you do that.
Re:Are you serious, or just killing time? (Score:4, Interesting)
I don't believe in overselling customers. I believe customers appreciate the fact that I'm looking to milk them for extra money. Really, I can score one big scale, or I can build a relationship and continue with them as needed. I've had customers not call for years because they didn't need anything, but the minute they do, I'm there for them.
Growth is a funny thing. A lot of places I've seen have had 4 desks with the intention of growing, and years later they still have exactly 4 desks. One place had a dozen or so servers with high hopes for the future. Those high hopes were a serious understatement. Their partial T3 became multiple GigE circuits, and their dozen server became over 100. Even the first big growth spurt overgrew the agreed upon server naming convention and it had to be changed after two years.
One place I worked at, which was growing rapidly, they were set up with a bunch of hubs (I'll explain the hubs thing in a moment), and terrible links between the suites (multiple suites in a complex). It was terrible. Literally, it was normal to have >100ms pings between suites on a good day. I got 6 Cisco Catalyst 2924XL-EN's with 4 port 100baseFX cards, deployed one switch per suite, and ran fiber between all the suites. Total expense was about $600. Then the economy took a dump. They started downsizing, and I believe they were down to something like 5 desks and 3 servers (don't ask).
Ok, now the hubs thing. I say "hubs" for any low end consumer grade unmanaged "switch". For some manufacturers, it was a marketing ploy to say "switch", which just meant "auto speed switching", where it would handle 10baseT/100baseT/100baseTX, but was still a hub (you could see all traffic on all ports). Some really are switches, but usually not at the level of a real managed switch. If you can get 5 ports for $20, it's a hub. :) I have seen some recently that act like a hub, which is really sad. Well, not just act. They'll even have a single collision light on the front. Oh, there's a big hint. :)
Re:Are you serious, or just killing time? (Score:4, Funny)
'Ok, now the hubs thing. I say "hubs" for any low end consumer grade unmanaged "switch". For some manufacturers, it was a marketing ploy to say "switch", which just meant "auto speed switching", where it would handle 10baseT/100baseT/100baseTX, but was still a hub (you could see all traffic on all ports).'
Your showing your age here my friend. This hasn't been true for many years.
Re: (Score:2)
You can get a decent machine (~2GHz, dual-core, 4GB memory) for less than $1000, today. I'd be surprised if that couldn't handle at least 3-4 gigabits, total throughput.
Agreed. I tested a 2GHz single core and reached over 1000Mbps when using pfSense, and that's without optimization.
Unfortunately, with the traffic shaper enabled, pfSense gets only about half the throughput on the same hardware.
Re: (Score:2)
FWIW:
PCI / PCIe x1 are both ~1Gbps max throughput (not counting overhead, that's raw bus speed).
All the other PCIe's scale linerly, thus a PCIe x4 is 4Gbps bus speed.
After communications protocol over the bus that speed drops (not sure how much). There are other factors as well but what it all comes down to is PCI or PCIe can really handle only about 500Mbps per link.
-nB
Re: (Score:3, Insightful)
FWIW:
PCI / PCIe x1 are both ~1Gbps max throughput (not counting overhead, that's raw bus speed). All the other PCIe's scale linerly, thus a PCIe x4 is 4Gbps bus speed.
After communications protocol over the bus that speed drops (not sure how much). There are other factors as well but what it all comes down to is PCI or PCIe can really handle only about 500Mbps per link.
-nB
Not quite. One lane of PCIe v1 is 250M/s, double that of PCI. One lane of PCIe v2 is 500M/s, double that of PCIe v1. So, a PCIe v2 4x slot would be able to push around 2G/s, or 16 Gbit/s, which is slightly more than the 500 Mbit/s you state.
Furthermore, given that built in gigabit ethernet ports on any motherboard built in the last 5 years or so are connected via PCIe, and I've never had an issue saturating the whole gigabit, it doesn't make a lot of sense to say that PCIe is limited to half a gigabit.
Re: (Score:2)
When someone is giving away exceedingly powerful, linux-equipped servers for free, let me know where and when. Until then, software (as in Microsoft vs. Linux) is *not* equal to software/hardware (as in Cisco). Plain and simple.
Re:Are you serious, or just killing time? (Score:5, Insightful)
And beyond that, just because a Linux box might support all of the protocols and implementations that Cisco has leveraged in their own products, it does not mean that the Linux box is going to configure itself. A lot of the reason that Cisco makes money is because they provide solutions. The solutions themselves leverage established technologies in many cases (RFCs are in the public domain), but Cisco makes them work together. It's the old discussion about Open Source vendors. They aren't making money selling people Linux because Linux is free. They are making money selling people Linux configured to perform specific tasks, and then selling support to keep the solution functioning and up to date.
Re: (Score:2, Informative)
Thus, these guys are setting most of the major network standards, as well as implementing them.
Re: (Score:2)
I worked on enough Cisco hardware in the late 1990s to not only know that what you say is true, but to also decide that I didn't particularly want to be a CCIE anytime soon. Slight differences in IOS releases can lead to serious headaches. I've heard tales that the Cisco CCIE test is basically being thrown into a locked room with a bunch of misconfigured gear and told to make it work within a certain time frame. Thanks, but no thanks. My philosophy is that I'll let the ISPs and telcos handle bringing th
Re: (Score:3, Interesting)
When someone is giving away exceedingly powerful, linux-equipped servers for free, let me know where and when. Until then, software (as in Microsoft vs. Linux) is *not* equal to software/hardware (as in Cisco). Plain and simple.
So you've never heard of the FrankenPix [packetattack.com], I take it?
I'd buy a claim of 'more stable', 'customized', or something similar, but 'exceedingly powerful' probably just isn't true. Barring some evidence to the contrary, I see a given Cisco device as about on par with an extremely weak desktop computer, in terms of pure 'power'.
I'm not undervaluing the total package. I just think you may have gone a tad too far with the Kool Aid.
Further, I don't think the question was asking about hardware anyway, so I'm not cert
Re: (Score:3, Informative)
Show me the Franken' Catalyst 2950/6500 Sup720 3BXL, Franken Cisco 12006, or Franken Juniper M7i/M320, and then I'll be impressed. Your desktop PC will not contain TCAM or other components required for a minimal level of forwarding performance needed by an ISP.
After all these years, a desktop PC still cannot perform the task of a simple 8 port switch, at nearly the same packet rates as the switch. The packet rates that can occur on an Ethernet network easily overwhelm the desktop PC's limited interr
Re: (Score:3, Informative)
Sure, the 2800 and 3800 ISR series can take full tables easily. You can get a 3845 starting at $10k. NM-1T3/E3 module is about $6k. Both the 2800 and 3800 take DDR-266 ECC SDRAM (except the 2801); don't feel the need to pay Cisco's prices for commodity RAM if you really don't want to. The 3845 is recommended to handle up to 2 DS3's. According to people I've asked, you can push a 3845 to 100-150 Mb/s. You can go as low as a 2811 ($2k) and still take full tables, but only at fractional DS3 speeds. I would gue
Free - sure! (Score:2)
Heck, that's the "not so open source model"! Build crap and give it away for free, then charge out the wazoo for "support".
Well, it's a good business model anyways. :)
Re: (Score:2)
Other way around. Cisco bought out LinkSys and is selling their stuff with a fancy Cisco label on it. It's definitely not the same as the normal Cisco stuff. If it's like other acquisitions, Cisco will eventually start producing them and what's under the hood will change. Another good example of this is when Cisco bought out Komodo and rebadged their voip box into the ATA-18x series. Cisco rewrote the software and made it a nice unit.
Be more specific! (Score:3, Interesting)
Without more performance and cost requirements, it's really hard to figure out what would work for you.
Are your users all in one building? Over a large area? Are you talking about a dozen access points or hundreds?
Without some more specific information, only advice I can give is:
Soekris boxes with FreeBSD.
Good luck.
Re: (Score:2)
Re: (Score:2)
Huh, I didn't know that FreeBSD pf is not up to date. FreeBSD does have multiple firewall options, though.
But since OP asked about Wireless support, does OpenBSD have good wireless support now?
Re: (Score:2, Informative)
The Coachella Valley is the area - all of it. A large area.
A dozen to start but hundreds in the near future - i'm going to provide high bandwidth service for next to nothing. So the routing HAS to work for minimal bucks.
Mutually exclusive (Score:2, Insightful)
I'm building a Wireless ISP using commercial grade, low cost equipment.
To me, some words in this sentence seem to be mutually exclusive.
To my humble opinion, a good ISP needs to have good reliable equipement. Sometimes, out of the box routers are better because they don't have moving parts and their firmware could be more stable than a full-blown OS (even if it is Linux).
Disclaimer: Not that I don't like Linux, I use it all the time.
Re: (Score:2)
OpenBSD packet filter supports transparent router redundancy pretty well I think. Used by pretty large corporations.
Re: (Score:2)
Re: (Score:3, Insightful)
To my humble opinion, a good ISP needs to have good reliable equipement. Sometimes, out of the box routers are better because they don't have moving parts and their firmware could be more stable than a full-blown OS (even if it is Linux).
If not for this reason, why do you suppose the question got asked?
Re:Mutually exclusive (Score:5, Informative)
You can have low-cost commercial grade services run using off-the-shelf hardware.
pfSense [pfsense.org] includes support for CARP, which lets you build high-availablity failover clusters. You can have two (or three or four...) cheap systems and if one dies, just fix/replace it as needed. The backup system(s) automatically take over and nobody would likely even notice the changeover.
When it's cheap, that is much easier to consider.
If you want no moving parts, you can use an ALIX box, Soekris, or perhaps even some atom-based boards. If you want to use server-grade boxes to make yourself feel warm and fuzzy, you can do that too. Supermicro even has a server-class atom board in a 1U rack which runs pfSense very well for us.
Re: (Score:3, Informative)
These guys:
http://www.applianceshop.eu/ [applianceshop.eu]
Sell embedded systems with monowall/pfsense preloaded.
Extremely easy to use and reliable.
I use a pfsense one at home, no idea how things would scale...
Re: (Score:2)
I don't think that is the point. Motorola's commercial gear does not support nearly the functionality AirOS and MicroTik do. It's great gear - you just can't make it do some of the stuff you need to do.
Re: (Score:3, Insightful)
"To my humble opinion, a good ISP needs to have good reliable equipement."
To my humble opinion, a good ISP needs to have good reliable *service*.
Ask i.e. Google to learn the difference.
Few requirements given but... Vyatta? (Score:3, Insightful)
Maybe Vyatta @ http://www.vyatta.org/ [vyatta.org] does what you want. I really don't have any idea what that is from the actual post, tho. You need some routing for thousands of users, and can't afford a Cisco UBR. I'm not sure exactly if you wanted to use the UBR for DOCSIS type support for some reason (a la cable modem) but the fact it'll be wireless leads me to believe it won't be. I'm assuming you don't need a lot of physical ports, just something to manage your VLANS, some routed subnets, a bit of BGP, etc. Maybe XORP is what you want, tho @ http://www.xorp.org/ [xorp.org] so you may want to look there. IHeck, 'm not even sure if you want to take a server with a bunch of PCIe ports and slam multiport switchable fabric cards in there like the ones DSS @ http://www.dssnetworks.com/v3/gigabit_pcie_6468.asp [dssnetworks.com] makes, or do something else. Maybe these links will help, and hopefully there'll be a detailed followup so we can aim at the real target :)
Re: (Score:2)
Where Cisco is a good value is enterprise licensed switching. You could buy 37XX, 4XXX, or 65XX gear depending on the level of residency you need and do lots of your heavy lifting there; BGP learning and advertisement and port access control and basic ACLs; you might then put some Linux servers behind some of that to do some of the really complex routing jobs (things with lots of rewrites and NAT operations; process authentication information, provide DHCP with dynamic DNS updates etc. You might save som
What on earth are you trying to actually do? (Score:5, Insightful)
Routing and ISP's are huge topics- what are you trying to do?
The main problem with routing isn't bandwidth- anyone can pump enough 1500 or 9000 byte frames per second to fill a gigabit pipe. The problem is when you have lots of small packets. At that point, dedicated routing hardware with a high-speed TCAM becomes really important.
What kind of line cards do you need? ADSL? Ethernet? OC12?
What kind of services do you need to run? BGP? OSPF?
What kind of bandwidth are you going to be pushing?
Re: (Score:2, Informative)
Tons of multicast video data will eat up 1/4 to half of my last mile bandwidth, followed by voip and data.
I'm trying to balance Access point range to around 1/2 mile without dropping bandwidth, so Ubiquiti AirMAX equipment seems to work in trial runs.
i don't want to drop below 100Mbit lan speeds, rates are fixed so if a customer can't connect they won't kill all the bandwidth for everyone else.
Client's actual throughput will be about 10Mbit down / 2Mbit up + about 45Mbit of Multicast video overhead - 100 cl
Ebay is your friend. (Score:5, Insightful)
Re: (Score:2)
you are aware that Cisco IOS is non-transferable so while you can certainly pick up the HW at a cheap cost your going to either a) run it illegally or b)take it up the ass to get it inspected, licensed and covered by smartnet
Citation? Surely the software that's on the device is all that's needed. Negating support of course. If you're suggesting thats non-transferable, that's like suggesting you can't legally buy an old PC or any mobile phone with similar licences second-hand.
pfSense (Score:2, Informative)
Big Sur Wireless (Score:3, Informative)
There's a small wireless ISP located in the Big Sur area of California that seems to have been up and running for a few years now. Maybe the OP wants to build a system like Big Sur Wireless [bigsurwireless.com]. Their web site includes a lot of details about their homebrew system.
Re: (Score:2)
As others have said... (Score:4, Insightful)
just a thought (Score:2, Informative)
OBSD or pfSense (Score:2, Informative)
My history is: started on OBSD (due to hardware support, ironically); played w/ FBSD; ended up on pfSense.
My observations:
OBSD is absurdly security conscious... for ISPs especially, this is a good thing.
OBSD tends to have a lot of focus on new network features (pf, carp)
most OBSD features get ported to FBSD... but take time (look into carpdev)
pfSense (built on FBSD) has some overhead vs FBSD raw (obviously), but has *nice* management UI, package support, etc
customizations are easy for pfSense (I added some
Vyatta (Score:2, Informative)
What about Vyatta? It's a good router based on linux and you can install it on any old box you want or buy their hardware for it. Even has a cisco like interface if you want.
Buy used. (Score:2)
Why not buy used Cisco routers? In the current economy, you should be able to make some pretty sweet deals.
Nagios, ssh, airOS (Score:2)
You can make ssh plugins with Nagios, AirOS supports ssh and key exchange. You should be able to achieve most things with that combination, what is it you are trying to do?
MicroTik has a strong API, have you tried doing what you need to do by using that?
A suggestion (Score:2, Informative)
RuralLink Ltd (yes, I work for them) does what you want, linux-based wireless network management. Get in touch with us at http://www.rurallink.co.nz/contact-us [rurallink.co.nz]
There's not a lot of info about that side of things on the website, but if you contact us we'll be happy to chat - and don't worry, we're all techs, there's no sales droids here.
Cheers,
Scott.
been there done that bought the tee shirt (Score:5, Interesting)
I founded and operate a wireless ISP serving about 1000 wireless subscribers, and have my own embedded linux distro inside just about everything. It would be a fair statement to say that linux literally saved our business on more than one occasion, by giving us the tools to overcome manufacturer software bugs, by establishing 'known good' systems of various types, by enabling read-only compact flash based systems running on solar power, by bringing a high level of utility and reliability into the critical parts of the network, by allowing us to make it anything it needed to be.
As a CPE, my linux distro never lets me down and never puts customers of at risk of 'stone dead - lights on but nobody home', like linksys/netgear/etc always seem to. Never having to tell someone 'just pull the power and plug it back in' for their connectivity is a real saving grace. And when in a business situation, I can equip these customers with connectivity devices that _do not fail_ and make us look stupid, while at the same time giving them useful feature sets unavailable in higher end router manufacturer gear (cisco 2621 - excellent hardware with great stabillity, just weak on features I get with dnsmasq, openvpn, tcpdump and others.. trying to diagnose network connectivity issues without tcpdump is just dumb.). Its also never choked and zeroed out it's own flash config for no goddam rason, unlike the previously mentioned low-end consumer devices frequently do. Basically, that consumer stuff puts you at risk and is suicide.
As a network appliance, linux flings packets just fine and gives you great tools to filer, mangle and generally control how and what it does. The ebtables code is awesome, the iptables stuff is killer, openvpn rocks asses, dnsmasq kills, there's just too many useful and cool things just go right. I have a pppoe server running rp-pppoe + my patches and userspace tools, running for years now and hit with every kind of client side bug and malfunction imaginable, and just keeps trucking along. Freeradius backed up with mysql is sweet as can be, and quagga for distributing my routes internally is just a dream. I have it all on read-only compact flash, so they never write and basiclaly will run until there is a show stopper hardware problem, at which point I will more than likely be able to remove the flash and put it into another machine and away I go.
There is a lack of management interface, and there is a learning curve to this route, but the upside is very low dollar cost and an attainable level of flexibillity, reliabillity and stabillity you are unlikely to find in any commercial solution anywhere. Cisco IOS is awesome, but you won't power anything that runs it off a 12v battery and solar panel on the side of a mountain and flinging/filtering 20mbps of traffic.
Good luck.
Need help (Score:3, Funny)
Meraki or open-mesh (Score:2)
Have you tried Meraki? Google bought into the company awhile ago and it all runs on Linux. There are proprietary bits nowadays so you can't put your own distro in place of the original code. However less than $200 for solid, lifetime warranty, outdoor gear is nice. The built in meshing control is impressive. The ranges with omni antennas are great. Also millions of users have connected to the 'net via meraki equipment according to the website. I'm currently writing this on a meraki mesh, 4th hop from the ga
FreeBSD, BGP, OSPF, pf (Score:3, Insightful)
Seriously, learn to love FreeBSD.
I am assuming that you will be doing a tree style network with a central location providing you bandwidth on a fiber link or T1/T3 etc.
Get a PAIR(at least, add more as necessary) of nice, quad core Dell Poweredge or HP DL series servers. FreeBSD+CARP them giving you as seamless load balancing/fail over as you can realistically get.
at each hub consider either buying commercial wireless routers or build your own. If you build just keep everything fanless as that is where your equipment will fail you.
Use OSPF on branches while being aware of scaling issues and where OSPF isnt ideal, kick in the BGP and you can link your OSPF clusters together giving an extra level on branch redundancy because traffic can hop to another branch if necessary.
OLSR in mesh cells, OSPF on the cells backhaul router linking these cells and providing multiple route options for redundancy, and BGP between groups of cells and between you and other ISPs etc etc.
You dont need to take the Mesh down to the client, only to the neighborhood AP level. The idea of mesh per client creates too many hopps and clients have too much latency. Ideally, you are no more that a 2-4 hops from the backbone, any more and you are going to be adding too much latency from the hops. When a backhaul link goes down and the OSPF saves your butt by routing traffic through a neighboring cell, you are already going to add latency and you dont need that complicated by 6 hops in the neighborhood and 5 more to the backbone (11 hops over wireless is just too many for broadband).
OpenBSD (Score:2)
Re: (Score:2, Funny)
The fact that you are asking on slashdot shows that you are not qualified, and what you're going to get back is a bunch of others, who aren't qualified, suggesting all sorts of half assed hacks to do it which will just result in a utterly shitty service overall.
Dude, don't shit on a well-known slashdot tradition! How dare you!
Re:Hire someone who knows what they are doing. (Score:5, Funny)
Wait, isn't shitting on topics a well-known slashdot tradition?
Re: (Score:2, Funny)
Nobody expects to get shit on!
(To answer your question, yes. Slashdotters shit on anything and everything. We're like a pack of wild pigeons when it comes to that.)
Re:Hire someone who knows what they are doing. (Score:5, Insightful)
The fact that you are asking on slashdot shows that you are not qualified, and what you're going to get back is a bunch of others, who aren't qualified, suggesting all sorts of half assed hacks to do it which will just result in a utterly shitty service overall.
I disagree. The Open Source community has a thousand hidden gems that a person might not have heard about. Proxmox VE for one: virtualization, with a GUI, with live migration, and if 2.0 turns out, with heartbeat and failover (high availability). Most people have never heard of this where I work even though half the place is virtualized with KVM, VMWare, Hyper-V, etc. I would think the Slashdot, with its plethora of experiences, might come up with a little-known or workable solution in an already developed product that you haven't heard of yet.
Re: (Score:2)
And that, my friends, is the Open Source quote of the day, right there.
Re: (Score:2)
Meh. That applies to all software: Microsoft, open source, games, etc. Proxmox has drdb planned for integration in 2.0. That's about it.
Re: (Score:2)
The Open Source community has a thousand hidden gems that a person might not have heard about.
True. But no "software gem" can do the human-level problem solving their guy needs.
Step in the right direction (Score:2, Informative)
I guess i'm looking for a scalable ISP-in-a-box solution. And if it doesn't exist, then let's build one. But Proxmox VE looks like it will fit well with managing computer resources between the handful of Dell 2950s slated for Zimbra, FreeIPA (Active Directory for Linux), Nagios, Cacti, and AIRControl. Still looking for a good FreeRADIUS server i can tie into FreeIPA - but i need lots of other stuff than just a router-in-a-box. A balance between smartest / practicality / economical directly translates into
Re: (Score:2)
Re: (Score:3, Insightful)
The problem is, if you ask a Cisco person to do it, you'll get a Cisco solution, even if it isn't the best solution for the task.
Re:Hire someone who knows what they are doing. (Score:4, Informative)
I think you have a good point, but I don't necessarily agree. First, we don't know what market the submitter plans on operating in or who his clientele are. We don't know what his experience is, how much resources he has, or exactly what level of service he intends to offer. Like the guy who criticized the submitter for refusing to buy a $300k Cisco router, I think you committed a common mistake in thinking that IT is just a series of 1-size-fits-all solutions, and that if you going to use the "right" solution to each problem, you shouldn't bother.
The era of entrepreneurship and hacking things together isn't over, and it probably never will be. Our tools and hacks may become more advanced, but hopefully there will always be people trying out new techniques and business models, testing new start-up technology, and finding different ways of accomplishing the same goals. The answer isn't always to pay an expensive expert or to use established tech.
As for this:
You could get by with this in the late 90s, but when you're going to compete with cell phone companies, cable companies and standard POTS companies, you probably need to have a bit of a clue.
That's true, but neither my phone company nor my cable company provide wireless access where I live. Cell phone companies provide wireless, but it's pretty spotty and slow, and I live in NYC. There are plenty of areas in the US where no service is available except through dialup. Obviously these large companies aren't interested in competing in all markets, so if you come up with a business model and think you can make it work, then I say go for it.
Re:Hire someone who knows what they are doing. (Score:5, Interesting)
The proper question is: How do I find someone qualified to do this for me?
You mean because he's humble enough to realize he doesn't know every thing, you believe he's unqualified anything. I suggest you look hard in the mirror and read what you just wrote to yourself.
His belief that Cisco UBRs are $300K is a give (Score:2)
away.
CRS1 maybe, but not CMTS routers.
Re: (Score:2, Insightful)
Re: (Score:2, Insightful)
Have you worked at a cell phone, cable, or standard POTS company lately?
What exactly do you think you are going to get there besides a bunch of unqualified, "half assed hacks to do it which will just result in a utterly shitty service overall."
Besides, many of the folks posting in this thread are probably those same unqualified, half-assed hacks who work at such companies. Corporations don't have any corporate voodoo that makes them special any more than someone working for the government makes them any sma
Re: (Score:2, Funny)
Re: (Score:2)
Seriously? That's your answer? You think this is that hard, and you presume that no experts or professionals read and contribute to slashdot? There is plenty of good advice in this thread, though this is not it. My area of expertise is not ISP inf
Re:m0n0wall is a great BSD distro (Score:4, Interesting)
I have to agree, although I registered a vote for PFSense above. PFS is based on m0n0wall and both are excellent routers filling slightly different niches. I currently use PFS at home for its packages (freeswitch, squid), but I recently worked for a growing WISP and got them onto m0n0wall, now serving something in the neighbourhood of a thousand customers.
If you want pure simplicity, go m0n0wall. Otherwise, I strongly recommend looking at PFSense for the squid caching and adjust-on-the-fly connection table size.
Re: (Score:2)
Most home hardware will not get you 5 9's availabilty and if the OP is aiming to be an ISP then you need as close to 5 9's as possible.
Re: (Score:2)
Not according to every home ISP ever. I highly doubt comcast and TWC and COX even come close to two nines. Heck, their ntp servers are probably not even one nine.
Re: (Score:2)
I meant dns, I don't think they have customer facing ntp servers at all.
Re: (Score:2)
So take some of the money you save and buy spare whatever you are using. No delivery is faster than one sitting on site in the closet and another at your second site in case of damage to the building.
We have some used cisco stuff and that is what we do. I could get 9 spares with every unit and it would still be cheaper than the retail pricing.
Re:DD WRT (Score:5, Insightful)
http://www.dd-wrt.com/site/index [dd-wrt.com]
It's Linux on low cost wireless routers.
Yeah, that's just what I'd want my ISP to run as a core router.
Re:no DD WRT (Score:4, Informative)
Re: (Score:3, Interesting)
The dd-wrt shop does have more powerful CPUs/throughput-hardware than is afforded by common WRT-class home routers. HOW much more powerful, or more throughput I do not know. Maybe someone else can comment, given the hardware available.
The prices are reasonable; it seems for about $75 you can buy a outdoor-unit that will blanket an area better than a home router.
http://www.dd-wrt.com/shop/catalog/ [dd-wrt.com]
Re: (Score:3, Informative)
They cut off your network access because of a report of infringement? Are you in the US? Do you think you could mail me at danny@eff.org with more info? We're always interested in the details of these incidents.
Re: (Score:2, Informative)
If you are just starting up, I'd suggest a couple of Cisco 3550 layer 3 switches with the IP Services image. They don't have all the features of the big routers, but they can handle a huge amount of traffic. I doubt you could build a linux router that would handle as much traffic for the same price as a 3550.