VPN Solutions for Distributed Installations?

merreborn asks: "I work for a very small software company (10 employees) that's developing a Point of Sale solution for a small retail chain (~20 stores in several states) on the other side of the country. We're going to be shipping Debian systems with our software installed to these locations -- all of which are connected to the Internet via consumer-grade DSL, and inevitably behind some sort of NAT box. Our office is similarly connected, and we've got a couple of dedicated, co-located servers off-site with static IPs. We'd like to be able to access these systems remotely for maintenance from the office -- what would that entail? Which VPN solutions are best suited to this situation these days (IPSec, PPTP, vtun, ssh, ssl/OpenVPN)? Are there any detailed, current books on the subject? (O'reilly's VPN book is 6 years old now)"

Yes.

Next question? :D

Seriously, OpenVPN would do the trick, and I do it right now. The only thing that bugs me about OpenVPN is that you either have to set up a key signing authority, or use pre-shared keys. The key signing authority process is well documented, it's just that I've never actually been able to make it work. Pre-shared keys works just fine though. The protection isn't as good however.

Once I get key signed OpenVPN working then this solution is a no-brainer.

Re:Yes.

If you are doing it yourself the choice is between OpenVPN and OpenVPN.

Advantages:

• Ease of setup. Once you have an SSL CA setup the OpenVPN part is a piece of cake
• Possibility to use multiple links, load balance, failover, hang yourself by any means necessary. See Caveats though...
• Possibility to use QoS and run VOIP on top with no worries. While IPSEC security is considerably better studied than OpenVPN (this does not mean it is better, it is just a devil we know), IPSEC has a major failing. In its most common VPN use which is tunnel mode it is utter piece of horrid shite as far as QoS is concerned. Shameless plug: you can lift off QoS setup for OpenVPN off my website [sigsegv.cx]
• Possibility to get hardware acceleration on the cheap. It is trivial to get OpenVPN to work with an SSL library which has via padlock support. A padlock capable motherboard is around 120$. This theoretically gives you 50Mbit hardware accelerated AES. Practically - see caveats
• Ease of debug and understanding. It provides you with the notion of interface. You can tcpdump it, collect stats, check its status, you name it. You do not get any of that with IPSEC.

What you need to keep in mind are a list of

Caveats:

• OpenVPN will copy from userland to kernel and back to perform its task. As a result it has a speed limit per client which cannot be worked around. It is a fundamental limitation and is around 5MBit per client (multiple clients bandwidth grows as a log of the number to a total of around 15-20MBit). For a distributed installation or road warriors this may prove to your advantage, because no single client can eat all the resources. There is always some resource to go around. If you want higher speeds on a single encrypted point to point link you are better off with IPSEC transport mode overlay of IP-in-IP or IPSEC overlay of PPTP.
• OpenVPN route mechanism has minimal error checks and will bugger up your routing table majestically of you decide to do something really fancy. If you want to run a large distributed infrastructure you have to run quagga and use OSPF or RIP for routing. Either works fine provided that you can do them and you use peer-to-peer mode of OpenVPN. This also allows you to interoperate nicely with any failover within your network and this is something you never get out of IPSEC.
• You cannot use the Server mode of OpenVPN along with routing protocols. Actually I think that there are some fixes in the Quagga CVS head that will allow this but I will advise against this. This is a mode for road warriors. If you want infrastructure you better set your tunnels properly as peer mode ones.

If you are doing it vs someone else, especially someone with an overgrown IT department full of certification waving droids you have to use IPSEC. I have had some bad experience with SWAN varieties and personally I would suggest using Racoon and the KAME stack. Anything else aside they have some good debugging and so far I have managed to make them interop versus every implementation I have tried.

Debian... and PPTP

You may be stuck, unless you're willing to switch to Ubuntu. I've tried them all, and only PPTP on a Ubuntu server or a recompiled Debian kernel w/the MPPE patches (or the latest 2.6.15+ kernel) works very well. I'm not sure how Debian is reacting that 2.6.x is standard now, but it's slow to change away from 2.4. Ubuntu is better in that reguard (it's base is 2.6 with patches).

IPCOP Works Well

In a similar fashion we provide support for an application base that we are growing. If they want "Premium" support then we provide an IPCOP firewall for the location and turn the VPN tunnels on only when we need to support them. IPCOP is free and very reliable and we then deploy it on a low profile microATX desktop case not much larger than a Cisco PIX. Works well.

Tinc

Someone already mentioned OpenVPN, i would also look at tinc (http://www.tinc-vpn.org/ [tinc-vpn.org]). It supports full mesh routing between all your sites, which would be a pain with OpenVPN. Of course if everyone is connecting back to a hub, then not a big deal.

Also for your NAT boxes, if you want to do it cost effectively, get some Linksys WRT54GL's and install OpenWRT. You can then run your VPN (openvpn or tinc) on those routers, which would make a much cleaner VPN network.

ssh could be good enough

If you know what the remote IP addresses are going to be (consumer grade but fixed IP addresses at remote ends) then ssh would be an adequate solution by itself, and a lot simpler than most of the alternatives. With its ability to forward ports and X windows displays, it can handle pretty much anything.

If you need constant monitoring and interaction a real VPN may make more sense, but ... think carefully about how much complexity you add in the management layer here. Does that overall improve or degrade the total environment's reliability and managability?

one word... Hamachi

http://www.hamachi.cc/ [hamachi.cc] Hamachi is a very easy to use and extreemly hard to block VPN that looks to your system as if it was another network device. these days I leave my laptop at home and access all of my daily needed data + VNC as if it was sitting right next to me

compartmentalize!

I'd have to disrecommend running a VPN between these sites simply for your convenience; it would mean that a security failure at any point on the network could jeopardize all of the machines in the network. I recommend you stick with ssh/scp for access to those machines.

Try SSH

I've has a very simmilar situation last year and we found that ssh was much easier to work with and any vpn solution. The only potenial issue is since you are using consumper grade DSL ip address may change. But that is very easy to get arround by having the remote systems cheching there ip address every so often and when is changes sending it to you either by email or posted to a web server.

"Some sort of NAT box"

Its called a commercial firewall. Its tempting to roll your own using a $45 Linksys and CIPE/OpenVPN/IPSEC/PPTP/Freeswan, but seriously, do you want to spend your time watching messages like "Processing a NONCE.." ?

Buy some small, even older, used, Netscreen firewalls for a few hundred each. If you do the preshared keys trick, and put them in aggressive mode, they'll all connect back to the central hub firewall, a Netscreen 10, or whatever model replaced it.

It just works, no dicking around with /etc/ubuntu/foo.key or chintzy NAT boxes that can't pass protocol 50, etc. etc.

OpenVPN rocks for this

We currently use openvpn for a remote management service that my company offers have been using it for over a year now, more than 50 customers up, works from behind nat, with dynamic IPs, through all sorts of nasty things, and as long as the internet is up, the VPN is up and we have connectivity. Ive used alot of different VPNs (openswan, cisco, PPTP) nothing comes close to the stability of openvpn tunnels, especially when dealing with adverse network conditions (NAT of any sort, multiple NATs, poor link quality, etc) even if the internet link is pretty spotty, openvpn does a very good job of automatically renegotiating the tunnels as soon as it has connectivity.

Quick breakdown of obvious options

Basically there are three groups of VPN "solutions" these days: IPSec, PPTP, and everything else.

I use IPSec pretty extensively. If you're dealing with inter-Linux-server communications where each end has a static IP address, IPSec is hard to beat. It's simple and pretty easy.

PPTP is mainly a Microsoft thing. Not applicable here obviously.

"Everything else" breaks down into application-specific protocols for specific applications. This is what I would recommend. Go take a look at OpenVPN. When you don't know the remote IP address, it's a great way to go. You give it a static IP address (I use 10.2.0.0/16 for this) via OpenVPN, and you can log in quickly and easily. OpenVPN has a plethora of options which make it very useful for unknown remote networks. The most useful ones are its decent support for TCP/IP (so you set your colo'd server's OpenVPN to listen on TCP/IP port 80), and the ability to use arbitrary ports (TCP/IP isn't the best protocol for a VPN application; UDP is better - set it to port 53, and that'll get past most over-anal firewalls).

Have fun

ZyWalls

We're using ZyWall 2 boxes [zyxel.com] for NAT/routing/IPsec VPN. At ~US$200 each they are pretty economical, and very easy to setup via http config. Even has support for being a DynDNS client, which is just fantastic for DSL without static IP. You would need a beefier model as the concentrator, but they arent much more expensive - eg Zywall 35 supports 35 sessions @ around US$600. They also can be configured to play nice with just about any other hardware (Sonicwall, etc) with proper IPsec support.

Easier

Create a web site that echoes back the requesters IP address. Put it on the "dark web" so it isn't spidered, and you don't get hit with traffic.

On your client box, run a script that hits the web site (wget) and fetches the IP address. If that has changed, post the new IP address, and installation name.

Now you have the clients and the assigned IP addresses. You can then use SSH to build whatever infrastructure you need to the client box, securely. No need to worry about the brand of router used, etc. About the only problem is if the client uses a dialup on demand connection. To accomodate this, the "poll for IP" can be modified to always submit information, and ask if the connection should be retained.

If the connection should be retained, the remote operator can be notified.

I used this approach to securely administer remote Linux machines over direct connection and dialup for years. Now I find none of my users use dialup anymore (finally).

Ratboy

Groove

http://www.groove.net/ [groove.net] is what you need. Supported by Microsoft too.

OpenVPN

I would recommend OpenVPN because I have some experience with it. OpenVPN is very reliable solution when you have to connect several remote sites to single L2 (ethernet) segment.

We use Intel-based Linux server at our datacenter as VPN server. It runs several instances of OpenVPN on different UDP ports (OpenVPN can use TCP as well) for different customers. Endpoints are Asus WL-500g Deluxe routers with OpenWRT Linux and OpenVPN installed. Maximum throughput is 3Mbps with blowfish encryption and authentication (limited by 200 MHz CPU). These devices are small, silent, inexpensive and reliable enough. Endpoints are connected using various types of Internet access -- DSL, Cable, LAN, WiFi etc. Some customers have ~70 endpoints without problems.

If you insist on using Debian computers as VPN endpoints, do not use harddisks!!! They will die. Use IDE flash, for example. Use fanless CPU and PSU if possible.

Made in Japan - The Teriyaki Experience

This Canadian customer of ours has about 80 restaurants and has fully deployed our Linux & X Window System POS solution in all of its restaurants all across Canada. HQ enjoys an open VPN link with each of them and all data from the restaurants, including credit/debit cards is remotely synchronized with the storage system at their Toronto HQ. The company's IT staff is actually just one person, Doug deLeeuw. The company is increasing its units by about 25% this year. When you have the kind of control that this company has you find something like that much easier to undertake and you're much more likely to succeed. I doubt that there's another restaurant organization in the world with this kind of advanced POS deployment, not to mention that one person did it all by himself. Perhaps in another five to ten years you'll be able to read about it in a book.

Hardware

If you really want a trouble free setup, I recommend using some sort of hardware VPN. Firewall/VPN boxes can be purchased for less than $400 and are great (Juniper Netscreens, Sonicwall, Watchgaurd). If setup correctly, the boxes will almost never fail. You can also use the firewalls/vpns in a situation where the client networks have dynamic routable IP's. This assumes that your office has a static.

This approch can even be taken to the open source "fanboys" Just download a firewall distro like smoothwall. Install on cheap whitebox.

Hardware is so much easier to maintain then maintaining each client and dealing with "some sort of NAT box"

m0n0 baby!!!

We use m0n0wall (http://www.m0n0.ch/wall [m0n0.ch]) for this exact thing...it supports a number of different hardware platforms, including PC, but my favorite is the pcengines WRAP boards (pictured in silver with antennas here)

http://img.m0n0.ch/gallery/brandon_kahler/01_19_06 _WRAP_Wireless_DSL_Large_Text.jpg [m0n0.ch]

They run off of compact flash and the WRAP boards + case are ~$200. They will act as your NAT firewall behind the commodity broadband interface (dsl/cable) and have a great number of features, including a captive portal if you want to allow customers to use the wireless network.

pfsense is based on m0n0, but not meant for the embedded platforms

OpenVPN all the way!

OpenVPN [openvpn.net] all the way! My server and client config files are each < 10 lines long. I manage my certificates with TinyCA [sm-zone.net]. I think all of this is readily available via apt. Also has Windows and OSX clients.

Router-based (Cisco 800)

I guess that if you're asking this question, you don't have any experience with linux-based VPN. I also think that if you are have to do troubleshooting, the last thing you want to debug is your VPN.

For my part, I also started with linux-based VPN (openvpn, ipsec) for private use (3 sites), but then, I come to the conclusion it wasn't worth the effort & time spent. I switched to the Cisco SoHo routers (the 800 series [cisco.com]) who are just working. I have automatic tunnels between all sites, and can to VPN connection directly to any of the sites, plus many other funny things (IPv6). All this with just simple configurations, mostly through the wizard (SDM [cisco.com]) or by copy, adaptation & paste of sample configs.

Of course, these routers may be a little bit too much (of configuration or price) for you, so you may also want to try consumer-grade solutions (e.g. Linksys BEFSX41, Netgear FR114P, ...).

Disclaimer : I wish I could get a percentage of Cisco sales ;)

PS : oh, and port tunneling with SSH is, from my experience, an awful solution for VPN.

Openvpn & hardware solution

hi, everone already has given their opinion about openvpn. so here's mine:

i've run an openvpn solution between corporate LAN and datacenter, and it worked okay but i'll take a look at some dedicated hardware box for the next implementation. maybe netscreen or so.

why?

Well first off, when one doesnt yet have a linux router/fw available one has to buy that. this'll probably cost as much as a cheap netscreen box.

second, when running openvpn on a nondedicated box openvpn has to fight over resources with other services on that box. with a netscreen box this is not a problem.

vtun devices

Check out http://vtun.sourceforge.net/ [sourceforge.net]. I know of at least one VoIP appliance company that uses vtun links to their home base for updates and managment.

FreeSwan

Having used FreeSwan on a few linux clarkconnect systems, I have found it to be a most reliable package when installed under debian, however, if you are a newb, and are feeling a bit out of your depth here, clarkconnect can offer a really cheap easy to set up solution that is well maintained software wise.