Server Redundancy for a Small Business? 81
SadPenguin asks: "I am currently working for a small company of about 15 people each with one to two workstation/laptop machines a piece. We are looking for a new server solution, as our last one crashed, and lacking any server redundancy, we nearly lost all of our data since our last backup (it was only a few days, but an important few). What the kind of server (and redundancy) solution would be appropriate for a company of my size? Most advertisements are for large scale enterprise serving solutions, but these are costly and excessive for my situation. I'm sure that there is a simple Redundant Server technology out there that is a bit less costly, but won't result in any downtime in the event of a motherboard component failing (like we faced this time when our mysterious surface soldered VRM failed). So what do you use? What should I use?"
A few recommendations. (Score:5, Informative)
I actually run a computer consulting firm specializing in small businesses. I'll outline some of the more common recommendations - with what I think is the most important first.
From my experience, the best approach is to layer your defenses. I'd REALLY recommend a UPS (I generally assume this is purchased with a server, but it isn't always) at very least. Your local power company is only required to provide you with something CLOSE to 120v. They generally can't keep it consistent enough for power supplies (and electronic componentry in general). Protect your investment, UPSes are generally relatively cheap.
The fact that you've got a backup solution is good, but (as you've seen) not enough. Evaluate it, and see if it's consistent with best practices - i.e., is it a tape (or optical) backup system that is done in rotation and taken offsite by somebody in the company? If not, set that in motion first.
Next, some sort of drive redundancy is in order. At very least, mirror your drives. I generally recommend RAID5 (or one of its variants), but in very small companies RAID5 isn't either required or affordable or both. IMO, the jury's still out on the long-term viability of IDE RAID, but I think it looks promising.
Finally, redundant power supplies and NICs (for those of us that are REALLY paranoid ;) ). I've had a couple of servers' power supplies die on me, but the server kept right on ticking thanks to a redundant unit.
If it's affordable to your company, consider hot-swappable server components, as well. This significantly reduces downtime to your coworkers... and expense to your company.
Hope this helps. Good luck!
Oh yeah, FP ;)
Re:A few recommendations. (Score:1, Offtopic)
Excellent one as well... anything else that follows will largely be redundant...
Re:A few recommendations. (Score:1)
Some folks will recognize someone else's post from a day or two ago.
Re:A few recommendations. (Score:3, Interesting)
A mysteriously soldered VRM sounds a bit odd, you might think about going wi
RAID 5? (Score:1)
Re:RAID 5? (Score:1)
PC Pitstop [pc-pitstop.com]
Cooldrives.com [cooldrives.com]
Adaptec [adaptec.com] (DuraStor line... a bit beyond your stated price range, though)
Hope this helps!
Re:RAID 5? (Score:2)
As for a standalone box, I don't know anything that's sub-SCSI, but I imagine someone sells something similar. It'd be really slow though. Firewire can average out around 14MB/s and firewire 2 is still missing critical mass. If in doubt, get a 4 channel firewire card, 4 drives and tape them to eachother
Re:RAID 5? (Score:2)
Re:RAID 5? (Score:2)
If you look in the External storage solutions at Adapter you'll find that there are NOT SATA/IDE interfaces. As for internally, you can use SATA hard-drives in the setup, but no matter how you slice it, a 12-drive storage array is not a viable option for:
Small businesses (5k for the chassis, n for
Re:RAID 5? (Score:3, Interesting)
I have a maxtor FW single HDD backup solution, but I definitely would not recommend that particular one for constant on situation (for lack of ventillation). It seems that when the drive does the temp calibration the FW insterface hiccups, and the ongoing transfer gets interupted. All is well after diconnectio
Re:A few recommendations. (Score:4, Informative)
The first red flag I saw was that although they had backups, they were three days old. If the data is worth saving, it's worth doing it right. Full backups on the weekends and incrementals nightly.
Ok, the redundant stuff... power supplies, hot swap drives, RAID5. You're approaching a $10k configuration. That, BTW, would have still gone down because they had a motherboard failure. And since they needed backups, their drives were corrupt, so the RAID probably would have been too.
Really though, this whole question is about designing their new server without any idea of the load required. Based on the info that is available, I think I would lean towards purchasing two servers. Make them a little smaller than what you would purchase if you only had one, and divide the load between them. If one fails, you can temporarily transfer to the remaining one until you can get it fixed. You could even go so far as to move drives and RAM temporarily if necessary. Just make sure the equipment is server rated. IE: my Dell 400sc Poweredge servers are rebadged desktop machines. My Compaq Proliant 800s are definately not. Even good equipment is getting pretty cheap if you have reasonable requirements.
Above that; daily backups. The UPS equipment like you suggested, just keep in mind that UPSs are consumables. And possibly IDE RAID-1. Drives are cheap and 15 users shouldn't need the performance of SCSI.
Re:A few recommendations. (Score:3, Insightful)
Re:A few recommendations. (Score:3, Insightful)
And $10k is a huge investment for a company of 15 employees if they aren't technology based. Most would start to squeal long before you hit $5000. Sometimes you just have to be happy that the 'server' isn't the owner's PC.
Re:A few recommendations. (Score:2)
With 'nix, even software RAID-1 works well. RAID-5 is also a choice. Doesn't take an insanely fast CPU (or a monitor that that matter)... so you can manage a multidrive backup machine for under $2000 or even under $1000.
Re:A few recommendations. (Score:2)
Let's be clear about what you have here... You've taken a box, stuck a pair of IDE drives in it and called it a server. While not necessarily a bad solution, it's not in-line with the post I replied to that suggested redundant
Re:A few recommendations. (Score:2)
It depends (Score:1)
Daily backups (Score:5, Insightful)
Daily backups !
general recomendations:
quality server (Dell/HP/etc)
NO ide drives!
SCSI in software raid5
minimum software install (e.g. no compilers)
get second 'devel' server to test/compile software before using on production server
If it is not broken, don't fix it. as in screw with the devel server.
Re:Daily backups (Score:2)
Re:Daily backups (Score:2)
Re:Daily backups (Score:3, Interesting)
But I've been in the industry for over fifteen years with thousands of clients and the last time I had a hardware raid do that was almost six years ago.
Software raid on the other hand inevitably takes more time/effot/energy to recover from failures (especially if you're so foolish as to use what's provided by Win2K!).
Hardware hot-swap RAID is easy, just change drives and nobody knows anything happened.
Software RAID usually requires at least a reboot if not fiddling with syste
Re:Daily backups (Score:1)
If they can afford to get a top end RAID5 setup from a quality vendor, it's the better choice. You can be relatively assured that when that raid controller dies in a few years, you can get another card that can import your config and recover your data.
If you are trying to do things on the cheap, and cannot get the top of the line RAID card, software RAID provides the hardware independence to upgrade cards and drives as needed, with what ever is cheapest at the time they fail
Re:Daily backups (Score:2)
SCSI is stupidly expensive -- I can build a 750GB raid 5 (4x250GB) for the same price as a 140GB SCSI drive. Both of these solutions are roughly $1000CAD. Even throwing in a 3ware SATA controller is still cheaper than doing a software Raid 1.
I also think that everyone is going a little overboard -- I'm pretty sure that original poster does not need redundant servers running linux HA and development and production servers. They can probably afford downtime over hot swappable memo
Re:Daily backups (Score:2)
Daily backups !
Why not hourly backups ? Hourly incrimentals + daily full backups ?
Alex
My small company solution (Score:5, Interesting)
I work for a small company that only has three full-time employees (including me). I use two Debian boxes (cheap-o machines that are just retired desktops with some big cheap IDE hard drives in them) running Samba. I use the rsync mirroring technique I found here [mikerubel.org].
One box is the "live" server and the other mirrors the live server every night. If the main server dies (which happened once - power supply failure), I can "promote" the backup server by changing one line in its Samba configuration. As a bonus, the backup server keeps "snapshots" back a week or two.
Re:My small company solution (Score:2, Interesting)
If you really wanted to save some more money you could use an external drive to rsync to although you would have to get your server fixed before you could copy the rysnc'd files back over.
Re:My small company solution (Score:1)
-Shane
Simple, cheap answers for redundancy... (Score:4, Interesting)
* Mirrored Raid in all servers
* A regular workstation with a good, large had drive that copies the server data to itself nightly
* A DVD-RW backup made nightly on yet another workstation, with at least one off site - 5 discs, one each weeknight, replaced a few times a year.
In most cases the server RAID (cheap ATA promise controllers) takes care of 90% of the problems - only one HD goes bad at a time, lightning strikes rarely take out the hard drives at all, nevermind both hard drives, etc. Even if it dies it's unlikely that the problem affected the HD backup on the other workstation, and it definitely didn't affect the cd-rw.
However, whenever you get a catastrophic failure in any component in the server, replace the entire thing. If the MB or power supply fails, copy the data to new hard drives, and use the old ones in less critical applications, etc.
Much cheaper than an 'enterprise' solution, and it should be because your application doesn't require such a solution. Use large tape drives in place of the dvd-rw if you must back up a huge amount of data on a nightly basis.
This sort of solution is very tolerant of cheap hardware, so replacing the server later may not be such a major cost.
-Adam
applications (Score:4, Insightful)
This is a hard problem(NP-Hard perhaps, I'm not sure), and you need to have a:
List of applications you want to protect
Budgeted amount
What threats you are trying to protect from
What kind of failures you will tolerate(do you need 99.9% uptime? or better? worse?
You could, for simple applications, like web service, bump up a pair of linux machines, gimmick some replication between the two, and hope nothing goes wrong, if you have a very low budget, and you'd probably spend a fair amount of work debugging later on, "synchronisation problems". But for redundant storage. The openssi project [openssi.org] is working on highly-available single-image clusters for linux, in an open source model, they might be your first place to look. It's not however, something for the unprepared to do, nor is it something that I'd recommend if you do other tasks for this company. Maintaining such a beast will require a significant implantation investment. The good news is that once everything works to your satisfaction, you can probably take a 4 week vacation somewhere with golden beaches and much sun, and let it take care of itself. I can't stress this enough, this is a hard problem, if you really want to do this right, you'll want to surround yourself with qualified people with experience in this field, it's non-trivial, and mistakes can lead to severe data-loss.
Our backup system (Score:5, Informative)
If the main fileserver goes down I simply change the configuration to read/write and change filemaping on the users machine and they continue to work. The whole process will take about 10 minutes to reconfigure the server and a couple of minutes per user machine.
As a bonus I dont delete the intermediate versions of changes files as I update the server. Instead I compress them with a unique filenames. So I can recover a fairly complete history of any given file. I have yet to fill up the 80gb drive so I havent needed to delete any backups. When the backup drive is full I will start deleting some of the older version, I should have room for about 6 to 9 months of backups at 15 minute intervals.
Re:Our backup system (Score:2)
One thing to add; I really hope you're not relying on the backup machine as your sole source of backups; if you lose the site (fire/flood), you lose all y
Re:Our backup system (Score:2)
As well I copy all files that have changed during the day to a dated directory I periodically burn them to a cd. I end up with about 1 cd a week going back over 2 years.
All together I have 5 levels of backup.
1) Onsite mirror updated every 15 minutes with incremental version
Re:Our backup system (Score:2)
You need multiple backups, and they need to be cheap.
First rule of backups is that when you need them, something is not the way it should be and any scheme that assumes everything is normal is quite likely to fail. This means you want any failures of the backup systems to be as independent as possible of failures in the main systems.
Second rule of backups is that every backup except the one that matters was a complete waste of time. Backups need to be chea
Re:Our backup system (Score:1)
What's your method for this? I recall something along those lines from the last time I read the rsync man page/docs, but I'm wondering how you go about it.
Re:Our backup system (Score:2)
For exam
don't fall into the RAID trap (Score:5, Insightful)
99.9% of the time when I've had to retreive data from backup, it was because of human error. I.E. someone deleted something they shouldn't have, or the moved the wrong directory to the wrong place, or an error was made during a software upgrade, etc..
the rest is due to random harware failure which would be a reason for using RAID. But pouring thousands into redundant servers and disks, is overkill for a biz your size.
If someone accidently wipes out a folder or data, your raid disks won't be any help.
Love,
Zaq
Re:don't fall into the RAID trap (Score:5, Insightful)
I think it's a mistake to make a blanket statement that a RAID array overkill for a small business. My company is similar in size (18 employees) and a RAID is absolutely essential for us from a downtime perspective. We simply can't afford to be down becuase a drive crashed.
Sure, backups are essential for the lost/deleted file, but a RAID (or at least a mirrored drive) keeps your server up and running. Not everyone needs that type of reliability, but if you figure the cost of recovering from a failed hard drive (even in a small company), the additional cost of a RAID upfront is well worth the investment.
Re:don't fall into the RAID trap (Score:2)
Exactly. RAID is all about buying time when a hard drive fails. My personal server ate it's OS drive, and from a user's perspective, you would never know it. Being lazy, I waited several months before I replaced it. OTOH, at work, I hav
Re:don't fall into the RAID trap (Score:2)
With WinXP SP2, it also seems you can use WebDAV shares like normal fileshares -- an interesting project would be interfacing this to Sub
Re:don't fall into the RAID trap (Score:1)
WHAT are you serving? (Score:1)
Daily backups, #1
What kind of server though?
Mail? SQL? Files?
Server Clusters and Raid 5 (Score:2, Insightful)
Set them up with a shared hardware Raid 5 device.
There is only one active Node in the cluster at a time, if that one fails the second one assumes the identity. Works great never fails!
We are a bit larger so we use EMC Symmetrix, however a smaller shop could probably do a low end EMC Clariion CX200 or the like.
Backup Server (Score:1)
The two main servers are xeons with raid 5, redundant PSU's etc etc. One server runs the domain and as a file server while the other runs the manufacturing software suite (heavy database workload). All the data is very important but I rarely have a problem with lost data , unless some smuck over-writes a file or something stupid like that.
The backup solution I implemented was a Debian box that runs rsync every night ba
Re:Backup Server (Score:2)
Karma be damned, you gotta learn the difference between loose and lose. Look, you lose your virginity to a loose MILF. Get it? DAMN, DOOD, every day. Idiots!
5000$ or less and you are in business (Score:3, Interesting)
An option (Score:3, Insightful)
I too have long experience doing small business consulting and in some other areas. One thing you could do is use RAID-1 with a spare drive. That way if you lose one, you aren't screwed. You also could have a couple spare drives in hot-swap carriers. Pull a drive every night and have a duplicate of your server. Fire up the duplicate server and pop in your known good pull and boot if you server fails.
OS dependent, you don't even have to have exactly the same hardware if you use a more generic kernel build and you can list a different NIC for the spare server in the conf file for modules assuming you aren't compiling them into the kernel.
Continue with good backups made to another machine, to tape/CD/hard drive, or off-site. This way, even if your good pulled drive is a little out of date, you can bring to data current in short order.
You don't mention the OS of the server or budget, but I'll assume that since you've got 2 machines per desk time 15, you can afford a spare server. You don't mention OS and that affects cost, but still, if you are doubling up on hardware on desktops, you can afford to do this or most any of the other solutions offered.
Of course, you get what you pay for and if the experience is lacking in house, hire a knowledgeable consultant or company you trust to do it for you.
Cheap Redundancy (Score:5, Informative)
* two identical servers, running linux (of course).
* heartbeat
* drbd
* two UPS
Notes, Ins, Outs and What Have You's
service redundancy
heartbeat is used to make 2 servers look as if they were one. if one of the servers dies, heartbeat makes sure the other assumes the ip address and has all the relevant services started.
data redundancy
drbd is a network block device. again, it looks like one device, but when data is written to it, its actually being written to 2 seperate locations. if one box goes down, heartbeat makes sure drbd makes the other box primary.
hardware
these two call for a dedicated network and serial connection. so 2 nics and a serial port per box.
definitely raid array of some sort.
see drbd.org for more details.
this is not a 100% proof setup, but its cheap and covers most of the bases.
of course, it requires a linux dude to get it all to work.
Re:Cheap Redundancy (Score:2)
I'm a linux dude, I'll do it (I already have for a company's mail servers). But I'm in the UK...
Re:Cheap Redundancy (Score:2)
Re:Cheap Redundancy (Score:2)
First, it's not trivial to set up. If configuring one server is 1 unit, configuring redundancy is (1+n)^2 units of work where 'n' is the number of services that need to fail over. Maybe that's a little high; ((1+n)^2)-n might be closer.
If the machine is internal-only (no public IP),
Re:Cheap Redundancy (Score:2)
You'll know what I'm talking about when you have a backup server die the day after you switched over to it when the primary failed.
Diversify at least the the mainboards, power supplies, and hard disks.
A little addition for the UPS's: plug them into different power outlets on preferably different circuit breakers (if unknown, try opposite ends of the room). No need
If you're using Linux (Score:4, Informative)
If you have appropriate hardware you might also appreciate Stonith [linux-ha.org], which provides forced-shutdown of a failed node (in the case that the failed node won't release the IP address, and hence you would otherwise have problems switching service).
If you're in the UK then give me a shout and I'll set it up for you (for a reasonable fee)! My contact details are available on my web site.
Simple things first (Score:2)
as a refreshing alternative... (Score:4, Interesting)
if people actually work with plain text docs, they would love how CVS,etc will merge multiple users' changes.
Of course you would back up your CVS server but in case of a crash, chances are that very important file can be found on the desktop of the user who edited it the last time. Much better than relying on a network drive and then it is just not there.
Rsync (Score:4, Informative)
I would suggest springing an extra $90 to get two extra gigabit ethernet cards and a crossover cable for a dedicated connection for rsync which doesn't compete with office traffic.
Using rsync as a basis, the solution could be made as low tech and simple or automated complex as you feel is needed.
-Pete
Do woodworking? 50 Router Bits [starvingmind.net]
Re:Rsync (Score:1)
My ideas.... (Score:2)
The other is redundant hardware. As people point out, RAID etc. only provides protection for the redundant components. If the controller, motherboard or such which is not redundant goes bad, you are screwed. The best solution is two server
A couple of solutions (Score:4, Informative)
If you are using Linux/UNIX/*BSD, you could use Rsync [samba.org]. There was a great article explaining Rsync usage in the June '04 print edition of SysAdmin [sysadminmag.com].
If you dont see (Score:1)
1. Do the things mentioned in other posts.
2. Distributed OS.
3. Offer offsite backups
4. Profit!
Re:If you dont see (Score:2)
Obviously, you'd have to limit your hardware configurations somewhat due to constraints, but that would be a good learning experience for why you needed the hardware and what each redundancy was buying you.
Re:If you dont see (Score:3, Informative)
* Small business owners are CHEAP. They don't want to spend a nickel on something that isn't an immediate problem.
* They don't see the value in disaster recovery until they experience the disaster.
* They are hard to sell and market to.
* They often use horrible niche-market server based solutions that are Windows only.
I spent a few weeks talking to various business owners about
Re:If you dont see (Score:1)
Re:If you dont see (Score:1)
Of course, the FUD angle is: what would you do if your server was eaten by worms/viruses?
Again, it is a great idea, but one that would be very problematic to actually sell.
Jonathan
Re:If you dont see (Score:1)
This reluctance makes me think about my car insurance. I have it and my driving license for almost 10 years but never had an accident but still pay about (average cause in the beginning its more expensive) $725 a year for now ten years.
The chance of me getting into an accident now are almost zero.
Re:pugservers (Score:1)
Network Server:
Ability to serve web pages, mail, files, printers and other traditional network resources. Also act as firewall and router.
Key feature:
You use one computer for multiple roles each role will run in its own space like a virtual machine. This is for enhanced security and stability. For exa
rdiff-backup (Score:1)
My solution.. (Score:2)
we have a nice server with 5 36GB drives, running RAID 5, and another old system, with 2 120 GB IDE drives running RAID 1 in software (redhat), this machine rsyncs every hour with the main server... Its been fine for 2.5 years now.. lost a drive once in the RAID 5, replaced it and everything came back up fine...
Google Style Redundancy. (Score:2)
I would suggest buying a number of the inexpensive wal-mart PCs and clustering them redundantly. Keep spares around for emergencies - emergency switches, nics, drives, etc.
This is a more technically complicated environment, because you have to worry about data consistency between computers, but, these walmart PCs are disposable and can work independen
Automated nightly remote encrypted backups (Score:2)
Think things through first... (Score:1)
Instead of using hardware that'll breakdown... (Score:1)
Cheap, reliable, efficient offsite backup? (Score:1)
Initially, we mirrored a Snap drive to a remote site via rsync, but dropped that when we downsized. We've used Backup Exec to a 30gig tape, but that's finicky - tapes seem to go south for no discernable reason. Currently experimenting with DVD, but it takes lots of di
pug servers (Score:1)
http://www.pugservers.com/