Server Redundancy for a Small Business?

SadPenguin asks: "I am currently working for a small company of about 15 people each with one to two workstation/laptop machines a piece. We are looking for a new server solution, as our last one crashed, and lacking any server redundancy, we nearly lost all of our data since our last backup (it was only a few days, but an important few). What the kind of server (and redundancy) solution would be appropriate for a company of my size? Most advertisements are for large scale enterprise serving solutions, but these are costly and excessive for my situation. I'm sure that there is a simple Redundant Server technology out there that is a bit less costly, but won't result in any downtime in the event of a motherboard component failing (like we faced this time when our mysterious surface soldered VRM failed). So what do you use? What should I use?"

Daily backups

[[e-mail]]

>> we nearly lost all of our data since our last backup (it was only a few days, but an important few)

Daily backups !

general recomendations:

quality server (Dell/HP/etc)
NO ide drives!
SCSI in software raid5
minimum software install (e.g. no compilers)

get second 'devel' server to test/compile software before using on production server
If it is not broken, don't fix it. as in screw with the devel server.

applications

[perlchild]

Depending on the applications you need to have redundant, you might be able to just use a compactpci server, with redundant hardware in it(this technology, while expensive, even allows removal of failed cpus during operation of the machine, it was developed for telecom carriers, and is rather expensive). That would protect you from component failures, but not from power outages without redundant power, nor from os failures.
This is a hard problem(NP-Hard perhaps, I'm not sure), and you need to have a:

List of applications you want to protect

Budgeted amount

What threats you are trying to protect from

What kind of failures you will tolerate(do you need 99.9% uptime? or better? worse?
You could, for simple applications, like web service, bump up a pair of linux machines, gimmick some replication between the two, and hope nothing goes wrong, if you have a very low budget, and you'd probably spend a fair amount of work debugging later on, "synchronisation problems". But for redundant storage. The openssi project [openssi.org] is working on highly-available single-image clusters for linux, in an open source model, they might be your first place to look. It's not however, something for the unprepared to do, nor is it something that I'd recommend if you do other tasks for this company. Maintaining such a beast will require a significant implantation investment. The good news is that once everything works to your satisfaction, you can probably take a 4 week vacation somewhere with golden beaches and much sun, and let it take care of itself. I can't stress this enough, this is a hard problem, if you really want to do this right, you'll want to surround yourself with qualified people with experience in this field, it's non-trivial, and mistakes can lead to severe data-loss.

don't fall into the RAID trap

[zaqattack911]

I've been a system admin for a production webserver for a few years now, and I can tell you this.

99.9% of the time when I've had to retreive data from backup, it was because of human error. I.E. someone deleted something they shouldn't have, or the moved the wrong directory to the wrong place, or an error was made during a software upgrade, etc..

the rest is due to random harware failure which would be a reason for using RAID. But pouring thousands into redundant servers and disks, is overkill for a biz your size.

If someone accidently wipes out a folder or data, your raid disks won't be any help.

Love,
Zaq

Server Clusters and Raid 5

[haplo21112]

If you can do it the best way to handle is Clusters with an external Raid 5 device that is a shared resource between the two(or more) servers.
Set them up with a shared hardware Raid 5 device.
There is only one active Node in the cluster at a time, if that one fails the second one assumes the identity. Works great never fails!
We are a bit larger so we use EMC Symmetrix, however a smaller shop could probably do a low end EMC Clariion CX200 or the like.

An option

[Halvard]

I too have long experience doing small business consulting and in some other areas. One thing you could do is use RAID-1 with a spare drive. That way if you lose one, you aren't screwed. You also could have a couple spare drives in hot-swap carriers. Pull a drive every night and have a duplicate of your server. Fire up the duplicate server and pop in your known good pull and boot if you server fails.

OS dependent, you don't even have to have exactly the same hardware if you use a more generic kernel build and you can list a different NIC for the spare server in the conf file for modules assuming you aren't compiling them into the kernel.

Continue with good backups made to another machine, to tape/CD/hard drive, or off-site. This way, even if your good pulled drive is a little out of date, you can bring to data current in short order.

You don't mention the OS of the server or budget, but I'll assume that since you've got 2 machines per desk time 15, you can afford a spare server. You don't mention OS and that affects cost, but still, if you are doubling up on hardware on desktops, you can afford to do this or most any of the other solutions offered.

Of course, you get what you pay for and if the experience is lacking in house, hire a knowledgeable consultant or company you trust to do it for you.

Re:don't fall into the RAID trap

[MarkGriz]

"But pouring thousands into redundant servers and disks, is overkill for a biz your size."

I think it's a mistake to make a blanket statement that a RAID array overkill for a small business. My company is similar in size (18 employees) and a RAID is absolutely essential for us from a downtime perspective. We simply can't afford to be down becuase a drive crashed.

Sure, backups are essential for the lost/deleted file, but a RAID (or at least a mirrored drive) keeps your server up and running. Not everyone needs that type of reliability, but if you figure the cost of recovering from a failed hard drive (even in a small company), the additional cost of a RAID upfront is well worth the investment.

Re:A few recommendations.

[afidel]

Forget incrementals, if the data is worth backing up its worth backing up correctly. You main expenses will be in the drive/changer and manpower. Tapes are kind of expensive but not as expensive as losing your data, 90% of businesses that suffer a catastrophic loss of data go out of business within 5 years. As for server solutions a pair of 2U Dell's with RAID5 and redundant PSU's can be had for under $10K, unless this is an unprofitable company that is cheap. I have quite a few companies with 35-50 employees with a lot more servers than just 2.

Re:A few recommendations.

[llefler]

If I remember correctly, the survey that I read was 90% of small businesses....

And $10k is a huge investment for a company of 15 employees if they aren't technology based. Most would start to squeal long before you hit $5000. Sometimes you just have to be happy that the 'server' isn't the owner's PC.