When Does Data Backup Become a Full Time Job? 30
nasteric asks: "Myself and 5 co-workers assume a number of responsibilities at my current job, including monitoring our tape backup jobs that run nightly. We do a good job of keeping non work related items (mp3's, wav files, etc.) off our servers, but our users keep eating up disk space very quickly. File storage, along with the fact that we have numerous projects 'going live' that each require one or more servers means more investment in our backup solutions. Fortunately, we have the capital to expand our backup solution (media, drives, autoloaders, software, etc.) but my boss cringes when I suggest hiring a full-time person to handle the backups. Ensuring the integrity of our nightly backups is critical, but my teammates are being spread thin due to the rapid expansion of our company. We really feel the best solution would be dedicated backup person. We currently backup approximately 3.5 terabytes of data and our enterprise expands to over 4 states. Does anybody have any suggestions as to when data backup becomes a full-time position? Are there any resources that specify when a full-time person dedicated to data backup should be hired? It would be nice to have some resources to refer to when proving my point to my boss."
Create a Whole Department! (Score:2, Funny)
The Department of Redundancy Department! (With apologies to Richard Lederer [netcom.com]...)
Obvious? (Score:5, Insightful)
-or-
Backup will be taken seriously right after you loose a good chunk of data.
Then your boss will hire two people. One to do backups and one to replace the guy that cheesed the last one.
Re:Obvious? (Score:2)
- "Then your boss will hire two people. One to do backups and one to replace the guy that cheesed the last one."
The Guy that cheesed the last one would then be the Boss himself ? =)With all due respect... (Score:4, Informative)
Maybe the easier battle to fight is to reduce the level of data backup, and show how it can save the company money. Chart the growth in number of tapes in your backup journal, and see if it is getting out of hand.
Of course, if you ever DO have a problem, you will be fired, but... what are the odds?
automated solutions? (Score:5, Interesting)
There's no way that this kind of thing *can't* be automated with enough work. The amount of billable time spent to get such a system going has to be much less than hiring a full-time backer-upper.
A place I used to work at had some shared drives on the servers - if you wanted your data backed up, you fricking saved it to those shared drives (dedicated folders for each employee & project), or else you didn't get your data backed up. And if something happened to your data that wasn't backed up, it's your problem (and thus, your ass).
Just a few thoughts.
Just a thought.
Re:automated solutions? (Score:5, Informative)
In The Real World(tm), users bitch and moan to their boss, who bitches and moans to his boss, who will then bitch and moan to your boss, who will bitch and moan to you.
When this happens, you _will_ drop what you are doing and restore said file.
My solution to this problem in the past has been simple. Start writing a log, simple spreadsheet, whatever, of how many hours per week you spend working on the backups (changing tapes/checking the logs/restoring). Do the same for your coworkers. If you and each of your coworkers spend 6 or so hours a day, that's about 40 hours, or one full time person. If it ends up being only a few hours a day, it might not be worth hiring a full timer.
Of course, this doesn't take into account intangibles, like how productive you are if you keep getting interrupted every hour to restore something. Or how much you know/don't know about optimizing backup solutions.
Another, perhaps more palatable option, is the rotating shift. Just like a rotating on call schedule, one person is responsible for all backup related issues during a given week. Presuming you have an automated system, this should really be limited to changing tapes, the occasional backup, and checking the logs to make sure things didn't break in the night.
===
-ajb
Re:automated solutions? (Score:2)
> Interesting, but it doesn't work like that in The Real World(tm).
Hmm. I think I smell a Market Opportunity(tm) here... :)
Hiring Temp vs. Full-Timer (Score:1)
Re:automated solutions? (Score:1)
Another, perhaps more palatable option, is the rotating shift. Just like a rotating on call schedule, one person is responsible for all backup related issues during a given week.
Mike "Did you switch the tapes today?"
Steve "Switch the tapes? You're supposed to be doing them this week!"
Mike "Me? This week is Marty's week!"
Marty "My week was last week! It's Dave's turn!"
Dave "What?"
Re:automated solutions? (Score:2, Interesting)
- As others have suggested, outsource it. There are plenty of storage service providers out there who specialize in this sort of thing and they will cost less than a full time employee.
- Keep all non-OS files on a network mounted drive and build the file server(s) using RAID arrays.
- Write a short shell script that tars up your files, scps them to another machine, and deletes any older backups from that other machine. Stick said script in your crontab. Now you don't have to mess with tapes (though you may want to copy the snapshots to tape every few weeks and take them offsite to guard against fire, theft, etc.).
- ...Or if you're in an environment that isn't easily scriptable (e.g., Windows), install a peer to peer backup solution to share backups within your office. HiveCache was mentioned here on Slashdot awhile back and it does this. I've also been working on such a solution myself which is currently in alpha testing [pensamos.com] and runs on Linux and Windows (although I think 3.5 terabytes is a bit more than it could handle at the moment). Even if these solutions don't perfectly match your needs, the cost of having them customized would most likely be less than hiring someone full time. A P2P solution would also address the original problem of growth a little better than the other solutions just because as more machines are added which require backup, those same machines add to a growing pool of storage for where backups are stored (so you don't have to worry about buying a bigger RAID array because you just surpassed X number of machines, for example).
Those are a few ideas on automating the daily plumbing of the backup process. There are also the issues of streamlining the set-up and recovery processes as well, which the original poster hinted at. Perhaps if his company is growing quickly enough, it is the actual introduction of new computers or projects into the backup process and their associated set-up which is dominating the time required for the backup process. It would stand to reason that this would be much harder to automate than the daily maintenance and perhaps this is what the poster wanted to hire somebody for, although if this is the case it would probably be best to just hire temporary help until the need subsides.Cost analysis (Score:5, Insightful)
Re:Cost analysis (Score:2)
The problem is of course that it likely doesn't equate to there being 40hrs worth of work if 1 person handles the backups.
Create a part-time postition (Score:5, Interesting)
Re:Create a part-time postition (Score:2)
Probably not. You would want to share responsibility for backups between the entire team on rotation, so that anyone can do a restore at 3AM without assistance if necessary. Don't forget that the point of the exercise is disaster recovery, no-one sane does backups just for the sheer hell of it.
You could hire a part-timer to come in and change tapes, sure, but not to do anything that requires thinking.
Price out automated backups and charge it back (Score:3, Interesting)
Oh, oh! (Score:5, Funny)
Re:Oh, oh! (Score:2, Funny)
Re:Oh, oh! (Score:1)
Re:Oh, oh! (Score:1)
Per year?
Kinda gives the term "will work for bandwidth" a whole new meaning, huh?
Re:Oh, oh! (Score:1)
1/2 person (Score:3, Informative)
The backups takes about 1/2 of his time. We have a backup guy for him but hs is only used when the other is away.
We do both tape and "DR" types of backups. We are testing a system right now from Avamar that does snapups, etc... and is a disk to disk system - pretty fast and works good for semi static data. Might want to check them out - www.avamar.com. If you got the cash to build a good backup system - it really should only take about a half a person to maintain it.
Hope that helps.
Duke
Re:1/2 person (Score:3, Informative)
Speaking of disk-to-disk, Maxtor's MaxLineII that will be out in a couple months is aimed at the mass archive/backup market.
250 and 320GB ATA hard disks, Rated the same MTTF as SCSI, 3year warantee, $400 MSRP each for the 320GB. 10TB for under $20,000.
For 1 to 10 TB this is a cheap and good solution, combined with rsync/rdiff incremental backup smearshots [slashdot.org] onto either a Linux NAS with 3ware serial ATA or direct attached storage in the form of something from ACNC [acnc.com] or an AXUS ATA-SCSI box.
With the direct attached storage, you could scale it up past 10TB, 4.4TB per 16 disk RAID5 with hot spare, string those together on as many SCSI channels as you need. Each AXUS 16 disk box costs about $6000, 16 of the 320GB disks costs $6400, so 4.4TB will cost about $12,000. Use software RAID0 to tie them into larger volumes if you need to.
Anyway, the potential is there for low maintenence, very cheap, and automated backups using this roll your own solution.
I can't wait until the 320GB disks come out!
Just a couple of quick thoughts (Score:2)
Also, how often do you find you're restoring files? If it's more often than normal, and you have the money to spend (as you say), then what about using a snapshot solution? Keep hourly (up to, say, 8 hours) and daily (3 days?) of snapshots on your volumes and the users can go retrieve their lost file very easily on their own.
Re:Just a couple of quick thoughts (Score:2)
Ah, I remember the good old days when the easiest way to do a backup was to uuencode a compress'd tar, split it into a thousand parts and post all of them to Usenet!
With Your new ($) Backup Solution (Score:1)
Several Problems, Several Solutions (Score:2, Informative)
Depending on your environment (you made no mention of what OS's, where the capacity is concentrated, etc.) you should be able to call one of the b/u software vendors directly (CA, Veritas, Legato, Bakbone, CommVault, etc) or call one of their channel partners and start sniffing for a deal. I work in this sector and man, everyone is slugging it out and dropping their prices to bottom just to win business.
Also consider some storage management software like SRM- It will let you know EXACTLY what you are backing up, who's hogging space (by user and group) and where your capacity is allocated (an mis-allocated)
The short version is this: With some SRM tools and a good automated backup system, you WILL reduce your workload. I know, I set this stuff up for a living.
Disclaimer: I am a tech for a Computer Associates reseller. I am biased towards their products, but will be the first to admit that they sometimes do suck. Hard. Like turbine of an F-16 hard.
If you'd like to, I will give you a format in which to present your Request For Quote (RFQ) to the various vendors.. It will give you an unbiased measuring tool for all that FUD and MarketingSpeak.
slineyp (at) hotmail (dot) com
To put your problem in proportion. (Score:1)
Both companies had a backup team. One was formed by two people, the other by three.
If your team is investing close to 40 hours per week on backups (thus mking you consider to hire somebody for that purpose) I would first look at what tools I am using for backups and if those tools fit the job. One thing that would be telling you if the tools or the understanding of them are not good enough would be if you find yourself doing backup work that has little to do with backup tasks (fixing problems with software or hardware, finding tapes, etc).
If you feel you have that you have the right tools, you may want to invest in training first. People using software to its full potential is more productive.
Another thing that may help is to designate one person responsible for backups in a rota basis (weekly, beweekly) so hecan concentrate fully on this task mainly.
Failling all that, then yes, perhaps you may need somebody. The point I am trying to make is that you have to try other things first.