Large-Scale Video Archiving? 494
BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"
Analog video systems still work (Score:4, Insightful)
Re:Analog video systems still work (Score:2, Informative)
Re:Analog video systems still work (Score:3, Interesting)
As odd as it may sound, good 'ol analog VCR's are the best and cheapest way to go for this sort of thing.
Casinos monitor every gaming table using 1/2 speed VHS tapes manned by operators swap tapes on a regular basis.
If there were an affordable (or even a not-so affordable) digital solution that provided what the original questioner wanted, the casinos would be using it.
Re:Analog video systems still work (Score:3, Interesting)
About 3 years ago I toured the security area of an indian casino, they had 2 vcrs for each recording node (some vcrs recorded 4 cameras)
the operator could cut over to the other vcr instantly to rewind the other tape to review something. people at tables frequently request that the camera be rolled back when a dealer or customer knocks over chips or does something clumsy and/or stupid.
i remember the security dude saying that they hold everything for 90 days, then erase everything except for cheating or other incidents that may require legal action.
Re:Analog video systems still work (Score:3, Insightful)
If you are willing to sacrifice full 30fps, you can use these. The way VHS is laid down on the tape is the data is written diagonally across the width of the tape. Look at your VCR heads -- see how they're cockeyed? that's intentional.
These specialty camera/VCR combos lay down the data from each camera on successive stripes, and can display 9 different cameras at once on a regular TV screen. You can choose a single camera to fill the screen, but at a reduced framerate.
This will reduce your storage needs by a factor of 9, but at the cost of framerate. This wil not be able to be digitized, unless there are specialty capture cards that can handle this.
From your description, it sounds like you're setting this solution up for a casino, or something very similar. If that's the case, you NEED the full framerate.
You are at the fork in the road -- fast, cheap, reliable: pick any two.
Analog is the way to go (Score:5, Insightful)
That'd be a storage nightmare.
I don't think so.
Let's assume one camera per VCR, full 30 fps. That's 3 8-hour tapes per day per camera, 3000 tapes a day from 1000 VCRs. 1000 VCR's should cost you $100,000 and take up one
medium sized room (power and AC will need to be enhanced). 3000 tapes per day shouldn't cost more than $3000, or $1 million per year.
You'll only need a few tape monkeys at any given instant, because they'll be around one tape needing changing every 28 seconds. A days's worth of VCR tapes (assuming we pack them in boxes with NO room to spare and stack the boxes in blocks) will take up about 1.5 cubic meters or 50 cubic feet (based on 1x4x8 inches per tape, my rough estimate). That means for a year's worth of tape you need 550 cubic meters or 20,000 cubic feet, which is 3300 square feet if piled six feet high. 3300 square feet is about the floor-size of one big house.
Question to original: Are you still sure you want to do this? If so you might be best off "spreadking the load around". IE: Don't do it all in one place. There are a million convenience store camera's and vcr systems in the world, but they're not all in one place.
Off-hand I can only think of one thing that would handle 3,000 terrabytes per year, and that's if the half million people using Morpheus donated 6 Gigabytes of space each year to your cause.
easy solution (Score:2, Funny)
[1000+ can easily be 10000+ or 100000000+, but lets be realistic
not for nothing, but is this for either
1. a reality based web-tv show
2. some bizarre web porn thing
3. some actual legitamite venture
4. security issue ?
hope you pull it off.
--donabal
I think not p0rn... (Score:3, Funny)
Casino (Score:3, Insightful)
Also, is keeping all of the footage forever a requirement? Or just some of the footage? I would think you may want to keep the footage for a couple of days or weeks at most. If something requires footage to be kept longterm then you would move that from the harddrive to cdrom or dvd.
This is a job for a cluster of iMac's if I ever saw one
Re:easy solution (Score:2, Interesting)
1. a reality based web-tv show
2. some bizarre web porn thing
3. some actual legitamite venture
4. security issue ?
You need to ask this the week after the anti-terrorism bill made it through the Sentate?
You ARE being watched (but don't worry as long as you don't do anything wrong.)
Re:easy solution (Score:3, Funny)
1000 typewriters, $100000
1000 cameras, $30000 per day
Capturing the moment when one of the monkeys types the complete works on Shakespeare? Priceless.
For a specialized solution (Score:2, Informative)
Re:For a specialized solution (Score:2, Informative)
1) Cheaper!
2) YOU get to manage it!
3) Hella faster
Plus it can store 12TB clustered. Dedicate one system to backups, and one to taking live data, and move the data over from one to the other with snapmirror every 4 hours, and your done.
~=)
Re:For a specialized solution (Score:3, Informative)
Since I thought the problem originally specified usage of 8 TB/day, stored indefinitely, I can't really see how this solution could work, as you would quickly overrun capacity, and I suppose buying new machines every couple of days is not an option.
NetApp too slow (Score:2, Informative)
Comment removed (Score:4, Informative)
Re:For a specialized solution (Score:4, Informative)
Are they looking at normally still settings? (Score:5, Insightful)
Re:Are they looking at normally still settings? (Score:2, Informative)
Trust me, I work on a security system that digitally records between 16-32 cameras in a retail environment (though we do have customers with 60+ cameras). We normally record at 2 fps during activity, and a much lower rate when not. Customers choose image sizes of approximately 10k per image (with 720x243x2 source images). We don't require that the user has tons of storage, so they typically get about a week's worth of video. Backup is very simple, using DAT tapes.
Greg
Re:Are they looking at normally still settings? (Score:2, Insightful)
Re:Are they looking at normally still settings? (Score:2)
300k/sec??? (Score:4, Informative)
You can do all this with a great program called Virtual Dub (http://www186.pair.com/vdub/)
Tape (Score:4, Interesting)
How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)
Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.
Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)
Mark
Re:Tape (Score:2)
Re:Tape (Score:3, Interesting)
-
Faster throughput
- Less mechanical wear on the tape
- higher density on the tape
Dumping to disk first will also mean that you can use off the shelf backup software (like NetBackup, mentioned before). Given the kind of moneyo you're going to be spending, it's probably going to be worth it.If you can do it, alternate between dumping to one disk array and reading from another (better yet, go through three, so you have a bit of buffer) you can get an effective increase in the effeciency of drives if they're not seeking to write and read at the same time. Obviously there will be a real advantage to using RAID.
It would also be to your advantage to have multiple CPUs controlling the tape drives. Each one would have it's own small farm. You should be able to have multiple CPUs feeding the drives on one tape library.
Given that you're going to need tape monkeys feed the tape library, it may be worthwhile to not use a tape library, but I'd suggest that the drives be in at least small tape libraries. The reason for this is that a tape library can read the bar code on the back of each tape as it goes into the drive .. Otherwise you are almost sure to have errors logging which tape was where/when. With the volume of tapes you're going through it could be absolute hell trying to track down a mis-labeled tape. A small tape library would also allow you to keep drives in more constant motion.
The last thing is to make sure that you have more drives than you 'really' need. With many drives in constant use they will break down from time to time. Make sure that you keep that in mind when you design both your hardware and your software. The horrid thing is that, if they're reasonably well built, failure could be clustered (identical drives with similar usage).
For 90MB/sec Super DLT promises 10MB/sec which means you'll need at least 10 drives. I'd budget for 15 drives -- it gives you some reserve capacity, and allowance for things like non- streaming and occasional drive failure (in spite of their lofty promises) and people who want to read the tapes. It's usually easier to get the budget for the extra drives when you start the project, than after you run into the inevitable problems.
Large Scale Storage (Score:5, Informative)
Re:Large Scale Storage (Score:2)
Store half as much (Score:4, Insightful)
Re:Not exactly (Score:2)
Re:Not exactly (Score:2)
Fibre Channel (Score:5, Informative)
Here are some options for FC disk storage:
- Sun T3
- EMC Clariion
- Compaq Storageworks
- HP VA7400 -- my fav
Just to warn you, you're looking at something on the order of 20k/day to operate this setup... now, I'm sure the price would go down QUITE a bit if you're purchasing 8-10TB a day, but even still, it's a huge cost.
I looked at a 10TB solution from the above vendors, and the cheapest I got it was $0.0425/MB!
1000 cameras? (Score:2, Funny)
Viva Las Vegas... (Score:2, Interesting)
A (very) large DLT array. (Score:3, Insightful)
In terms of sizing, assuming you get 6MB/s per DLT drive, you'll need at least 15 drives. Go for 20. This gives you room to do cutovers, and the like. I'd recommend fronting this with a LARGE disk for scratch space (preferably solid state, but if that's not in the budget, a big old SCSI disk'll do.) You'll need a pretty hefty server to handle all this (at least a pair of Sun E450s for redundancy). You'll also chew through at least 200 tapes a day at a native capacity of 40G/tape.
HOWEVER, this is by no means cheap. The virtue of the fact that you're talking about 8 terabytes a day should be a clue to that. The sort of tape archive, tape supply, and tape library you'll need is... vast. You're talking very high-end hardware here. You'll need a good cataloging system, and some serious software to maintain all this. You'll need to keep about 75% of your drives streaming all day every day. Tape costs alone will run to about 10k/day, let alone electricity, storage, maintenance and initial outlay. I'd venture a project like this is probably a $15 million dollar outlay to do it right, with at least 2 full time support staff and budget on the order of $40k/day . But if you've got the money, go for it.
Celestial Solution (Score:4, Funny)
Retrieval, though, can be essentially arbitrarily slow
Oh, so your looking for a storage medium with infinite space but slow retrieval time?
Easy. Free-Space Medium.
Just use an extremely high gain antenna, a ton of power, and the space around us. Transmit the compressed data stream, aimed at a distant planetary body of your choosing. I would reccomend something in the 100 light year range or so. Now, when the waves hit the body and are reflected back to earth, you will have what is essentially a 100 light year long piece of storage.
And when the waves get back to earth, the technology for terrestrial storage will be extremely inexpensive, and the reception equipment will be too.
Arbitrary Retrieval time (wasRe:Celestial Solution (Score:2)
If you are saving that much and care that little about it, you don't need to save that much.
Re:Celestial Solution (Score:2, Funny)
I think a better idea is to aim the signal to intergalactic space, and then when you need the data, just wait for warp drive to be invented so we can fly past the data stream and receive it as it passes by.
Similar problem, lesser scale... (Score:2, Interesting)
Still, it's fun to read about [cmu.edu]
why store all the images? (Score:3, Interesting)
A simpler solution might be to put a motion detector on each camera and have them only record when there is motion. Using your 100:1 image compression, you estimate 8 terabytes/day. I would expect you could get (warning: pulled from thin air) 1/10th of that by ignoring anything that isn't moving.
But then you have a quandry: was there really no motion at camera #469 at 12:30am, or was something just broken?
similar system (slightly OT) (Score:3, Interesting)
Are you sure about those numbers? (Score:3, Interesting)
Are you capturing in digital format? Are you sure that your systems are even capturing at 30fps?
It's unlikely that any digital conversion device(s) would be able to handle the input from 1000+ cameras and then be able to get that data to a central storage location through a local network.... the bandwidth needed for something like that would be incredible (90 MB/s ??), perhaps even requiring it's own seperate gigabit fibre network. Even in a high-end server with several devices connected to it you'd be lucky to capture 20fps.
But if you were able to get it done, storage options would probably consist of some type of RAID array (with a HUGE number of disks to be able to hold 8TB/day).
Storing that much data indefinately would require enormous rooms dedicated to storage devices, which may not be feasible. Storing data for a week or even a month would be a challenge in itself.
Having things in digital format is nice for indexing and fast retrieval, but in this case it may be too costly. Storing data on video tape may not be as fast or convenient, but it's much easier to store 2 twelve-hour tapes per day per camera than it is to set up and maintain 8 Terra-bytes of hard drive space per day.
Re:Are you sure about those numbers? (Score:2, Insightful)
You're thinking too centrally. Each camera (in a setup such as this) is likely distributed. The cameras can likely be grouped into clusters with a local digitizing system for each. These can transmit back to a central server for archival via a standard switched 100MB/s network (since the total aggregate bandwidth is 90MB/s, the local nodes can't reach that.) Assuming 10 cluster distributed homogeneously, you get 9MB/s from client to server. You still have to deal with the aggregate bandwidth at the server, but the network is no longer clogged (most switches have backbones around 12Gbps+). I'd go with either fiber Gig-E to the server, or channel bonded Fast Ethernet (4 channels should do it, yielding 400MB/s local bandwidth at the server.)
Not really. Rooms dedicated to tape archival maybe. But for the most part it could be done with several large tape cabinets and off-site storage for tapes older than about 5 days.
Mini-DV Tape Library (Score:2)
Why set up a high-tech solution for a low-tech problem?
Real-time 100:1 compression? (Score:5, Insightful)
Very interesting problem, with one more very interesting challenge that hasn't been raised yet:
Because the video is streaming in 24/7, you'd have to build a real-time compression system that could handle the 9MB/s and produce a 100:1 ratio. You could perhaps distribute that across multiple machines/CPUs, or build a custom parallel hardware setup to handle the encoding, but at this scale, the overhead of everything might prevent you from reaching the essential criteria of real-time.
Does anyone know what the hardware requirements are for real-time encoding one 640x480 stream? Now, multiply by 1000.
Re:Real-time 100:1 compression? (Score:2, Informative)
Why does this topic seem so familiar? Oh, its what I do for a living! (though on a much smaller scale)
Greg
Re:Real-time 100:1 compression? (Score:2, Insightful)
If each camera or small cluster of cameras had a storage unit with 2x the space needed for one day, then you could store one day and compress the other, then send the data over to central one day late. Of course, I don't know a whole lot about compression, and I'm working under the assumption that real time compression is slower for some reason. If it takes more than a day to compress a days worth anyway, then you still have the same problem.
Re:Real-time 100:1 compression? (Score:2, Funny)
Re:Real-time 100:1 compression? (Score:3, Informative)
Re:Real-time 100:1 compression? (Score:5, Interesting)
At Conservor, we offer a new set of IP-based Storage Area Network services that offer the speed, capacity, and features of traditional Fibre Channel based SANs, but at an order of magnitude less cost. This approach uses Gigabit Ethernet to make disk and/or tape resources appear as locally attached devices at very nearly wire speed.
We have been told by our partners that our IP SAN experience is considerably broader and deeper than even the largest consultants such as EDS, Accenture, and IBM Global Services. We are also experts at storing and managing VERY large datasets - we work routinely with the oilfield service and exploration companies to do exactly this sort of thing with very large 3D seismic datasets. It's not clear from your post, but you may well need some type of content management system, as well, to ensure propoer indexing and speedy retrieval of such a volume of data, and we can do that, too.
Your write speed and capacity requiremeents, while larger than normal, are not a problem, and can be accommodated without having to resort to exotic
technologies. Of course, there aren't enough details to propose even a rough solution from what you've posted, but it sounds like a tape solution. Still, if your retention time is not too long, we might even be able to do it with disk (we can provide high-speed Fibre-Channel storage arrays as little as 2 cents/MB: MUCH LESS than the competition, and capable of RAID 3, which you may want to use if video streaming for playback is much of an issue - we're working with next-gen cable headend guys on this stuff, too.)
As for the tape components, I need much more info to be able to even speculate on your needs. Anything from mid-range high-performance LTO libraries to full-scale mainframe-type 3590 silos may be needed, depending on a number of variables.
All our solutions are also available as complete service offerings, preventing you from having to acquire, own, or maintain any hardware, software, or management staff. In addition, since our fee is entirely for the service, it is a tax-deductible operating expense, which most companies find quite attractive. (We can also make everything look like capital for those companies (Real Estate, etc.) that want to capitalize everything in an effort to boost EBITDA.)
Storage is changing - there's no reason to do things the old way anymore, when there are better and cheaper solutions at hand. Some of the big guys in storage are going to learn this the hard way...
Have you validated the compressed data rates? (Score:3, Informative)
Any video compression algorithms worth using for this kind of application do comparisons from one frame to the next, and only compress the differences (except for occasional reference frames.) Some of them do substantial motion compensation to model the differences, others don't. Many of them let you tweak the frequency of reference frames - is it every 10? Every 100? Do you need the ability to go backwards, or is smooth forward and clunky backwards good enough?
Very few locations actually generate much motion on a 24-hour basis, except for road traffic cameras, and I'd be extremely surprised to see an application need to store those on a long-term basis (as opposed to storing for a week or so in case there are traffic accidents - anything you need longer than that should probably be handled by license-plate recognizers.)
THE answer to your problems (Score:2, Funny)
Now, for data recovery as you said it may be as slow as it gets, something
Throw your stuff to /dev/null and it will be very happy to watch your videos.
What are they filming? (Score:5, Interesting)
Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?
Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?
You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?
Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...
Re:What are they filming? (Score:3, Informative)
Thanks - Rob
Whole lotta TiVO's (Score:3, Funny)
Face Recognition Application (Score:5, Interesting)
Now you can make your own decision about helping him out (or not).
Re:Face Recognition Application (Score:2)
sPh
Re:Face Recognition Application (Score:2)
The fact the he only need to store faces makes the job easier - no need to store video of people milling about in the bacground. I presume his software is smart enough to tell if there is a visible face - he should just discard the video if there are no large faces visible to scan.
Re:Face Recognition Application (Score:5, Informative)
That system rocked. But it has nothing to do with this post.
Rob
Re:Face Recognition Application (Score:3, Insightful)
So then, what IS the system being used for?
Re:Face Recognition Application (Score:2)
I'm sometimes saddened that many
wrong but... (Score:2)
Freenet would be a good idea : store it "everywhere" but "nowhere"...
Or you may put crypt/uuencode it and have Google cache it for you...
Or post bits in (lots of) public fora...
Or all at once:
because your 8TB data per days just seem to be a lot it is obvious you'd even need 10,000 square inch of these [slashdot.org] to store one year of your data...
But now, tell us, what is it for ?
Is it to monitor some people ?
Is it legal ?
(Actually do I mind ?)
OK, let's take your problem the opposite way
How much storage can you afford ?
Just decide which of the nb of cameras, pictures resolution, fps, etc. you'd sacrify to fit in what you'll actually get...
RRDNS, SANArray, DVDWriter (Score:2)
Set up a Round Robin DNS system - and write a small application to run on the servers that makes the DNS point to the next machine when it's full...
Put a DVDWriter on each machine and viola
Storage silos... (Score:5, Insightful)
Storing 8TB/day for a long time with quick access would probably require a tape silo, which is essentially a tape library the size of a small house. StorageTek [stortek.com] is one of the leaders in silos (And might be the only vendor making them these days.), and they make some pretty nice stuff. Their PowderHorn 9310 is a nice model for bulk storage and quick recovery. A downside to the silos is that they do not often handle DLT tapes, which can make it hard to use tapes outside of the library.
If you do not need fast access to the data, and have time to root through tapes for restores, just get a smaller tape library (Anything in the 50-100 tape range from ATL/Quantum [quantumatl.com] Adic [adic.com] or Qualtstar [qualstar.com] running SuperDLT drives controlled by Veritas Netbackup [veritas.com] would give you an easy way to handle all the data. NetBackup has excellent archiving capabilites (IE record data, wipe data from disk.), works on just about any platform out there, scales well, and keeps files in GNUTar format for easy access. As for storing the tapes themselves, if you have a small retention time just keep around a few hundred tapes to cycle through. If you need to store the data for a long time, get a few thousand tapes and a set of nice shelves to keep them on. If you do not have somewhere to store them, Iron Mountain [ironmountain.com] does a great job storing data, I have worked with them before and toured one of their facilities, and I can vouch that they do a great job storing data.
Re:Storage silos... (Score:2)
The perfect application for StorageTek (Disclaimer, I work for and own stock in StorageTek). Leader in tape silos for a reasons. Some of the smaller silos will deal with DLT tapes, but there are other drives we can sell you that are better in some way or anouther.
However you are talking a lot of data. We have bigger customers, but not many, and most don't store data forever. Are you sure you want this stored forever? I think you need to only store most tapes for 3 months, after which you will know which ones can be overwritten. (I'm assuming security, if no crime is reported in 3 months erase the tape) You really need to re-think this store forever idea. Nasa is about the only one who can't say when data is no longer useful.
check out hitachi storage devices (Score:3, Informative)
alex
Seriously non trivial (Score:3, Insightful)
Your going to have to get VERY friendly with your local "Storage Area Network" vendors. What we came up with as a best SHORT term solution was this - Store the video on Video tape or DVD (depending on quality requirements - DVD is NOT broadcast quality), and then use multiple players - things like DVD jukeboxes/tape changers. They can either be manually loaded, or a robot. You then use a cache to store the vidio on a last in/last out basis if you need fast playback (assumption here - the most recently used tapes are most likely to be used again)
Encoding isn't that bad a problem - you just use multiple encoding stations - You say you have 1000 cameras - you're probably going to need better than 1000 encoding stations (don't forget spares) - you batch up 1/2 hour (for example) files and write those out to the SAN when your done - while one station is encoding, the next is recording, and you batch the encoded file up into Near line storage, so you don't NEED real time
Storage is going to take space/money BIG MONEY - your talking about 30 DVDs worth of data/day depending on your robots. Figure 1000s/day
Charlie
Re:Seriously non trivial (Score:2)
If I remember right, it was going to cost a significant fraction of a million/month
Easy (Score:5, Insightful)
Don't Do It.
This is someone either playing a theoretical game (in which case, the answer is "outsource it") or its someone who has no idea what they really want. You have, ultimately, many conflicting specs here.
You may as well ask for a space shuttle that can fly to pluto in two minutes with no fuel.
Any system that is recording a thousand video inputs is unlikely to need 30 fps for 24/7 (I can't think of anything short of national security installations that would even desire to record 30 fps 24/7, and you'd still have trouble justifying 1000 cameras to cover every building in Washington, DC). Not to mention the logistical implications of DELIVERING 1000 full-frame video feeds to a central location -- you could saturate the entire radio spectrum for the eastern seaboard or have to build the largest gigabit LAN ever deployed.
If you have a real question, please ask it, but this is as bad as a pointy-headed boss spouting off insane specs as the "requirements" for a project because he wants to be on the cutting edge.
And BTW, you won't need 300k per frame for a grayscale 640x480 video image (except that you desire insane specs, which point we've already covered). A fine quality image could be stored in 25-50k, even less depending on the real needs (of which this project seems to lack).
Re:Easy (Score:3, Informative)
30 fps for 24/7 is what our customer wants. End of discussion.
Gigabit ethernet is becoming common, and you only need the serious bandwidth for the last few links before the destination. Everything before that can run on 100 Mbit.
I hope this clears some things up.
Re:Easy (Score:4, Insightful)
Hey, I want a lot of things I can't have, either.
Part of your job is to make sure the customer isn't making life harder for himself than it needs to be (at least if you're a good consultant/engineer and not just trying to get the bucks regardless of outcome).
I suspect the end of this particular discussion is going to end up with:
- the customer not getting what they want
or
- the customer spending a hundred million dollars to custom-build a system that in three years will cost ten grand, be available by mail-order, and fit under your desk.
SO, the real answer to your question always has been and still remains to call up someone who has a clue, not slashdot -- we can't spec out custom hardware installations for you. This is not a software problem.
If this really is necessary, then call up a video company and get a VAR in your office to figure out how to build a thousand+ cameras with 100:1 hardware codecs that can transmit video over whatever arbitrary distances to whatever arbitrary equipment you have.
Then get THAT VAR and a storage VAR together and figure out how the hell you can store terabytes a day -- they'll build a nice online/offline disk and tape mixture that will cost enough to fund a third-world country, but it'll work, but you'll probably be able to buy the entire storage company for the budget you'll be spending, so look into building your own company or buying one out in order to save some money.
Then call the contractors to install all the cameras and network links and build a control room with monitoring equipment.
THEN -- figure out how to back up all this data, since its clearly very valuable stuff that you've spent a couple million on already, you don't want to lose it. Have that storage VAR get a system that'll automatically dupe all the tapes before storage for redundant storage.
Re:Easy (Score:3, Insightful)
Your customer could have saved themselves your fee by posting this to Slashdot themselves.
Two questions:
1. How did you get this customer if you have no idea how to meet their needs, and
2. What kind of consultant doesn't work with the customer to refine project goals and requirements?
Your customer has a need, 30fps is not a need, and they belive the only way to satisfy it with full framerate video 24x7. If they are wrong it is your job to show them why and how you can satisfy the need without 30fps.
Gigabit ethernet is becoming common in network cores. Unfortunately GigE is still very expensive, and to go a reasonable distance you need costly single mode fiber. Even 100MB goes much farther over fiber than copper, and I can't imagine your 1000 cameras will be very close together. Just running the fiber to 1000 cameras could easily cost you several million.
If this projecet is going to happen it will probably cost more than 10 million so your customer is wither a government agency or someone with really deep pockets. Either way if you want to get good information here you will need to give more information. If you are worried about disclosing too much you should go talk to some people at vendors like IBM and EMC. They do this kind of thing everyday. They can handle the data storage part and you evidently can handle the video part. Talk to the folks at Cisco about putting it all together. Of course when you go to these vendors they will need more information as well. "That's what the customre wants," doesn't cut the mustard when developing specs for cutting edge projects. Sorry.
Re:Easy (Score:3, Insightful)
Now, I have some idea of what your talking about. Before choosing my current job I was interviewing with Dictaphone as a support tech for their systems. I have also worked with a lot of techs from Vegas (and spent a bit of time working out there). I have also worked in security, including a nice high profile skyscraper that housed a very large banks headquarters. Their system comprised over 600 cameras, with 95% of those pinhole. So I might have some idea of what I'm talking about.
I think several assumptions can be made from your other posts. First, you are going to run this over a Gigabit backplane, probably fibre. Second, the cameras are sending their signal back to a central (or one of several sub-central locations) location. From here you want to archive the data. I am assuming that the problem of getting the digital camera feed from the camera to where the data is processed is already solved. Your probably wanting to convert an existing tape system over to a digital format.
Since this isn't a gov job there are only a few customers it could be. Most skyscrapers don't require this kind of coverage, with a very few notable exceptions. Even so, they typically only archive data to 90 days, 1 year or three years depending on the location being filmed.
It can also be assumed that since you want to record everything, that you don't want motion activated systems and with your frame rate of 30 FPS you are looking for quick subtle movements that would be easily missed with a standard security camera system (filming once every so many seconds). This requirement rules out something like an airport, as they would not need that speed. This leads me to believe you are working for a large casino since they like to keep their data a nice long time (lawsuits etc), and they would need a frame rate of 30 fps for sleight of hand issues and the higher resolution that you listed. Regardless, even if that is the case, there is nothing wrong with submitting an ask slashdot article about your needs.
As for solutions,
Re:Easy (Score:3, Insightful)
As for solutions, you only have a few practical solutions. Certain high end tape jukebox silos (StorageTek) etc can do this. You could also record to DVD with similiar jukebox silos from Dictaphone. I am guessing that there is going to a centralized processing and collection point.
From there you are probably looking for long term storage at an off site location that is accessable from the central location. This is where you can enter the real problem. Jukebox silos are large, and take up a lot of prime (expensive) real estate that you probably don't want occupied for long term storage. This means you need a way to archive the data but still search it from a centralized location (what happened at door #117 on Sept 23rd 2001). This means that you have to have fibre to an offsite location where additional jukeboxes are stored.
Since the speed that you would need makes telco solutions ungodly expensive you make need to lay your own fibre underground to an offsite storage center. You couldn't get the speed you need through solutions like microwave. This also rules out solutions like iron mountain. Make sure your data backbone for this system is completely seperated from normal building IT traffic. Best yet, put in an air gap for this system from the normal IT system, good for security too.
Assuming you lay your own fibre (or already have it as many large financial institutions do), than you would simply add in additional jukeboxes at the other end of the fibre. I also have to assume that any such fibre would have a redundant connection no less than six feet parallel from the primarly line (Norwest Airlines learned that lesson the hard way when their Sonet ring lost an argument to a backhoe - cutting the redundant fibre line right along with the primary).
Storage media that could suit your needs would include tape and dvd solutions. CDr's would be too impractical with the standards you dictated that must be met. Fibre hard drive solutions are a possibility but would be very expensive now, getting cheaper as hard drive capacities grew (assuming the number of cameras and data collection rates stay the same), so might be more favorable from a long term cost analysis than a short term one. Since you want to keep data "forever" this would weigh in more than normal.
Hope this helps
No problem. (Score:2, Redundant)
-russ
Who would need such a system? (Score:5, Funny)
Re:Who would need such a system? (Score:2)
> Security" agency??
Maybe he would like to...
http://slashdot.org/comments.pl?sid=23094&cid=2
Re:Who would need such a system? (Score:3, Interesting)
Yes, or something close to it. See this Slashdot comment [slashdot.org]:
The guy who submitted this story "Bondheadguy" resolves to RobJMcCready@yahoo.com... A quick search on "Rob McCready" yields a University of Toronto grad student (or maybe former grad student now) who is developing hardware based face recognition equipment. Check out This link... [nce.gc.ca]
Now you can make your own decision about helping him out (or not).
An HSM solution is most likely the answer... (Score:3, Informative)
So what HSM means is Hiarchial Storage Management. Basically, when file hits a threshold of time, space or whatever, it will take that file and put it to tape. Then, it will replace the original file with a stub of a file that says 'when this file is needed, it's located here!'
Now, for tape storage, I highly recommend going with LTO as a tape format. You might consider doing SCSI LTO tape drives with a Crossroads 4450 connected to Broccade switches to make a SAN as well. By putting it on a SAN, you'll have the ability to spread around your clusters that you'd be putting in. LTO can spool data at about 10-20 MB/sec. Hence, if you get an STK or IBM storage library with LTO, you can fit around 20 tapes in there, and do 200 MB/sec. Plus, LTO has variable speed when writing to it, so it's better than DLT in that regard. Not to mention LTO's 100 Gig native capacity and a better compression ratio than DLT. (2.0 vs 2.2) Then, it's just a matter of cycling tapes through. If you're honestly talking that high amount of data to keep INDEFINATLY, then you might want to look at STK's Powererhorns, which hold around 2000 tapes. Plus, you can always add another wall of Tapes if you're not getting the throughput you're expecting. Or you could look at some of the larger scale robots out there, but they don't support LTO tape format yet.
By doing the EMC SAN solution to an STK powderhorn, you're looking at an enterprise level solution that will support you for years to come. Course, this comes from someone who's a vendor-neutral consultant with experience with similar technology, so your YMMV. `8r)
Let us know how it goes!
How About.... (Score:5, Funny)
Pros: Extremely high write speed
Cons: Hard to get data back out, but since "Retrieval... can be essentially arbitrarily slow" you've can just re-film whatever it was that you missed. With the money you save on the video gear, you should have a nice little production budget, too...
Consult with a Casino if you can... (Score:5, Insightful)
However, any information I've seen shows them to still be mostly analog capture for any storage, or at least digital-to-analog conversion for storage.
Although they probably won't talk about their security systems, they'd be a great resource.
MadCow.
Re:Consult with a Casino if you can... (Score:2)
As for capturing the data, if going digital is a must, I'd probably spread the capturing out over a number of computers with the data going to a central repository. Actually, you might want to get capture boards with onboard compression (if you don't already have them) to offload a lot of the compression from the CPU, as that's going to be your most CPU-intensive aspect.
Handling the incoming 90MB/sec shouldn't be a problem if you distribute the network load properly and use a fibre backbone.
Your biggest problem is going to be downtime. If, for example, your data server fails, you're looking at a pretty expensive reboot, especially if you don't have a journaling file system (I assume you'd use a journaling file system). Still, the data server goes down and you lose all your feeds. For this reason, going to analog is much safer. It's rare that ALL of the analog recorders would fail at once.
Just my thoughts. The idea of consulting with the casinos is probably the best idea. Actually, the casinos would probably be willing to share this kind of technical information, as it wouldn't provide any information on how to get around their work. What they're less likely to talk about are the types of tricks they look for.
Re:Consult with a Casino if you can... (Score:2, Interesting)
Actually, the main problem with checking with the casinos is that they themselves are trying to move to digital. Untill recently, the technology simply was not available, so they were stuck using analog (and rack after rack of tapes).
You're right about the capture though, you need to either use cards that do the processing or move it to seperate machines (video encoder/decoders).
As for the server downtime, getting a reliable video server, setting up everything correctly, and going from there means very little (if any) downtime. This depends on how much money you are willing to spend; remember many tv stations broadcast from digital signals, and their downtime is minimal.
Big Brother needs you... (Score:2, Insightful)
This exactly the type of thing that could be done, but shouldn't. Stay away from this, we can only hope that any competent people will stay away from this and they will never get it to work.
Sleep with one eye open.......
640x480? (Score:3, Insightful)
And since you said retrieval can be "arbitrarily" slow, I'd look into using VHS videotapes--even if you store compressed digital on them--as a storage medium. They're slow as hell for rerieval, but the media might be cheap, especially compared to the likes of AIT and such.
The first question is, WHY are you doing this? (Score:3, Insightful)
Just a few minor questions.
sPh
Comment removed (Score:3, Funny)
Doesn't scale up very well (Score:3, Insightful)
I mean 1000 cameras is only enough to put one camera into each home in a fairly small community. Most of the solutions I'm seeing posted so far don't scale up very well. What if you need multiple cameras per home? And what to do about large cities? Maybe this should be a seperate Ask Slashdot question?
Easy and nearly free (Score:2)
Roughly 2.21184e+11 bits per second, uncompressed.
Hmmm.
Why not just erect a satellite dish and stream the data into space? The cost of having an alien intelligence beam it back to you would probably be less than conventional storage costs.
I would especially favor this plan if it's some kind of face-recognition software or other privacy-deprivation program.
Another plan would be to hire 1000 people to sit and swap tapes on their home VCRs every few hours - just hire the people who currently plan to make a living stuffing envelopes at home.
Yet, something tells me that you're not doing anything that I would want to encourage...
Cheers,
Jim, hopefully out of your Jurisdiction
It's a data reduction problem (Score:2)
Hire your tape monkeys to run tapes and/or get a disk silo. I suspect you could get away with much less storage than you think by compressing early.
Even better would be motion detection on the compression or camera side so that only the cameras that are sensing movement would be working.
As for your network, you'll need to partition it to meet your bandwidth requirements.
Good luck!
You're gonna have to go for a jukebox of some kind (Score:2)
That being said, remember that a run-of-the mill 16/10/40 CD-Recorder records at up to 140MB/s, which is well over your goal of 90MB/s. High-capacity CD-Rs can go to 800MB, so you'll end up writing 10,000 per day, and you'll want to use about 100 or more cd-rom drives to do the recording (for reasons of robustness, and to lengthen the average MTBF).
BUT it sets you up for an easy swap in later to switch to DVD-R, or the newest 100GB disc (vapor-hardware) technology, though you can't go much cheaper than CD-R, along with storage reliability of 5 years or so in decent circumstances, 10+ years in climate controlled lockers.
I'd imagine making CD-R cassettes that hold 10,000+ CD-Rs which simply get changed once a day (remember - 100 spindles of 100 CD-Rs is not much space, and can easily be handled by standard warehouse automation equipment). As a bonus, you can store them by day. Need a few hours on Oct 29? Just pull out one cassette. You'd have to trade out a CD-Recorder daily as well, but overall you're going to spend a LOT less than any other storage solution out there.
You can't get much cheaper than this, and the tradoffs may well be worth it to you. But it would likely be a fully custom which would take time to design. It might compare favorably to a cassette of 100
-Adam
don't over-centralize (Score:2)
So find a 200 video solution, and duplicate it five times, situating each site in a geographically convienient manner. A bit of coding glue on the back end might let you have a centralized index.
A single mechanism to handle that much traffic is going to run into many problems and probably won't be satisfactory. bandidth, storage, encoding time, etc for a smaller number of cameras is far more tractable for a smaller number of machines.
This also gives you a way to scale. What happens six months later, when now you have 1300 cameras to deal with? add another site. With a good template, your costs become very predictable, you have a built in mechanism for pilot testing (you will learn ALOT from building the first site), and your project time to scale up becomes easier to estimate.
What are the real requirements? (Score:2, Insightful)
You've put some big numbers up there, which all the hardware-heads have been happy to answer for you. But without real information, this is all just big-clock-speed hardware masturbation.
The real question is: what kind of single system/application would produce 24,000 hours of unedited high-quality video per day and storing it until the end of time?
Most respondents seem to assume that you're running a network of security cameras. If so, other posters have indicated that your quality or recording time requirements are probably 1-2 orders of magnitude too high.
If you're producing something where this video is actually going to be watched by people who care about beautiful full-color full-frame-rate production quality, where is your 200 million dollar production staff that will be watching and cataloging this data? And if you can afford that, surely you can afford at least one knowledgable systems engineer who knows how to design a storage system!
My absolute favorite bit of your post: "Reliability should be good enough to not be annoying long term". So how much lossage is "annoying"? Will you save space by randomly culling videos from the previous 7 days? If the whole system breaks down once a month, will that be annoying? If you lose one minute out of each hour, will that be annoying?
This is the sort of problem that can be designed and solved if you've got the need and budget for it. But it's not a turnkey solution. That's why you hire engineers.
This is a large scale face recognition project (Score:2, Offtopic)
This explains his crazy requirements : looks like he is busy thinking about large scale applications of his work.
http://www.nce.gc.ca/en/success/9920/micronet3_e.
Quick thoughts (Score:2)
First off, there are commercial solutions that tape 10:1 onto VHS tape... maybe you could take one of these a step further (custom ASIC dev) and do 100:1 for grayscale.
Now, assuming analog is not the way to go (sounds like you want to stay digital), first you'd probably want to do motion detection. Only capture the stream from a camera that is reading changes. Depending on the application, that can cut your storage space down significantly, especially in off-peak hours.
Second, I'd probably rethink 30 fps. 15 fps is nearly as good, and takes half the storage space. ATM cameras can range from 5 fps to 1/5 fps... and give good enough results in most circumstances.
That combination should take you down to between 1 TB and 100 GB per day (maybe less depending on usage)... which is definitely more manageable. At that point, an HSM is the answer... look at any of the other excellent posts regarding this.
I worked at one point for a major US corporation with similar needs, and they have a data backup center... it's basically a huge warehouse with tape drives, robots to change tapes and cart them around the warehouse, and tons of archive space for the said tapes. Perhaps this would be a good solution for you? As I recall, they spent about $5 MM for the hardware, and about the same for the actual installation (which is terrorist resistant, etc.)
Hope that helps!
I've got it. (Score:3, Funny)
Digital VCR's (Score:2)
ideal solution (Score:2)
Ok, here's what you do: Take all of your signals (don't even bother compressing them... as other people have pointed out, it looks like 9 GB/sec might be hard to compress) and beam them straight out into outer space. This solves both problems - the massive amount of data, and the high bandwidth.
Now, current technlogy offers few options for playback, but since you don't mind waiting for the playback (you did say arbitrarily slow), I'm sure solutions developed in the future will be both feasible and cheap. (if not, just wait a little longer). Here are a few possibilities:
Develop ultra-focused and ultra-sensitive receivers to detect the signal bouncing off of a distant object.
Teleport a big mirror ahead of the signal to bounce it back. Listen for it to return.
Develop a faster-than-light vehicle to pass the signal, and then use future technology to record it on some futuristic ultra-dense medium. (Note: this can be done obeying Einstein's theory of relativity. Space isn't quite a vacuum, so the radio signal is travelling at a speed less than the speed of light in a vacuum. You could evacuate an area of space for your vehicle to travel so that it is less dense than the space the radio signal is travelling through, and thus has a higher speed of light. You'll be able to catch up without exceeding the speed of light in your path, but exceeding the speed of light in the radio signals path).
Wait for either the universe to collapse in on itself, or a black hole to devour both the earth and the radio signal. This should, with any luck, warp space such that the signal hits the earth again.
Direct the signal at a black hole such that it orbits it. The signal will be trapped in orbit. You could send a vehicle to this area to read the signal, and then retransmit it with more energy to escape the black hole. Of course, you'll only get a brief snippet of data before the vehicle is sucked in, so many may be required. (I'm not too good at black hole theory; hope it doesn't show!)
Anyway, there are lots of possibilites!
-- morcheeba
founding member, literalist society.
Re:compression!! (Score:2, Insightful)
Still compression might be the way to go, except not compression of the single frames, but some way to only store each x'th frame, and between such a store, and the next xth, only store differences, and again compress the entire stream (stored frames +differences).
The efficiency of the differences storing should be improved by a preprocessing (to compression) step to try to reduce small variations in color,
iow to make two surfaces match
in color in the digitized picture, if they are equal in color
If one, while designing the difference-detection
algorithm, is able to differ between background and foreground, one could try to further increase
compression, while maintaining quality by using lossy compression for the
background only, while keeping the foreground (e.g. faces, important with security cams) sharp.
Since some security camera's send home nearly static, or a set of static images recurring after
a certain time (moving cams), this should increase compression.
I can't imagine that something like this is not already available, e.g. as a sideproduct of all
the research that went into the DVD/DivX/MPEG
standards.
Re:welll... (Score:2, Funny)
Of course, nobody around here wants to hear that, so why don't you just go with some Linux thing where the user has to type "cat
Re:Archiving on the cheap. (Score:2, Insightful)
heres a solution :
3 Breece Hill (or SpectraLogic OEMed) 2.75TB AIT-2 autoloaders with 2 sony ait-2 drives (66Mb/s throughput per autoloader) each - $30,000 each x 3 = $90,000
3 x Sun Ultra 2's @ $2000 each = $6,000 (yep they can do 66MB/sec and max out the breece hills and they can handle the compression/video..total throughput will be 66MB/sec x 3 = 198MB/s which should be more than enough to handle the video feed.)
AMANDA tape backup software (free download) and Sun solaris + a shell script.
30 AIT-2 50GB catridges (about the size of a cigarette pack) x 3 = 90 catridges (should fit in a small briefcase) per day for 2.75 x 3 = 8.25TB/day @ $3000/day
So..it can be done for $100,000 and running costs at $3000/day (plus 1 guy 10 minutes to swap the catridges from the autoloaders and cycle em at the end of the day)...which is fairly cheap for this setup. A small standalone sony AIT-2 drive in a PC can read back the tar files of the video if necessary. One large room should be able to hold all the tapes for a couple of months or so. still at $90,000 its not exactly cheap.