Slashdot is powered by your submissions, so send in your scoop


Forgot your password?

Large-Scale Video Archiving? 494

BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"
This discussion has been archived. No new comments can be posted.

Large-Scale Video Archiving?

Comments Filter:
  • by jgrider ( 165754 ) on Monday October 29, 2001 @09:36AM (#2492376) Homepage
    Although you are probably looking for a digital solution, don't overlook the solutions that already exist. Security camera VCR's (available at RadioShack et al.) can put 24 hours (or more) of video on a single VHS tape. Get a few VCR's (at $200 each), and a pallet of VHS tapes at Sam's club, and you could record all the video you want!
    • Not at 30 frames per second. They get lots of time on a tape by recording only a few (or maybe only 1) frame per second. The problem specified 30 frames per second.
    • You are right on the money.

      As odd as it may sound, good 'ol analog VCR's are the best and cheapest way to go for this sort of thing.

      Casinos monitor every gaming table using 1/2 speed VHS tapes manned by operators swap tapes on a regular basis.

      If there were an affordable (or even a not-so affordable) digital solution that provided what the original questioner wanted, the casinos would be using it.
      • Prolly depends on the casino.

        About 3 years ago I toured the security area of an indian casino, they had 2 vcrs for each recording node (some vcrs recorded 4 cameras)

        the operator could cut over to the other vcr instantly to rewind the other tape to review something. people at tables frequently request that the camera be rolled back when a dealer or customer knocks over chips or does something clumsy and/or stupid.

        i remember the security dude saying that they hold everything for 90 days, then erase everything except for cheating or other incidents that may require legal action.
    • There are specialty security camera/VCR combinations that are built to handle this kind of storage.

      If you are willing to sacrifice full 30fps, you can use these. The way VHS is laid down on the tape is the data is written diagonally across the width of the tape. Look at your VCR heads -- see how they're cockeyed? that's intentional.

      These specialty camera/VCR combos lay down the data from each camera on successive stripes, and can display 9 different cameras at once on a regular TV screen. You can choose a single camera to fill the screen, but at a reduced framerate.

      This will reduce your storage needs by a factor of 9, but at the cost of framerate. This wil not be able to be digitized, unless there are specialty capture cards that can handle this.

      From your description, it sounds like you're setting this solution up for a casino, or something very similar. If that's the case, you NEED the full framerate.

      You are at the fork in the road -- fast, cheap, reliable: pick any two.

  • by donabal ( 116308 )
    simply get 1000+ computers with as many large SCSI hard drives as possible.

    [1000+ can easily be 10000+ or 100000000+, but lets be realistic :) ]

    not for nothing, but is this for either
    1. a reality based web-tv show
    2. some bizarre web porn thing
    3. some actual legitamite venture
    4. security issue ?

    hope you pull it off.


    • Didn't you notice the "retrieval can be arbitrarily slow" clause ? : )

    • Casino (Score:3, Insightful)

      by Sixty4Bit ( 6131 )
      Obviously this is some sort of security system that watches a large amount of space. So we are talking either a Casino or park of some kind. If not, then these are the people to ask.

      Also, is keeping all of the footage forever a requirement? Or just some of the footage? I would think you may want to keep the footage for a couple of days or weeks at most. If something requires footage to be kept longterm then you would move that from the harddrive to cdrom or dvd.

      This is a job for a cluster of iMac's if I ever saw one :)

    • Re:easy solution (Score:2, Interesting)

      by SnapShot ( 171582 )
      not for nothing, but is this for either
      1. a reality based web-tv show
      2. some bizarre web porn thing
      3. some actual legitamite venture
      4. security issue ?

      You need to ask this the week after the anti-terrorism bill made it through the Sentate?

      You ARE being watched (but don't worry as long as you don't do anything wrong.)
    • 1000 monkeys, $4000000
      1000 typewriters, $100000
      1000 cameras, $30000 per day

      Capturing the moment when one of the monkeys types the complete works on Shakespeare? Priceless.
    • yer nutz.
      1) Cheaper!
      2) YOU get to manage it!
      3) Hella faster

      Plus it can store 12TB clustered. Dedicate one system to backups, and one to taking live data, and move the data over from one to the other with snapmirror every 4 hours, and your done.

      • Plus it can store 12TB clustered.

        Since I thought the problem originally specified usage of 8 TB/day, stored indefinitely, I can't really see how this solution could work, as you would quickly overrun capacity, and I suppose buying new machines every couple of days is not an option.
      • NetApp too slow (Score:2, Informative)

        by Sierran ( 155611 )
        Careful...I've used NetApp boxen, and (I'm assuming you're talking about NAS) they just don't have the sand to handle that kind of load reliably. Granted those that I worked with are now ~18 months old, but we had an Oracle update stream hitting one with a bandwidth of less than 15 MB/s and it started dropping NFS connections. If you try this, be careful to use high speed connections and to multiplex onto as many interfaces as possible!

      • by tentac1e ( 62936 ) on Monday October 29, 2001 @11:05AM (#2492870) Journal
        1) Cheaper!

        You get what you pay for.

        2) YOU get to manage it!

        I have no idea what you mean by this. Celerra + CIFS support == even an windows user can manage their data.

        Do you mean IT management?

        I wonder what IT professionals have to say []. EMC is #1 in IT satisfaction, product reliability, interoperability standards, service-level guarentees, product innovation, and after-sale service. NetApp is in the bottom third of the vendors for seven out of the ten catagories surveyed.

        3) Hella faster

        I don't know what benchmarks you've been seeing. An EMC Celerra loaded with 14 data movers, each connected via fibre channel, is the damned near fastest thing I've ever seen. This guy wants 90 MB a second. That's child's play with one or two data movers.

        If I were this guy, here's what I would do.

        First off, I'd get a Symmetrix for the daily storage. Last I checked (over summer), the maximum storage was 36 terabytes, RAID. If you bought the whole enchillada, you could have quick access to the last 4 days of video at your fingertips.

        Next, get a Celerra to interface your Symmetrix with your SAN. One Celerra with four data movers should be more than enough I'd wager. Use the Celerra to move the data to a long term storage solution (tape).

        And for the record, no, I don't work for EMC.

    • by sql*kitten ( 1359 ) on Monday October 29, 2001 @11:18AM (#2492923)
      NASA also thought about this [], all the way up to Petabytes.
  • by RobKow ( 1787 ) on Monday October 29, 2001 @09:40AM (#2492404)
    Like a security camera in a stairwell or something? In that case you can use motion detection to start/stop recording and save well over 100:1. The choice of video codec is going to be important if it's for security (so faces, etc. can be recognised), but if not, you can crank the compression ratio up quite high on most codecs, especially the video codecs that do frame-by-frame motion differencing (i.e. not MJPEG).
    • And to make it even better, drop the requirement that the video has to be 30 fps. For retrieved video, 5 fps plays great if you are looking at movement of people. 15 fps is great if you are looking at movement of cars and such.

      Trust me, I work on a security system that digitally records between 16-32 cameras in a retail environment (though we do have customers with 60+ cameras). We normally record at 2 fps during activity, and a much lower rate when not. Customers choose image sizes of approximately 10k per image (with 720x243x2 source images). We don't require that the user has tons of storage, so they typically get about a week's worth of video. Backup is very simple, using DAT tapes.

  • 300k/sec??? (Score:4, Informative)

    by Foss ( 248146 ) <`moc.ssoftae' `ta' `ssof'> on Monday October 29, 2001 @09:41AM (#2492411) Homepage Journal
    300k/sec seems very excessive. You could try converting it all to mpeg4 with a DivX encoder ( and that should compress it right down. If you've got sound in there too, strip it out or at least convert to MP3.

    You can do all this with a great program called Virtual Dub (
  • Tape (Score:4, Interesting)

    by mfarver ( 43681 ) on Monday October 29, 2001 @09:42AM (#2492419) Journal
    Gonna be expensive,

    How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)

    Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.

    Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)

    • Oh, can I be one of the tape monkeys?
  • Large Scale Storage (Score:5, Informative)

    by follower ( 110336 ) on Monday October 29, 2001 @09:43AM (#2492426) Homepage
    What your really looking at is some kind of Heirarchical Storage Solution. What happens is that once you have predetermined how much data will be saved from the camera each night. You can get some kind of disk array to store it on. That disk array will also be attached to some kind of HSM solutions such as what is provided by StorageTek's SAMFs. That solution will automatically backup the data that is stored on your disk and remove it from your disk so new data can be stored on the costly disks. From now on your OS and applications think that the data is on disk but in reality its on tape. When the data is requested the software will automatically get it from tape and place it back on the disk. This can be rather costly however.
  • Store half as much (Score:4, Insightful)

    by Terry Cumming ( 200255 ) on Monday October 29, 2001 @09:43AM (#2492428) Homepage
    Be pragmatic and only archive 15fps. This cuts your archive media costs by ~50% no matter what solution you choose. 15fps should be adequate, although who knows your exact parameters.
  • Fibre Channel (Score:5, Informative)

    by Erk ( 17215 ) on Monday October 29, 2001 @09:47AM (#2492456)
    Go with a FC solution - stay away from EMC, as they will try to sell you a massive Symmetrix for your needs. Sounds like you need a building block approach, one block a day. Doesn't need to be TOO fancy, eh?

    Here are some options for FC disk storage:

    - Sun T3
    - EMC Clariion
    - Compaq Storageworks
    - HP VA7400 -- my fav

    Just to warn you, you're looking at something on the order of 20k/day to operate this setup... now, I'm sure the price would go down QUITE a bit if you're purchasing 8-10TB a day, but even still, it's a huge cost.

    I looked at a 10TB solution from the above vendors, and the cheapest I got it was $0.0425/MB!
  • That's alot of simultaneous pr0n recording...
  • The first solution I would look at is how they do this at the casinos in Vegas (or anywhere else). Everytime I see one of those 'On the Inside' shows about casinos on TLC/Discovery Channel, they are always boasting about their video camera capabilities, and their ability to archive everything that happens. Whatever solution you find there, while not always the most cost effective, has definately been tested in an environment on the scale of 100's, if not 1000's of cameras.
  • by j.e.hahn ( 1014 ) on Monday October 29, 2001 @09:49AM (#2492470)
    A suitably large DLT library with a fairly large number of drives would probably do this. Couple it with some HSM (Hierarchical Storage Management software) and you're probably all set.

    In terms of sizing, assuming you get 6MB/s per DLT drive, you'll need at least 15 drives. Go for 20. This gives you room to do cutovers, and the like. I'd recommend fronting this with a LARGE disk for scratch space (preferably solid state, but if that's not in the budget, a big old SCSI disk'll do.) You'll need a pretty hefty server to handle all this (at least a pair of Sun E450s for redundancy). You'll also chew through at least 200 tapes a day at a native capacity of 40G/tape.

    HOWEVER, this is by no means cheap. The virtue of the fact that you're talking about 8 terabytes a day should be a clue to that. The sort of tape archive, tape supply, and tape library you'll need is... vast. You're talking very high-end hardware here. You'll need a good cataloging system, and some serious software to maintain all this. You'll need to keep about 75% of your drives streaming all day every day. Tape costs alone will run to about 10k/day, let alone electricity, storage, maintenance and initial outlay. I'd venture a project like this is probably a $15 million dollar outlay to do it right, with at least 2 full time support staff and budget on the order of $40k/day . But if you've got the money, go for it.
  • by kc0dby ( 522118 ) on Monday October 29, 2001 @09:50AM (#2492478) Homepage
    and you have to store it indefinitely.

    Retrieval, though, can be essentially arbitrarily slow

    Oh, so your looking for a storage medium with infinite space but slow retrieval time?

    Easy. Free-Space Medium.

    Just use an extremely high gain antenna, a ton of power, and the space around us. Transmit the compressed data stream, aimed at a distant planetary body of your choosing. I would reccomend something in the 100 light year range or so. Now, when the waves hit the body and are reflected back to earth, you will have what is essentially a 100 light year long piece of storage.

    And when the waves get back to earth, the technology for terrestrial storage will be extremely inexpensive, and the reception equipment will be too.
    • Might I ask why your requirements include arbitrary retrieval time? I would put it to you that if you have arbitrary retrieval time, you DON'T have to save everything. Are you saying that if I built you a system that saved everything at your highest bandwidth constraints, but took a year to do a retrieval, that is acceptable? I seriously doubt it.

      If you are saving that much and care that little about it, you don't need to save that much.
  • At CMU, there's work being done with ~60 cameras capturing synchronized data in real-time to reconstruct 3-d mapped images of arbitrary scenes. As far as I know, they only capture about 10-seconds at a time, though.

    Still, it's fun to read about []

  • by bpowell423 ( 208542 ) on Monday October 29, 2001 @09:52AM (#2492492)
    Are we presuming that there will be action most of the time in all 1000+ cameras? If not, then why store the images where there is no action? I'm doing a similar digital surveillance thing, albeit much smaller. (7 cameras, 1 fps, only at night [when there's not much action anyway]) My images all go to jpeg's and I wrote a little C program to throw away all the "similar" images. My algorithm is somewhat conveluded, but more-or-less only keeps the images if they differ by more than a certain amount. I'm sure video compression schemes like mpeg would pretty much take care of this if you're storing to video format. Storing to jpegs has the benefit of being able to easily pick out any time stamp, but I can't watch them like a video.

    A simpler solution might be to put a motion detector on each camera and have them only record when there is motion. Using your 100:1 image compression, you estimate 8 terabytes/day. I would expect you could get (warning: pulled from thin air) 1/10th of that by ignoring anything that isn't moving.

    But then you have a quandry: was there really no motion at camera #469 at 12:30am, or was something just broken?
    • Recently at a trade conference (Crestech []) I saw a demo of a system some grad students had put together which paired a panoramic low-rez camera with a moveable hi-res camera. The panoramic would observe an area, and software would recognize areas of activity and focus the other camera on it. But this isn't really what the Ask Slashdot'er was looking for, obviously.
  • by Rackemup ( 160230 ) on Monday October 29, 2001 @09:52AM (#2492494) Homepage
    Those numbers seem a little high to me... and I have a few questions. (My company does video-capture and storage security systems so i know these issues do come up)

    Are you capturing in digital format? Are you sure that your systems are even capturing at 30fps?

    It's unlikely that any digital conversion device(s) would be able to handle the input from 1000+ cameras and then be able to get that data to a central storage location through a local network.... the bandwidth needed for something like that would be incredible (90 MB/s ??), perhaps even requiring it's own seperate gigabit fibre network. Even in a high-end server with several devices connected to it you'd be lucky to capture 20fps.

    But if you were able to get it done, storage options would probably consist of some type of RAID array (with a HUGE number of disks to be able to hold 8TB/day).

    Storing that much data indefinately would require enormous rooms dedicated to storage devices, which may not be feasible. Storing data for a week or even a month would be a challenge in itself.

    Having things in digital format is nice for indexing and fast retrieval, but in this case it may be too costly. Storing data on video tape may not be as fast or convenient, but it's much easier to store 2 twelve-hour tapes per day per camera than it is to set up and maintain 8 Terra-bytes of hard drive space per day.

    • It's unlikely that any digital conversion device(s) would be able to handle the input from 1000+ cameras and then be able to get that data to a central storage location through a local network.... the bandwidth needed for something like that would be incredible (90 MB/s ??), perhaps even requiring it's own seperate gigabit fibre network. Even in a high-end server with several devices connected to it you'd be lucky to capture 20fps.

      You're thinking too centrally. Each camera (in a setup such as this) is likely distributed. The cameras can likely be grouped into clusters with a local digitizing system for each. These can transmit back to a central server for archival via a standard switched 100MB/s network (since the total aggregate bandwidth is 90MB/s, the local nodes can't reach that.) Assuming 10 cluster distributed homogeneously, you get 9MB/s from client to server. You still have to deal with the aggregate bandwidth at the server, but the network is no longer clogged (most switches have backbones around 12Gbps+). I'd go with either fiber Gig-E to the server, or channel bonded Fast Ethernet (4 channels should do it, yielding 400MB/s local bandwidth at the server.)

      But if you were able to get it done, storage options would probably consist of some type of RAID array (with a HUGE number of disks to be able to hold 8TB/day).

      Storing that much data indefinately would require enormous rooms dedicated to storage devices, which may not be feasible. Storing data for a week or even a month would be a challenge in itself.

      Not really. Rooms dedicated to tape archival maybe. But for the most part it could be done with several large tape cabinets and off-site storage for tapes older than about 5 days.

  • Why not do what TV stations and big production companies do to store large quantities of footage: make a big library of digital recordings. I know the restrictions are 640x480x16, but why not go with 720x480x32 instead? Because its digital, you can get lossless recall through firewire in any halfway decent video camera, and backups to hard drives for compression, editing and analysis is pretty easy.

    Why set up a high-tech solution for a low-tech problem?
  • by image ( 13487 ) on Monday October 29, 2001 @09:55AM (#2492504) Homepage
    100:1 video compression brings that down to ~90 KB/s.

    Very interesting problem, with one more very interesting challenge that hasn't been raised yet:

    Because the video is streaming in 24/7, you'd have to build a real-time compression system that could handle the 9MB/s and produce a 100:1 ratio. You could perhaps distribute that across multiple machines/CPUs, or build a custom parallel hardware setup to handle the encoding, but at this scale, the overhead of everything might prevent you from reaching the essential criteria of real-time.

    Does anyone know what the hardware requirements are for real-time encoding one 640x480 stream? Now, multiply by 1000.
    • Easy, compress the stream AT the camera, or at least, off the server. That way the data transmission is already down to its lowest point. And yes, for a scalable solution the system should be broken up across multiple compression units/servers.

      Why does this topic seem so familiar? Oh, its what I do for a living! (though on a much smaller scale)

    • You wouldn't necessarily need real time compression.

      If each camera or small cluster of cameras had a storage unit with 2x the space needed for one day, then you could store one day and compress the other, then send the data over to central one day late. Of course, I don't know a whole lot about compression, and I'm working under the assumption that real time compression is slower for some reason. If it takes more than a day to compress a days worth anyway, then you still have the same problem.
    • Not a problem. Compression is in-camera. So the data stream out of the back of the camera is already compressed. Rob
      • by dublin ( 31215 ) on Monday October 29, 2001 @01:44PM (#2493628) Homepage
        Disclosure: I am VP/GM of Conservor, whose product & service offerings are discussed in this post.

        At Conservor, we offer a new set of IP-based Storage Area Network services that offer the speed, capacity, and features of traditional Fibre Channel based SANs, but at an order of magnitude less cost. This approach uses Gigabit Ethernet to make disk and/or tape resources appear as locally attached devices at very nearly wire speed.

        We have been told by our partners that our IP SAN experience is considerably broader and deeper than even the largest consultants such as EDS, Accenture, and IBM Global Services. We are also experts at storing and managing VERY large datasets - we work routinely with the oilfield service and exploration companies to do exactly this sort of thing with very large 3D seismic datasets. It's not clear from your post, but you may well need some type of content management system, as well, to ensure propoer indexing and speedy retrieval of such a volume of data, and we can do that, too.

        Your write speed and capacity requiremeents, while larger than normal, are not a problem, and can be accommodated without having to resort to exotic
        technologies. Of course, there aren't enough details to propose even a rough solution from what you've posted, but it sounds like a tape solution. Still, if your retention time is not too long, we might even be able to do it with disk (we can provide high-speed Fibre-Channel storage arrays as little as 2 cents/MB: MUCH LESS than the competition, and capable of RAID 3, which you may want to use if video streaming for playback is much of an issue - we're working with next-gen cable headend guys on this stuff, too.)

        As for the tape components, I need much more info to be able to even speculate on your needs. Anything from mid-range high-performance LTO libraries to full-scale mainframe-type 3590 silos may be needed, depending on a number of variables.

        All our solutions are also available as complete service offerings, preventing you from having to acquire, own, or maintain any hardware, software, or management staff. In addition, since our fee is entirely for the service, it is a tax-deductible operating expense, which most companies find quite attractive. (We can also make everything look like capital for those companies (Real Estate, etc.) that want to capitalize everything in an effort to boost EBITDA.)

        Storage is changing - there's no reason to do things the old way anymore, when there are better and cheaper solutions at hand. Some of the big guys in storage are going to learn this the hard way...
      • Your big driver here is the compressed data rates - it's a large enough project that being off by 2:1, or even by 10%, can be worth serious money. Have you validated that the estimates you're using are correct? Obviously the 640x480 x Colordepth x Frames/Sec x NumberOfCameras are correct, but the compression ratios are a big variable, and it's potentially worth spending more money on compression to improve the ratios. Have you run tests for typical locations to see what the real data rates are, and evaluated whether you can improve them?

        Any video compression algorithms worth using for this kind of application do comparisons from one frame to the next, and only compress the differences (except for occasional reference frames.) Some of them do substantial motion compensation to model the differences, others don't. Many of them let you tweak the frequency of reference frames - is it every 10? Every 100? Do you need the ability to go backwards, or is smooth forward and clunky backwards good enough?
        Very few locations actually generate much motion on a 24-hour basis, except for road traffic cameras, and I'd be extremely surprised to see an application need to store those on a long-term basis (as opposed to storing for a week or so in case there are traffic accidents - anything you need longer than that should probably be handled by license-plate recognizers.)

  • I think /dev/null will be very happy to accept as much Terabytes you want. Write speed shall be no problem at all.
    Now, for data recovery as you said it may be as slow as it gets, something /dev/null certainly is.

    Throw your stuff to /dev/null and it will be very happy to watch your videos.

  • by KFury ( 19522 ) on Monday October 29, 2001 @09:57AM (#2492516) Homepage
    If things aren't actually moving in most of the shots (ATM or warehouse surveillance cams, for example) then you'll be able to get far better than 100x video compression.

    Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?

    Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?

    You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?

    Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...
    • All of the various requirements were compiled into the post :). No, we can't expect to have largely static images. The most we'd want to store along with the video is timestamps. Any other data that results from processing the video stream will be dealt with separately...this is just for archiving the raw video. I'm not sure what the break-even point is on speeding up retrieval. Assume for the moment that we don't want to spend more than 5% of the total cost on speeding up retrieval.

      Thanks - Rob
  • by Ukab the Great ( 87152 ) on Monday October 29, 2001 @09:58AM (#2492523)
    Get a thousand TiVO's. Why settle for AVI quality when you can see your terrorists and burglars in stunning MPEG-2?
  • by cwhittenburg ( 532699 ) on Monday October 29, 2001 @09:58AM (#2492524)
    The guy who submitted this story "Bondheadguy" resolves to A quick search on "Rob McCready" yields a University of Toronto grad student (or maybe former grad student now) who is developing hardware based face recognition equipment. Check out This link... []

    Now you can make your own decision about helping him out (or not).

    • The guy who submitted this story "Bondheadguy" resolves to A quick search on "Rob McCready" yields a University of Toronto grad student (or maybe former grad student now) who is developing hardware based face recognition equipment. Check out This link... []

      Now you can make your own decision about helping him out (or not).
      I see no reason why this should be modded "Offtopic". The question of intent is most certainly a key aspect of this topic!


      • Bondheadguy should have told us his intentions, as it would make his problem easier:

        The fact the he only need to store faces makes the job easier - no need to store video of people milling about in the bacground. I presume his software is smart enough to tell if there is a visible face - he should just discard the video if there are no large faces visible to scan.
    • by BondHeadGuy ( 529371 ) <`robertmccready' `at' `'> on Monday October 29, 2001 @11:29AM (#2492964) Homepage
      Actually, I developed (past tense) a hardware-based object detection system. Faces are interesting objects to try and learn to detect, because they have intresting and considerable variation. They also make for a fun demo and get more press. But I could have been detecting anything. Recognition, on the other hand, is different; it is the detection of individual faces rather than faces in general, and it would be nearly impossible to put entirely into a hardware system because of the large face database required.

      That system rocked. But it has nothing to do with this post.

    • The true solution to avoiding applications of technologies we don't like is to ignore and discourage the development of the technologies that make said (unliked) application possible. Then we will all be happy.

      I'm sometimes saddened that many /. readers are libertarians first and geeks second.
  • I once discussed with some colleagues about the possibility to store data indefinitely by sending them around on the Internet...

    Freenet would be a good idea : store it "everywhere" but "nowhere"...

    Or you may put crypt/uuencode it and have Google cache it for you...

    Or post bits in (lots of) public fora...

    Or all at once:

    because your 8TB data per days just seem to be a lot it is obvious you'd even need 10,000 square inch of these [] to store one year of your data...

    But now, tell us, what is it for ?

    Is it to monitor some people ?

    Is it legal ?

    (Actually do I mind ?)

    OK, let's take your problem the opposite way :

    How much storage can you afford ?

    Just decide which of the nb of cameras, pictures resolution, fps, etc. you'd sacrify to fit in what you'll actually get...
  • 10 machines with SANArray FFx-2 [] from Mylex - that's up to 9TB per machine... It writes up to 270MB/s...

    Set up a Round Robin DNS system - and write a small application to run on the servers that makes the DNS point to the next machine when it's full...

    Put a DVDWriter on each machine and viola ;)
  • Storage silos... (Score:5, Insightful)

    by supabeast! ( 84658 ) on Monday October 29, 2001 @10:01AM (#2492541)
    Since you did not state a retrieval time or storage/retention needs, I am going to offer to scenarios; one for long term, fast access storage, one for short term and/or slow access storage.

    Storing 8TB/day for a long time with quick access would probably require a tape silo, which is essentially a tape library the size of a small house. StorageTek [] is one of the leaders in silos (And might be the only vendor making them these days.), and they make some pretty nice stuff. Their PowderHorn 9310 is a nice model for bulk storage and quick recovery. A downside to the silos is that they do not often handle DLT tapes, which can make it hard to use tapes outside of the library.

    If you do not need fast access to the data, and have time to root through tapes for restores, just get a smaller tape library (Anything in the 50-100 tape range from ATL/Quantum [] Adic [] or Qualtstar [] running SuperDLT drives controlled by Veritas Netbackup [] would give you an easy way to handle all the data. NetBackup has excellent archiving capabilites (IE record data, wipe data from disk.), works on just about any platform out there, scales well, and keeps files in GNUTar format for easy access. As for storing the tapes themselves, if you have a small retention time just keep around a few hundred tapes to cycle through. If you need to store the data for a long time, get a few thousand tapes and a set of nice shelves to keep them on. If you do not have somewhere to store them, Iron Mountain [] does a great job storing data, I have worked with them before and toured one of their facilities, and I can vouch that they do a great job storing data.
    • The perfect application for StorageTek (Disclaimer, I work for and own stock in StorageTek). Leader in tape silos for a reasons. Some of the smaller silos will deal with DLT tapes, but there are other drives we can sell you that are better in some way or anouther.

      However you are talking a lot of data. We have bigger customers, but not many, and most don't store data forever. Are you sure you want this stored forever? I think you need to only store most tapes for 3 months, after which you will know which ones can be overwritten. (I'm assuming security, if no crime is reported in 3 months erase the tape) You really need to re-think this store forever idea. Nasa is about the only one who can't say when data is no longer useful.

  • by fluxs ( 194501 ) on Monday October 29, 2001 @10:02AM (#2492548)
    hitachi has several very large storage arrays that are very competitive with EMC last i checked. again, that is if you need it to be in digial format and need it to be online.

  • by CharlieG ( 34950 ) on Monday October 29, 2001 @10:03AM (#2492554) Homepage
    I've looked into almost this exact problem (we had about 100 hours of full color video/day - broadcast quality)

    Your going to have to get VERY friendly with your local "Storage Area Network" vendors. What we came up with as a best SHORT term solution was this - Store the video on Video tape or DVD (depending on quality requirements - DVD is NOT broadcast quality), and then use multiple players - things like DVD jukeboxes/tape changers. They can either be manually loaded, or a robot. You then use a cache to store the vidio on a last in/last out basis if you need fast playback (assumption here - the most recently used tapes are most likely to be used again)

    Encoding isn't that bad a problem - you just use multiple encoding stations - You say you have 1000 cameras - you're probably going to need better than 1000 encoding stations (don't forget spares) - you batch up 1/2 hour (for example) files and write those out to the SAN when your done - while one station is encoding, the next is recording, and you batch the encoded file up into Near line storage, so you don't NEED real time

    Storage is going to take space/money BIG MONEY - your talking about 30 DVDs worth of data/day depending on your robots. Figure 1000s/day

  • Easy (Score:5, Insightful)

    by NMerriam ( 15122 ) <> on Monday October 29, 2001 @10:11AM (#2492586) Homepage
    This is actually a pretty easy question to answer:

    Don't Do It.

    This is someone either playing a theoretical game (in which case, the answer is "outsource it") or its someone who has no idea what they really want. You have, ultimately, many conflicting specs here.

    You may as well ask for a space shuttle that can fly to pluto in two minutes with no fuel.

    Any system that is recording a thousand video inputs is unlikely to need 30 fps for 24/7 (I can't think of anything short of national security installations that would even desire to record 30 fps 24/7, and you'd still have trouble justifying 1000 cameras to cover every building in Washington, DC). Not to mention the logistical implications of DELIVERING 1000 full-frame video feeds to a central location -- you could saturate the entire radio spectrum for the eastern seaboard or have to build the largest gigabit LAN ever deployed.

    If you have a real question, please ask it, but this is as bad as a pointy-headed boss spouting off insane specs as the "requirements" for a project because he wants to be on the cutting edge.

    And BTW, you won't need 300k per frame for a grayscale 640x480 video image (except that you desire insane specs, which point we've already covered). A fine quality image could be stored in 25-50k, even less depending on the real needs (of which this project seems to lack).
    • Re:Easy (Score:3, Informative)

      by BondHeadGuy ( 529371 )
      Apparently you didn't actually read my post. I said I was assuming 100:1 compression on the video stream. That brings each frame down to 3K, which is much better than 25-50K.

      30 fps for 24/7 is what our customer wants. End of discussion.

      Gigabit ethernet is becoming common, and you only need the serious bandwidth for the last few links before the destination. Everything before that can run on 100 Mbit.

      I hope this clears some things up.
      • Re:Easy (Score:4, Insightful)

        by NMerriam ( 15122 ) <> on Monday October 29, 2001 @12:18PM (#2493209) Homepage
        30 fps for 24/7 is what our customer wants. End of discussion.

        Hey, I want a lot of things I can't have, either.

        Part of your job is to make sure the customer isn't making life harder for himself than it needs to be (at least if you're a good consultant/engineer and not just trying to get the bucks regardless of outcome).

        I suspect the end of this particular discussion is going to end up with:

        - the customer not getting what they want
        - the customer spending a hundred million dollars to custom-build a system that in three years will cost ten grand, be available by mail-order, and fit under your desk.

        SO, the real answer to your question always has been and still remains to call up someone who has a clue, not slashdot -- we can't spec out custom hardware installations for you. This is not a software problem.

        If this really is necessary, then call up a video company and get a VAR in your office to figure out how to build a thousand+ cameras with 100:1 hardware codecs that can transmit video over whatever arbitrary distances to whatever arbitrary equipment you have.

        Then get THAT VAR and a storage VAR together and figure out how the hell you can store terabytes a day -- they'll build a nice online/offline disk and tape mixture that will cost enough to fund a third-world country, but it'll work, but you'll probably be able to buy the entire storage company for the budget you'll be spending, so look into building your own company or buying one out in order to save some money.

        Then call the contractors to install all the cameras and network links and build a control room with monitoring equipment.

        THEN -- figure out how to back up all this data, since its clearly very valuable stuff that you've spent a couple million on already, you don't want to lose it. Have that storage VAR get a system that'll automatically dupe all the tapes before storage for redundant storage.
      • Re:Easy (Score:3, Insightful)

        by monkeydo ( 173558 )
        30 fps for 24/7 is what our customer wants. End of discussion.

        Your customer could have saved themselves your fee by posting this to Slashdot themselves.

        Two questions:
        1. How did you get this customer if you have no idea how to meet their needs, and
        2. What kind of consultant doesn't work with the customer to refine project goals and requirements?

        Your customer has a need, 30fps is not a need, and they belive the only way to satisfy it with full framerate video 24x7. If they are wrong it is your job to show them why and how you can satisfy the need without 30fps.

        Gigabit ethernet is becoming common in network cores. Unfortunately GigE is still very expensive, and to go a reasonable distance you need costly single mode fiber. Even 100MB goes much farther over fiber than copper, and I can't imagine your 1000 cameras will be very close together. Just running the fiber to 1000 cameras could easily cost you several million.

        If this projecet is going to happen it will probably cost more than 10 million so your customer is wither a government agency or someone with really deep pockets. Either way if you want to get good information here you will need to give more information. If you are worried about disclosing too much you should go talk to some people at vendors like IBM and EMC. They do this kind of thing everyday. They can handle the data storage part and you evidently can handle the video part. Talk to the folks at Cisco about putting it all together. Of course when you go to these vendors they will need more information as well. "That's what the customre wants," doesn't cut the mustard when developing specs for cutting edge projects. Sorry.

  • No problem. (Score:2, Redundant)

    by Russ Nelson ( 33911 )
    No problem. Just broadcast the video streams into outer space on 1,000 different channels of TV. When you want to fetch that video, just go the appropriate number of light years away from earth and start watching.
  • by TheSHAD0W ( 258774 ) on Monday October 29, 2001 @10:12AM (#2492595) Homepage
    Do you work for that new "Homeland Security" agency??
  • by GoNINzo ( 32266 ) <GoNINzo&yahoo,com> on Monday October 29, 2001 @10:12AM (#2492596) Journal
    You'd have these cameras dump your choice in disk arrays. I might suggest an EMC Symmetrix as they are particularly reliable and very fast. also, they support the sizes of data you're talking about, and even have a specific unit designed for video. The nice thing is if it's on disk, then your retrieval time is going to be very quick. then you get a product such as NetBackup's TSM or ADSM's HSM solution. Hook that to a library, done.

    So what HSM means is Hiarchial Storage Management. Basically, when file hits a threshold of time, space or whatever, it will take that file and put it to tape. Then, it will replace the original file with a stub of a file that says 'when this file is needed, it's located here!'

    Now, for tape storage, I highly recommend going with LTO as a tape format. You might consider doing SCSI LTO tape drives with a Crossroads 4450 connected to Broccade switches to make a SAN as well. By putting it on a SAN, you'll have the ability to spread around your clusters that you'd be putting in. LTO can spool data at about 10-20 MB/sec. Hence, if you get an STK or IBM storage library with LTO, you can fit around 20 tapes in there, and do 200 MB/sec. Plus, LTO has variable speed when writing to it, so it's better than DLT in that regard. Not to mention LTO's 100 Gig native capacity and a better compression ratio than DLT. (2.0 vs 2.2) Then, it's just a matter of cycling tapes through. If you're honestly talking that high amount of data to keep INDEFINATLY, then you might want to look at STK's Powererhorns, which hold around 2000 tapes. Plus, you can always add another wall of Tapes if you're not getting the throughput you're expecting. Or you could look at some of the larger scale robots out there, but they don't support LTO tape format yet.

    By doing the EMC SAN solution to an STK powderhorn, you're looking at an enterprise level solution that will support you for years to come. Course, this comes from someone who's a vendor-neutral consultant with experience with similar technology, so your YMMV. `8r)

    Let us know how it goes!

  • by John_Booty ( 149925 ) <johnbooty@bootyp ... minus poet> on Monday October 29, 2001 @10:13AM (#2492603) Homepage
    /dev/null ???

    Pros: Extremely high write speed
    Cons: Hard to get data back out, but since "Retrieval... can be essentially arbitrarily slow" you've can just re-film whatever it was that you missed. With the money you save on the video gear, you should have a nice little production budget, too...
  • by MadCow42 ( 243108 ) on Monday October 29, 2001 @10:14AM (#2492604) Homepage
    The Casino industry is probably the most advanced in the business of surveilence... the average Vegas casino probably approaches the scale you're talking about already, however they probably don't archive indefinately.

    However, any information I've seen shows them to still be mostly analog capture for any storage, or at least digital-to-analog conversion for storage.

    Although they probably won't talk about their security systems, they'd be a great resource.

    • In fact, despite what one user said, the Casino industry is one that would do this. They have hundreds if not 1000+ cameras in most of the larger casinos. It needs to be at least 30fps and it needs to pretty much be 24/7. 30fps to catch the slight of hand stuff and there's going to be very little you can get from motion detection as they tend to run 24/7 with at least someone on camera (usually a dealer at the very least).

      As for capturing the data, if going digital is a must, I'd probably spread the capturing out over a number of computers with the data going to a central repository. Actually, you might want to get capture boards with onboard compression (if you don't already have them) to offload a lot of the compression from the CPU, as that's going to be your most CPU-intensive aspect.

      Handling the incoming 90MB/sec shouldn't be a problem if you distribute the network load properly and use a fibre backbone.

      Your biggest problem is going to be downtime. If, for example, your data server fails, you're looking at a pretty expensive reboot, especially if you don't have a journaling file system (I assume you'd use a journaling file system). Still, the data server goes down and you lose all your feeds. For this reason, going to analog is much safer. It's rare that ALL of the analog recorders would fail at once.

      Just my thoughts. The idea of consulting with the casinos is probably the best idea. Actually, the casinos would probably be willing to share this kind of technical information, as it wouldn't provide any information on how to get around their work. What they're less likely to talk about are the types of tricks they look for.
      • Yes, casinos do have that much video; if you can get them to talk about it (after dealing with them for less than six months), my hat goes off to you. =)

        Actually, the main problem with checking with the casinos is that they themselves are trying to move to digital. Untill recently, the technology simply was not available, so they were stuck using analog (and rack after rack of tapes).

        You're right about the capture though, you need to either use cards that do the processing or move it to seperate machines (video encoder/decoders).

        As for the server downtime, getting a reliable video server, setting up everything correctly, and going from there means very little (if any) downtime. This depends on how much money you are willing to spend; remember many tv stations broadcast from digital signals, and their downtime is minimal.

  • The guy who wrote the post, is working on face-recognition technology. (thanks for researching him, cwhittenburg)

    This exactly the type of thing that could be done, but shouldn't. Stay away from this, we can only hope that any competent people will stay away from this and they will never get it to work.

    Sleep with one eye open.......
  • 640x480? (Score:3, Insightful)

    by hatless ( 8275 ) on Monday October 29, 2001 @10:23AM (#2492650)
    Why 640x480? That's higher resolution than broadcast TV. Do you need that? Broadcast TV is 460x360. Capturing at that resolution will lose you detail, of course, but if it's detail you can lose, your storage requirement just dropped by 40%.

    And since you said retrieval can be "arbitrarily" slow, I'd look into using VHS videotapes--even if you store compressed digital on them--as a storage medium. They're slow as hell for rerieval, but the media might be cheap, especially compared to the likes of AIT and such.
  • by sphealey ( 2855 ) on Monday October 29, 2001 @10:26AM (#2492666)
    That's an awful lot of data. Why exactly are you doing this? What is the application? Who are you working for? If you are working for/in the United States, does this application meet the requirements of the 4th and 1st Amendments to the Constitution?

    Just a few minor questions.

  • hah! (Score:3, Funny)

    by austad ( 22163 ) on Monday October 29, 2001 @10:32AM (#2492698) Homepage
    Like we're going to tell you Mr. NSA/CIA/FBI/Big Brother. :)
  • by DickBreath ( 207180 ) on Monday October 29, 2001 @10:37AM (#2492724) Homepage
    Only 1000 cameras?

    I mean 1000 cameras is only enough to put one camera into each home in a fairly small community. Most of the solutions I'm seeing posted so far don't scale up very well. What if you need multiple cameras per home? And what to do about large cities? Maybe this should be a seperate Ask Slashdot question?
  • (((640 *480) * 24bpp) * 30FPS) * 1000 =
    Roughly 2.21184e+11 bits per second, uncompressed.
    Why not just erect a satellite dish and stream the data into space? The cost of having an alien intelligence beam it back to you would probably be less than conventional storage costs.
    I would especially favor this plan if it's some kind of face-recognition software or other privacy-deprivation program.
    Another plan would be to hire 1000 people to sit and swap tapes on their home VCRs every few hours - just hire the people who currently plan to make a living stuffing envelopes at home.
    Yet, something tells me that you're not doing anything that I would want to encourage...
    Jim, hopefully out of your Jurisdiction
  • You need to reduce as early as possible as fast as possible. Choose the minimum resolution/bit depth/frames you can live with. Find a compression device (i.e. mpeg2/4) that can handle n cameras realtime and drop res/bits/frames to your specs. These will be expensive, but you can probably make up for it by reducing your storage and network requirements. Funnel n cameras into the devices using analog cabling, then spit the resulting digital stream out a network to m archiving stations with lots of high bandwidth storage to use as buffer. Of course, you'll need spares for each of these devices.
    Hire your tape monkeys to run tapes and/or get a disk silo. I suspect you could get away with much less storage than you think by compressing early.
    Even better would be motion detection on the compression or camera side so that only the cameras that are sensing movement would be working.

    As for your network, you'll need to partition it to meet your bandwidth requirements.
    Good luck!
  • No matter how you look at it and what media you use, if you plan on storing this stuff indefinitely you'll have to go for a large (think medium - large shed size) jukebox. Even if you store it on, say, an EMC solution, you'll have to take old data offline at some point for cost and efficiency reasons.

    That being said, remember that a run-of-the mill 16/10/40 CD-Recorder records at up to 140MB/s, which is well over your goal of 90MB/s. High-capacity CD-Rs can go to 800MB, so you'll end up writing 10,000 per day, and you'll want to use about 100 or more cd-rom drives to do the recording (for reasons of robustness, and to lengthen the average MTBF).

    BUT it sets you up for an easy swap in later to switch to DVD-R, or the newest 100GB disc (vapor-hardware) technology, though you can't go much cheaper than CD-R, along with storage reliability of 5 years or so in decent circumstances, 10+ years in climate controlled lockers.

    I'd imagine making CD-R cassettes that hold 10,000+ CD-Rs which simply get changed once a day (remember - 100 spindles of 100 CD-Rs is not much space, and can easily be handled by standard warehouse automation equipment). As a bonus, you can store them by day. Need a few hours on Oct 29? Just pull out one cassette. You'd have to trade out a CD-Recorder daily as well, but overall you're going to spend a LOT less than any other storage solution out there.

    You can't get much cheaper than this, and the tradoffs may well be worth it to you. But it would likely be a fully custom which would take time to design. It might compare favorably to a cassette of 100 .1TB IDE Hard drives, which would take up less space, and you don't have to worry so much about hardware failure since the drive electronics are switched daily.


  • I'd recommend finding a solution that works for 200 or so video stations, then split up your cameras/back end systems into chunks of that size.

    So find a 200 video solution, and duplicate it five times, situating each site in a geographically convienient manner. A bit of coding glue on the back end might let you have a centralized index.

    A single mechanism to handle that much traffic is going to run into many problems and probably won't be satisfactory. bandidth, storage, encoding time, etc for a smaller number of cameras is far more tractable for a smaller number of machines.

    This also gives you a way to scale. What happens six months later, when now you have 1300 cameras to deal with? add another site. With a good template, your costs become very predictable, you have a built in mechanism for pilot testing (you will learn ALOT from building the first site), and your project time to scale up becomes easier to estimate.
  • The original question is basically meaningless without a description of the real project requirements.

    You've put some big numbers up there, which all the hardware-heads have been happy to answer for you. But without real information, this is all just big-clock-speed hardware masturbation.

    The real question is: what kind of single system/application would produce 24,000 hours of unedited high-quality video per day and storing it until the end of time?

    Most respondents seem to assume that you're running a network of security cameras. If so, other posters have indicated that your quality or recording time requirements are probably 1-2 orders of magnitude too high.

    If you're producing something where this video is actually going to be watched by people who care about beautiful full-color full-frame-rate production quality, where is your 200 million dollar production staff that will be watching and cataloging this data? And if you can afford that, surely you can afford at least one knowledgable systems engineer who knows how to design a storage system!

    My absolute favorite bit of your post: "Reliability should be good enough to not be annoying long term". So how much lossage is "annoying"? Will you save space by randomly culling videos from the previous 7 days? If the whole system breaks down once a month, will that be annoying? If you lose one minute out of each hour, will that be annoying?

    This is the sort of problem that can be designed and solved if you've got the need and budget for it. But it's not a turnkey solution. That's why you hire engineers.
  • Using FPGA hardware, Rob McCready, the original poster achieved around an 88% face detection rate in real-time from black and white pictures with a 30 frames per second video camera.
    This explains his crazy requirements : looks like he is busy thinking about large scale applications of his work. tm []
  • Okay, mod me as redundant, but I'd like to know what the possible real world application is...

    First off, there are commercial solutions that tape 10:1 onto VHS tape... maybe you could take one of these a step further (custom ASIC dev) and do 100:1 for grayscale.

    Now, assuming analog is not the way to go (sounds like you want to stay digital), first you'd probably want to do motion detection. Only capture the stream from a camera that is reading changes. Depending on the application, that can cut your storage space down significantly, especially in off-peak hours.

    Second, I'd probably rethink 30 fps. 15 fps is nearly as good, and takes half the storage space. ATM cameras can range from 5 fps to 1/5 fps... and give good enough results in most circumstances.

    That combination should take you down to between 1 TB and 100 GB per day (maybe less depending on usage)... which is definitely more manageable. At that point, an HSM is the answer... look at any of the other excellent posts regarding this.

    I worked at one point for a major US corporation with similar needs, and they have a data backup center... it's basically a huge warehouse with tape drives, robots to change tapes and cart them around the warehouse, and tons of archive space for the said tapes. Perhaps this would be a good solution for you? As I recall, they spent about $5 MM for the hardware, and about the same for the actual installation (which is terrorist resistant, etc.)

    Hope that helps!
  • by anotherone ( 132088 ) on Monday October 29, 2001 @11:45AM (#2493047)
    Just pay everyone on slashdot $10 to run a special version of morpheus. Then just send everyone feed from one camera. When you need to access it, just hope they're online.
  • I was posed a similar question at my place of employment, and the only thing I could think of that was remotely feasable was to use DVCR's or security vcrs. Thus the bulk of the information, could stay on tape, and we'd only transfer onto a computer, what we actually needed.
  • > Retrieval, though, can be essentially arbitrarily slow. ... and lets you save some bucks on the (slow) read side?

    Ok, here's what you do: Take all of your signals (don't even bother compressing them... as other people have pointed out, it looks like 9 GB/sec might be hard to compress) and beam them straight out into outer space. This solves both problems - the massive amount of data, and the high bandwidth.

    Now, current technlogy offers few options for playback, but since you don't mind waiting for the playback (you did say arbitrarily slow), I'm sure solutions developed in the future will be both feasible and cheap. (if not, just wait a little longer). Here are a few possibilities:

    Develop ultra-focused and ultra-sensitive receivers to detect the signal bouncing off of a distant object.

    Teleport a big mirror ahead of the signal to bounce it back. Listen for it to return.

    Develop a faster-than-light vehicle to pass the signal, and then use future technology to record it on some futuristic ultra-dense medium. (Note: this can be done obeying Einstein's theory of relativity. Space isn't quite a vacuum, so the radio signal is travelling at a speed less than the speed of light in a vacuum. You could evacuate an area of space for your vehicle to travel so that it is less dense than the space the radio signal is travelling through, and thus has a higher speed of light. You'll be able to catch up without exceeding the speed of light in your path, but exceeding the speed of light in the radio signals path).

    Wait for either the universe to collapse in on itself, or a black hole to devour both the earth and the radio signal. This should, with any luck, warp space such that the signal hits the earth again.

    Direct the signal at a black hole such that it orbits it. The signal will be trapped in orbit. You could send a vehicle to this area to read the signal, and then retransmit it with more energy to escape the black hole. Of course, you'll only get a brief snippet of data before the vehicle is sucked in, so many may be required. (I'm not too good at black hole theory; hope it doesn't show!)

    Anyway, there are lots of possibilites!

    -- morcheeba
    founding member, literalist society.

Whenever a system becomes completely defined, some damn fool discovers something which either abolishes the system or expands it beyond recognition.