Ask Slashdot: What Is Your View On Sloot Compression? (youtube.com) 418
An anonymous reader writes: A Dutch electronics engineer named Jan Sloot spent 20 years of his life trying to compress broadcast quality video down to kilobytes -- not megabytes or gigabytes (the link in this story contains an 11 minute mini-documentary on Sloot). His CODEC, finalized in the late 1990s, consisted of a massive 370Mb decoder engine that likely contained some kind of clever system for procedurally generating just about any video frame or audio sample desired -- fractals or other generative approaches may have been used by Sloot. The "instruction files" that told this decoder what kind of video frames, video motion and audio samples to generate were supposedly only kilobytes in size -- kind of like small MIDI files being able to generate hugely complex orchestral scores when they instruct a DAW software what to play. Jan Sloot died of a heart attack two days before he was due to sign a technology licensing deal with a major electronics company. The Sloot Video Compression system source code went missing after his death and was never recovered, prompting some to speculate that Jan Sloot was killed because his ultra-efficient video compression and transmission scheme threatened everyone profiting from storing, distributing and transmitting large amounts of digital video data. I found out about Sloot Compression only after watching some internet videos on "invention suppression." So the question is: is it technically possible that Sloot Compression, with its huge decoder file and tiny instruction files, actually worked? According to Reddit user PinGUY, the Sloot Digital Coding System may have been the inspiration for Pied Piper, a fictional data compression algorithm from HBO's Silicon Valley. Here's some more information about the Sloot Digital Coding System for those who are interested.
Not so fast (Score:2)
If it were just as easy to kill off someone to "solve" an issue like this then the bitcoin scaling debate would have been resolved a long time ago...
Re:Not so fast (Score:5, Interesting)
Hear the rhythm of the beat and you will know there's maths in music. Yep, an entire orchestra can play of a few pages of dead wood. Voila problem solved.
Do not think so much about compression, think more about robotic simulation and scripts. So create a simulated robotic orchestra and write them a script and the play the audio, visual role. That script is the compressed version of the orchestra's efforts.
So can you create a computer program that would 'act' out and entire program based upon scripts provided, of course you can, it would take quite a bit of development but once you develop one virtual human robot, you have developed a virtual infinite number of them.
Re:Not so fast (Score:5, Insightful)
Sheet music as a form of compressed script is a very lovely image. Repeat signs, first and second endings, D.C. al fine are all ways to put more music on fewer pages. I have to give it to you, that's nice, and I plan to use it.
Re: (Score:3)
Virtual infinite monkeys? I wish I would have thought of that.
Re:Not so fast (Score:5, Funny)
Yep, an entire orchestra can play of a few pages of dead wood. Voila problem solved.
For once it would have been almost appropriate to misspell "voilà" as "viola"...
Re:Not so fast (Score:5, Insightful)
Sheet music is not "the" music. Its merely a non complete representation, a suggestion if you will, how to perform the music.
If it would be the case, then MIDI files are all we would ever need.
Most of the music is how the director, and the musicians interpret it, and is not codified in the sheet.
An analogy would be that sloot compression would reproduce a decompressed movie about "a" cat, but it would not look like the homevideo of "your" cat.
Not an information theoreticus myself, but Im pretty certain that sloot compression went beyond what entropy would allow, aka more info was "supposed" to be there than entropy allows. I don't believe in violating the laws, of nature.
Re: (Score:2)
That's a completely different problem - there are no secrets in the bitcoin algorithms or implementation. It's a political issue, and resolving the debate requires consensus among a critical mass of stakeholders.
On the other hand, if it's the knowledge in a single person's mind that's a problem for you, then it really can be "just that easy" to solve. That's pretty much the entire reason for the existence of witness protection programs.
Re: (Score:2)
consensus among a critical mass of stakeholders.
The threshold is 51%, and the 3 biggest miners control 53%. The top 3 are all in China. If the 3 of them agree, they can do anything they want with the Bitcoin blockchain.
Re:Not so fast (Score:4, Informative)
No, they can't. They could do double-spending attacks, but with fairly low probability and not without people noticing. They couldn't add arbitrary transactions, because transactions need to be signed by private keys that miners don't have access to. They can't commit invalid blocks because all other full nodes (crucially including the ones run by payment providers and exchanges) check the validity of blocks. They could fork and make their own chain, but anybody can do that and the fork wouldn't be very interesting if nobody was using it.
They can't do "anything they want".
Re: (Score:2)
The issue is there is a person that by their continued existence creates a "problem". The conspiracy argued here is that Sloot was killed to stop his invention. I am arguing that there are similar economic incentives and far more shady people involved in the bitcoin debate. It's just not as easy as killing this guy or that guy or we would have a lot more dead bodies in the world.
Re:Not so fast (Score:5, Informative)
The conspiracy theory ignores Jevon's Paradox [wikipedia.org]. As computing efficiency goes up, people buy more computing/storage/bandwidth since the increased demand driven by new applications swamps the decreased demand from greater efficiency. So Sloot's compression algorithm (if it really worked) would have likely driven demand up, and killing him would have made no sense.
Disclaimer: I didn't kill him, and this post is not an effort to cover up the conspiracy.
Re: Not so fast (Score:5, Funny)
It's always interesting when someone makes it a point to deny what no one has accused them of.
Why are you trying to shift the blame to me? Where were YOU on the night of July 11th, 1999?
Re: Not so fast (Score:5, Funny)
Which means BeauHD is the singularity; a really highly compressed AI that feels the guilt of killing it's creator!
Re: (Score:3)
BeauHD is the singularity; a really highly compressed AI
I can buy the "A" but where's the evidence for the "I"?
Re: Not so fast (Score:5, Funny)
Sloot's death was an inside job!
All heart attacks are.
Actually... (Score:5, Insightful)
They killed him because you could feed a random input into his decoder and the movie that came out would be better than anything Hollywood can produce.
I love it that parent is modded "Insightful" (Score:5, Insightful)
That really says a *lot* about the quality of today's movies.
Re: (Score:3)
That really says a *lot* about the quality of today's movies.
That also says a lot about /. readers.
Re: (Score:3)
But in 1999, Hollywood movies weren't so bad. That's the year that gave us The Matrix.
That's also the year that Hollywood gave us Dudley Do-right, Wild Wild West, and Deuce Bigelow: Male Gigolo.
It's not a thing (Score:5, Interesting)
So, you want to replace every frame in a movie with a collection of images or snippets that correspond to each part of the frame, right? And you're going to store a dictionary of snippets, referenced by number, then say "this frame takes snippet 1234, 6543, and 9274". The problem is that the number of snippets you'd have to store is enormous, and that each snippet itself is going to be a ginormous number (like the bits of the string of bytes in that snippet).
See where this is going? You're basically establishing a mapping of small numbers to much larger numbers. Either that set of big numbers is tiny (in which case you can only represent a small number of frames in the output video and picture quality is awful) or it's huge, in which case the index numbers themselves become roughly as big as the numbers they're referring to, and oh yeah, good luck searching through that space bunches of times per frame.
The idea isn't inherently bad if you have a small number of states you want to represent. For instance, Zstandard [github.com] lets you precompute a dictionary of common strings you want to shorten. Imagine if you trained it on HTML so that each tag or other common string just takes a few bits, then you can distribute that dictionary to the whole world so that you can save the bandwidth of transmitting it alongside the compressed data each and every time (like we do with Zip, Gzip, etc.). That's a nice thing! But the search space of "things you can display on a screen" is a hell of a lot bigger than "things you can sent in an HTTP header".
Re: (Score:2)
Vectors vs. bitmaps, which compresses better?
Re: (Score:2)
As I understand it, fractal compression works somewhat similarly--you take the snippet and create a fractal equation that reproduces that part of the image. Since you're basically just storing inputs to the equation, you can get very small compression. Of course, you need pretty good CPU power to generate the pixels and the equation will start to fall apart if you scale too high.
Re: (Score:3)
Re: (Score:2)
the amount of CPU required to encode and render it would be insane.
GPU not CPU. GPUs are insanely powerful. The GPU in my laptop has 1024 cores running at 2GHz. That is 2 trillion ops per second.
Re: (Score:3)
An 8x8x24-bit rectangle has 10^462 possibilities, and fractals are hella sensitive to initial conditions. Finding an approximately match is going to be less painful than naive brute force searching, but it still laughs in the face of a little teraflop GPU.
Re:It's not a thing (Score:5, Interesting)
An 8x8x24-bit rectangle has 10^462 possibilities
I can encode 100% of the random noise possibilities in only a few bits driving an LFSR... and you wont be able to tell the difference.
Repeat after me: "Everything I learned about the pigeon hole paradox is true, but I didnt actually understand any of it"
This is not a pigeon hole issue because nobody is pretending that these compression technologies do equally well with every kind of data, and even the kind of data (noise for instance, which is traditionally hard to compress) doesn't apply because these compressors are lossy such that random data can be approximated by literally any of a large collection of simple random functions.
You cant tell the difference because if you could then it would have been compressible to begin with. You can only tell when the coherence is amiss, but its the coherence that is "accurately" compressible.. its only the incoherent stuff.. aka noise.. that is hard to encode accurately in only a few bits
We arent even close to the limits of lossy compression. as its just a time/space trade-off.
Re: (Score:2)
Yeah, it's not completely insane to think that a really brilliant fractal technique could do some amazing lossy compression, but the amount of CPU required to encode and render it would be insane.
We're kind of living with insane computing power in our phones these days.
That said, it never seems to be enough for some things.
Re: (Score:2)
It's not "impossible", you could equally make a pointer to any place in an irrational number (eg. find the nth digit of Pi) but the pointer is going to have the same size as the result.
Eg. you can find sla in Pi at position 91104776 and thus you could work back and forth the text. But encoding 3 bytes took ~3.5 bytes. Even if you were to compress that pointer (which can only be up to 2:1), your encoding is no better than what compression already exists.
Re: (Score:2)
Since you're basically just storing inputs to the equation, you can get very small compression.
Oh, for sure. But it could be that you have to use a lot of digits of precision to describe the parameters that yield the results you want. Maybe render(3, 5, 23) gets you a blue/orange gradient across a rectangle, but render(3+1/2^193, 5+87/2^432,053, 23+5943/2^1,404,504,305) gives you a cat holding a sign saying "eat more chicken".
Re: It's not a thing (Score:2, Interesting)
https://rogerjohansson.blog/2008/12/07/genetic-programming-evolution-of-mona-lisa/
You were saying?
You don't know what you're talking about (Score:2, Interesting)
The presumption isn't that it actually stored images but more likely that it contained numerous generators. It's entirely plausible is that similar to the way a neural net uses a combination of several algorithms to classify images, a similar approach could be taken to constructing them.
Computer magazine's ruby rod (Score:2)
Back in the day when Dr. Dobb's Journal and other computer magazines were a thing, one editorial writer was reflecting on how allowing multiple levels of an analog signal could encode more than one bit per symbol or cell or splotch or what have you. The idea was extended to a very hard, dimensionally accurate ruby rod upon which a very thin scribe line was made, and the writer mused that a very large number of bits of data could be encoded on that one physical object.
Since Claude Shannon's insights, Inf
Re: (Score:3)
That's nothing. BLAZE MONGER can compress 16kx16k video at 150 fps with 1-color (which must be black) at 0 bits/sec.
The blank personalized license plate (Score:3)
When I insisted that the null set should be included the enumeration of subsets in answering a problem on an Information Theory homework, the professor was reminded that his colleague and author of our textbook once asked for "the null license plate" (i.e. a blank plate) for his personalized license plate but was turned down as I was about to be in my request for points back on the assignment.
Whereas there are certain plates you cannot have -- obscenities along with "1" or "A1" or similar license numbers
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
It would be interesting if it could be done without the table on the compression side, and just be computationally intensive. Then transmit the compressed file. Then have a computationally intensive decompression that references a gigantic table.
I once came across
Re: (Score:2)
There are hard limits to how much you can compress stuff, especially losslessly. It's always possible to construct an input to any particular algorithm that compresses to be larger than it started. It has to be that way! Suppose that you have an algorithm that can compress any number between 1 and 100 to the range of numbers between 1 and 10. That means that any number in the output [1..10] range maps to an average of 10 numbers in the input [1..100] range. If you see the output number "7", which input does
Re: (Score:3)
For instance, Zstandard lets you precompute a dictionary of common strings you want to shorten. Imagine if you trained it on HTML so that each tag or other common string just takes a few bits, then you can distribute that dictionary to the whole world so that you can save the bandwidth of transmitting it alongside the compressed data each and every time (like we do with Zip, Gzip, etc.).
HTTP/2.0 actually does this [github.io] for HTTP headers as part of the HTTP Header Compression specification.
Re:It's not a thing (Score:5, Interesting)
I agree with you, but I do find this whole idea interesting at least, for a couple of reasons:
1) Video codecs today do use some form of the index-to-dictionary technique already (e.g. I-/P-/B-frames), the big differences include the fact that the dictionary is in the file itself (I-frames), and it is not comprehensive for the whole media typically but is instead relevant to only a small portion of frames before a new "dictionary" is created.
So not really the same thing as what Sloot was possibly doing, but it does make me wonder if there is something to the idea of having some builtin library of sample blocks that can be "good enough" replacements for many cases, or maybe act as starting samples that can be transformed into something suitable for various uses, etc. This never escapes the indices-become-too-big problem that you rightly pointed out, but there could very well be merit in the general concept. I doubt it'd yield the massive reduction in bitrates Sloot was apparently hoping to achieve, but maybe it'd be enough for the step in compression improvements? I mean h264/h265 already do some pretty impressive and creative stuff already, so something along these lines isn't completely outlandish.
2) Media compression (and to a lesser extent decompression) always battles the cost tradeoff, both the compute power required but also just the time it takes to do the compression. Because of the complexities of getting a codec adopted everywhere (especially in silicon like in a phone), there is a push to have general purpose codecs that can be used for both live and pre-recorded media, and the type of analysis that is done on each frame or on groups of frames is still driven by available computing power - even when you fiddle with the compression settings between "just get this done in realtime" to "take as long as you want and analyze each frame to death", the difference in time between the two extremes is usually just an order of magnitude or so - a big difference to be sure, but still kind of within the same realm.
The point is just that the types of analysis done are impacted by whatever is considered to be, at that point in time, a feasible amount of computational work. When we think ahead to 5 or 6 orders of magnitude of more compute power (due to faster CPUs or harnessing gazillion-core GPUs or something else entirely), it's hard for us to really grasp what doors that opens in terms of analysis techniques - we can mostly think in terms of what we do today and how we could make that faster, but as we get there we'll also start to take on completely novel approaches simply because things that were too ludicrous to even consider before become worth exploring.
For example, we are still at the dawn of GPU-based video compression (in the sense that right now GPUs can do some things to speed it up, but in /general/ they are just optimizing the CPU-based approach - we're at the early stages of actually have codecs designed for GPUs specifically. Given vastly more computing power, maybe we really could get to the point where the sample blocks mentioned above can be distilled down to equations that produce them (them or a good enough representation). When we get to a relatively absurd amount of compute time available when compressing the video, there's no telling what sorts of new ideas and techniques will be explored.
Or maybe we'll take on a completely different approach to video altogether. I mean, what if we move away from trying to reproduce individual pictures and their pixels and instead start transmitting the scene info? A TV show has a limited number of sets, what if the 3d model/texture info for most everything was transmitted in advance so that during playback all of that information is expressed in a few identifiers and their 3d transforms (ditto for the actors at some point too). We already see this in simplified form in replays from video games - a replay of a minute of gameplay can recreate the scene with perfect fidelity, and yet it is tiny compared to wh
Re: (Score:2)
Great answer, and I think you're likely right. I can at least imagine a movie that's shipped as a Blender file that gets rendered in realtime, or maybe a collection of object hashes ("you don't have cylinder #837 yet. Let me fetch it! Also standard longhair cat #294.") where you build of a local cache over time. That seems practical, or at least possible. But I agree it's not too similar to Sloot's idea once you move past a one-sentence description of the two ideas.
Re: (Score:2)
He would compress an entire full-length movie, supposedly in 8kB. It's physically impossible, entropy has a thing to say about it.
Re: (Score:2)
It's not physically impossible - we can imagine a super AI (or huge team of humans) that could reconstruct a reasonable facsimile of a movie based on grainy thumbnails of the cast and a quick synopsis of the plot and shot framing (and with work you could fit that in 8k).
But yeah.. with "fractals", not so much.
it cannot logically work. sorry. (Score:4, Insightful)
You're forgetting that it's 8kB ... plus the 370MB of data preloaded in the de/compression engine.
look. it doesn't MATTER if it's 370mb of data or if it is 1000 terabytes of data. given a random movie file it would have to be able to tell apart the same movie but lets say a 10 seconds of it is just blacked out in the other version or replaced with a random text(this is to take out possibility of having the whole movie already in the 1000 terabytes of data). you cannot index chunks with such a little data to go match there. IT CANNOT BE DONE. if it could be DONE THEN THIS WOULD REVOLUTIONIZE HARDDRIVES AND ALL DATA TRANSFER - not just videos.
besides 8000 bytes / 120 min is about 66 bytes per minute. thats like a byte per second.
so basically, at least the claims are an order of magnitude more ludicrous than is possible at all.
also a few years ago I ran into some startup guys who were claiming that (among other things) they had a new compression algorithm that could do 1gig to 100kb or some such... I tried to explain them the theoretical maximum they could attain is way worse what they were claiming (due to simple logic)... they never did anything with it or showed it of course - even if it would have been 1000000x more valuable than what their current company was doing. kind of lost faith in that then.
furthermore you could embed 50 megs of data easily visually on a normal vcd movie of 650 megs. what this is suggesting is that you could embed any random data of 100 megs into 8 kilobytes. it doesnt add up. it cannot add up.
Re: (Score:3)
given a random movie file
I can encode 100% of your random movie files in a single 16-bit LFSR.
.. family videos.. there are random elements, but the data is neither arbitrary nor can be said to be random. That pixel next to pixel 148675 is related to it. There is coherence.
Nobody cares about compressing random data.
Thats the fallacy of your argument. People only care about the coherent data.. hollywood movies
It is the coherence that is compressed. The random stuff is just replaced with different random stuff.
Re:it cannot logically work. sorry. (Score:4, Insightful)
yes I know what demo guys have done, and it's done by hand, tweaked to produce something that looks nice, and by that I mean the procedural algorithms are written by hand to make something that looks nice - not something arbitrary they have in their mind.
it can look magical but really it is not. none of the demoscene demos break logical rules.
and yes, something like elite frontier fitted on a floppy and had a billion stars to explore with planets. but that is not an argument for this video encoding scheme at all - it's not like that on some planet in there you might find a tv playing jaws - it cannot have that and you can know it cannot have that.
however sloot claimed his video encoding to be so good that you could have on a floppy all the movies in the world- because recursion, if his scheme worked you could encode the next movie in the images in the credits scene of the previous movie - THIS SHOULD clue people in that it was just all bullshit, but for some reason doesn't? how hard is it for people to grasp this recursion problem of arbitrary information? it has nothing to do with if the compression scheme is procedural or not, it's just about if you can point to so much arbitrary data with so little data.
he was claiming benefits wayyy beyond of an automated mod/tracker creator.
look even if it was seeded with aqua teen hunger force episodes it still wouldn't be able to do the compression he claimed.
like come on dude, he was basically claiming that he could do better video encoding than what the best image encoding could do. if he had something that worked he would just have patented it and made a bank.
and if you're thinking of fractals - that too works only if you don't grasp bits and data, the pointers to the data in the fractal would have to have such a precision that it wouldn't be able to perform so well as he said. simply put, if you had kilobytes to represent a movie then that batch of kilobytes would have to represent multiple movies and that simply cannot work.
Re: (Score:2)
The problem is that I can't imagine a 370MB blob containing enough information to accurately render "Gone With The Wind", "Jaws", and "The Matrix". I just don't think they'd have much overlap in primitives.
Re: (Score:3)
Yeah, but what if you replaced everything with stick figures?
Shark in Jaws could be just a single triangle. Hell... make that just two lines...
Someone get me Spielberg on the line!
I have an idea for a very low budget movie! It'll make back on its investments thousand-fold... million-fold!
Re: (Score:2)
I think it's possible to give some reasonable lower bounds on lossy compression using a little bit of perceptual hand waving.
The key is to generate a large set of "different" movies. These don't have to be good movies, but it needs to be obvious to a human that they are different. Let's take the top 5000 movies of all time and break them up into 10 second pieces. Assuming on average each movie is around 2 hours long this give one an alphabet of 3,600,000 symbols. Now I can use this alphabet to creat
Re:It's not a thing (Score:4, Interesting)
For any given number of any given complexity, there are adjacent numbers that have significantly lower complexity.
I have toyed with the idea of using "indexing" between two points on a number line, where the two points are very low complexity, but the number you want is somewhere between-- With a high precision percentage of the now clearly defined space between the points on the line, we can skip over and start playback of the number we want, if we also state how many digits this number is.
Say, the number we want is between 10^23 and ((10^23)+(10^10)). Those are both impossibly huge numbers, but they can be defined very easily because they are not complex. We will then say that the number we want is found 10% of the distance between those two numbers, and that it is 10^6 digits long.
Just using some fun math with the natural logarithm, we can produce that number, from those modest ingredients.
Video files are just very very large numbers. They could be found on the infinite number line in exactly the same way.
A complex encoder that has broken the file to be searched into a series of smaller (and thus easier to compute/derive) numbers that get concatenated together, and has a working knowledge of number space for numbers of those data block sizes, could reduce hundreds of megabytes into a few kb of metadata. The decoder would have to compute very large numbers (or have very large numbers already stored statically, and just crawls along...) to initialize the playback, but it could easily do so using nothing but lots of RAM and brute force CPU.
I have considered looking into creating a compression system of this type myself, which is why I find it kinda spooky that it would need a huge decoder.. because my theoretical system would need a huge decoder too.
Re:It's not a thing (Score:4, Informative)
It doesn't work. I've written some papers on this.
Basically, every set of easily-compressible numbers has the property that the next one gets farther and farther away at exactly the rate needed to ensure that the offsets from one to any other natural number require (in total) exactly the same number of bits as you'd need to just write those numbers out yourself.
Proof:
1) Suppose you could represent any natural n in [0,max] as c(n).
2) Then you obviously need to be able to decompress any c(n) to the corresponding n. Hence, c is bijective.
3) By (2) c(n) is just a permutation of [0,max].
5) As is obvious since (3) is a permutation, the total compressed size of the output range is equal to the total compressed size of the input range: Information content is conserved by permutations (Pigeonhole principle)
Now, your particular method sets c(n) = (k,o), where k is the nearest compressible number, and o is the offset, perhaps also compressed. It doesn't actually matter what compressor functions you use, the result is general:
The proportion of strings in your domain which compress down to less than m bits remains unchanged. In other words, there are 2^m strings in m bits, and the proportion in [0,max] is just (2^m)/max.
Set max = 2^(1000000000000), or the number of terabyte-sized files. Now just evaluate (2^m)/(2^1000000000000) and you'll quickly see why your compression algorithm doesn't work. It'll work for a vanishingly small number of bitstrings, just like how we can represent 10^23 using only 5 symbols, but as you go further and further out from those easily-compressible numbers, the offsets themselves become uncompressible large numbers!
--
- Oskar Lidelson
Of course it didn't work (Score:2, Insightful)
Information theory stablishes what is really possible and that kind of compression is simply to much, even considering its lossy nature. Infomation entropy has very real limits no matter the encoder used in the data compression.
Re:Of course it didn't work (Score:5, Interesting)
Yeah, I don't think his work would have panned out either, but in theory the idea isn't totally implausible, in part because general purpose media compression is a bit of a special case in that the goal is not to try to have lossless compression, due to the way human audio and visual systems work.
When you look at compression on individual frames, there is a huge amount of acceptable error (differences versus the original) that isn't even discernible when you're looking at still frames side by side, and then when you're playing back the frames at a rate of 30 or 60 or more per second, there's a whole *additional* layer of imperceptible error that can be introduced. And then on top of *that* you can introduce more and more error that is in fact discernible on some level, but it becomes a game of tradeoffs of what you can get away with vs the various costs involved.
Another way to put it is that this isn't just lossy compression, it's lossy compression that takes advantage of quirks in the way human eyes and ears work, and so the limits of compression potentially go far beyond what you might expect in a strict information theory sense (there are still limits of course, they just might be a lot further out than you might expect).
On some level it's kinda like how paintings work - you see a barn with a tree next to it in the shade, and yet when you look at it really close you realize that it's just a couple of strokes of paint - your brain perceived far more detail than is actually there. To be clear, I'm not saying this is how media codecs work, just saying that the goal in media compression isn't necessarily to accurately reproduce the source material, rather the goal is to get people's brains to *perceive* that the final version is an accurate reproduction, and in practice that opens up a lot of possibilities.
Re: (Score:3)
Indeed - our brains are incredibly good at filing in "missing" detail - a rough suggestion of something is often enough to "see" incredible detail.
Re: (Score:3)
Yes, and it's some really neat stuff - research talked about in these articles is fascinating:
https://www.sciencedaily.com/r... [sciencedaily.com]
https://medicalxpress.com/news... [medicalxpress.com]
And here's a site that lets you experience just one of the ways the brain manufactures some of the detail you perceive: http://www.uniformillusion.com... [uniformillusion.com]
A common theme in all of these is that "sight" isn't entirely achieved with our eyes, but our brains get involved very early in the signal processing stage and even make up a lot of info based on wha
Re: (Score:3, Interesting)
This is how literature works... a short story is in effect a full movie highly compressed down to a few kilobytes. 'Reading' is decompressing the data.
Re: (Score:3)
Actually, information theory states the opposite. Determining entropy of unknown source is an intractable problem [hutter1.net], and you can't generally state amount of entropy for piece of data unless you're certain it's a quantum pink noise beforehand, all we know that the better the compressor, the closer you get. That's why one time pads use truly random codebooks, not a PRNG (PRNG has very little entropy - that of PRNG seed).
While extremely important as
Re: (Score:2)
The problem is real video has lots of noise it and things that look like noise don't compress well. They can't, as they appear to be random.
But that can work to your advantage. If you can determine what is random noise and what is not, you can just drop the random noise and replace it with generated noise. It's like the difference between compressing a picture of a boat and a digital representation of a boat without all the noise. If you can get a "good enough" virtual represetation of the boat, it still looks like a boat to the human eye but it's missing all the noise. It might look a little too crisp to the human eye but you can add back
More plausible explanation: (Score:5, Interesting)
Jan Sloot had some ideas on how to compress things and despite hyping it up, he had gotten nowhere close to anything functional. With all the hype he generated, he had a deal lined up for the technology but he didn't have the goods. The stress of the impending revelation of his fraud caused him to suffer cardiac arrhythmia and he died. The source code was never found because it never existed.
Re: More plausible explanation: (Score:5, Funny)
So you're saying he should have waited for Kickstarter to be invented?
Re: (Score:2)
He might have had success with a limited range of inputs, like news broadcasts with little or no clips to show. Broadcast quality was probably 320*240, or the PAL equivalent, and that can take lots of lossy compression and survive on a CRT. It could look promising but really be near the peak of its ability.
It's not implausible to have something like 2 tone 4*3 blocks, sending a block ID and primary and secondary color, and the decoder assembles the appropriate mosaic. Like pictures made of bunches of other
Re: (Score:2)
Why bitmaps? It could have been vector graphics using a range of primatives to match regions of the image. I was using a thing that built up vector graphics from bitmaps in the late 1980s on an Atari ST FFS.
Considering who he saw as the market it didn't have to be computationally cheap, encoding for days was not out of the question. Decent hardware and a lot of time can make a lot of approaches that seem ridiculous on a home PC make sense.
It's
I'm not sitting through an 11 minute video to see (Score:2)
I'm not sitting through an 11 minute video to see if there are any examples of his compression - in particular the output compared to the input.
But the proof is in the pudding - what was the output quality like?
What was (Score:2)
What was that file format that was parts of sampled songs that were played back like a midi? It was "popular" in the late 90s I think. Maybe earlier. Pre MP3.
Sounded like a chopped up version of the original song. Much larger than midi but much smaller than wav.
Re: (Score:2)
Mo3, Tracker, MOD, all similar or synonyms probably.
Re: (Score:2)
I want to say .mod?
Yes, it's possible (Score:5, Funny)
Myself, I've come up with a system that can compress a video file down to one byte. Unfortunately, it has some limitations. The size of the decoder is approximately the same size as the uncompressed video file, and it will only work on one specific file.
Damn, should I be afraid for my life now?
Re: (Score:2)
I can do the same thing and decompress 256 video files, so you're probably safe.
Dead scammer, not dead inventor. (Score:5, Interesting)
I believe it was a scam. He never really had that good compression. Either he got cold feet and offed himself before he had to deliver the product that would not work, or someone else found out and killed him in a a rage at being tricked.
While yes, there are a few scientific advancements that are remarkable, in fields like computer file compression that are:
1) Essential to an existing, highly profitable business
2) Mathematically interesting
3) Being heavily researched by multiple people.
then any advancements get duplicated in less than 10 years. Too much money, brains and corporate greed focused on this issue for us not to figure out it or something similar by ourselves.
This has not been duplicated, therefore it was fake.
Very interesting (Score:2)
I had a similar idea to develop an algorithm that would follow through and propagate successive frames of motion in a video, first analyzing it to create certain required textures and meshes, then applying it to various meshes that would be created by tracking the individual characters, objects, and sets in a video. I always believed it could work but it's a huge undertaking.
Compression Tweaks (Score:5, Insightful)
Re: (Score:3)
Yes but DNA does not "encode" a human. It encode a process for generating a body, which in the human case takes about 20 years. The encoding is not one-to-one either, see twins.
Pseudoscientific claptrap (Score:5, Interesting)
Sloot was nothing more than another of a long line of scam artists (or delusional inventors) who claimed to have created a "magic" compression scheme. In his case, he said he could compress an entire movie down to 8 kilobytes.
Simple mathematics show why such schemes don't work. 8 kilobytes = 8192 bytes = 65536 bits. Assuming you have a two hour movie, then each second of the movie must be mapped into about one byte, which can have only 2^8 = 256 possible values to represent any conceivable second of video. It's mathematically impossible.
Engineers and mathematicians have been debunking these claims for decades, but they still occasionally pop up. I remember one scheme that got some press about 30 years ago. A guy claimed to have a compression program that could take any data file and compress it down to about 1 kilobyte. And it seemed to work, according to several people who tried it. As it turned out, the "compressed" file was nothing more an alias pointing to the original file, which was hidden from directory view by the program. When you "uncompressed" the file, the original file was unhidden. But it was a neat trick as long as you didn't try to copy the "compressed" file to another disk.
Sloot's program was "lost" because it never existed, just like the magic 300 mpg carburetors where the plans were "lost".
Re: (Score:2, Funny)
Assuming you have a two hour movie, then each second of the movie must be mapped into about one byte, which can have only 2^8 = 256 possible values to represent any conceivable second of video. It's mathematically impossible.
That might actually work, as long as you restrict the input domain to the set of Michael Bay movies.
Re:Pseudoscientific claptrap (Score:5, Interesting)
You may be thinking of OWS, the "fractal compression program". The "compressed" file was nothing more than a list of blocks on the disk that the original file occupied. You could test it by compressing a file, deleting the original, then decompressing (=undeleting) it.
But if you copied it to a floppy and took it to your friends house, it mysteriously failed...
Here is a mention of it here on slashdot back in 2006: https://slashdot.org/comments.... [slashdot.org]
And a very brief mention in the compression FAQ: http://www.faqs.org/faqs/compr... [faqs.org]
Internet references to it are spotty. It got passed around on BBSs and sneakernet back before home PCs really started connecting to the internet in a big way.
You may also be thinking of WIC, which I didn't encounter, so I tend to assume that it didn't achieve as wide a distribution as OWS. The mechanism described for WIC sounds more like what you are describing. Either way, you are off by about 10 years. They were in the mid 90s, not the mid 80s.
Re: (Score:3)
You're right, I was thinking of OWS. Thanks for filling in the gaps in my memory.
I recall that it actually got some press in the local newspaper back
Re: (Score:3)
We tested it out, and were both amazed and suspicious when it seemed to work. But as soon as we tried copying the "compressed" file to another machine and it failed we knew something was up, and it didn't take long after that to figure out wh
Example of Pseudoscientific claptrap (Score:2)
Offtopic maybe, but I had a guy with something like that bring it in to the University where I worked for independent testing. He was an artist with not a lot less technical skill than a mechanic, and paranoid as anything, but he couldn't hide what he'd done. The trick (on the perpetrator more than anyone else) is to tune the engine for maximum efficiency at idle - thus it uses very lit
Re: (Score:3)
You could try transmitting the actual manuscript - compressed, with a partial dictionary known on the other side - and then have it decompressed by letting an AI "acting" it? The same way you could transmit sheet music instead of an actual performance?
Obviously, the actual performance will differ and there is no way he had technology to do anything like that...
Re: (Score:2)
Yes, but by the time you have encoded every possible combination that can be represented in a 'movie' (everything that's ever passed any screen), you need an infinite size 'database' and as your database grows, so will your pointers so your pointers will eventually become larger than the thing you're trying to encode.
His encoding scheme was basically: Let's send everyone all movies ever made (which with good inter-frame compressions would be as close to mathematically optimal) and use a DRM key to unlock wh
Snake Oil (Score:2)
What you throw away, you cannot get back. Think of compression as a plastic bag filled with snow and maybe a little air at the top. Assume air represents '0' and water represents '1'. A snowflake can be said to be represented then by a sequential pattern of 0's and 1's. The more you squeeze the bag, the more air (0's) you throw away. Initially you're merely squeezing the air at the top out but as you continue to squeeze this bag, you start throwing away the 0's contained within the snowflakes themselve
Saw the documentary (Score:2)
Then, people who actually saw Sloot's invention talk about "movies" without any more details, like length or quality. Not much convincing. So, maybe Sloot invented a device that simply uses a kind of wifi?
Re: (Score:2)
Machine vision problem (Score:2)
While retina and optic nerve run roughly at 8Mbits (retina already does a lot of pre-processing), visual cortex reduces this to 10kbit/s per eye and feeds it further into brain. Which is the same ballpark claimed by this kook.
It's obviously possible with *heavy* post processing to make very high representation of objects perceived. To replay, you need to hallucinate it back (akin to dreams). Problem is you need something as good as humans visual cortex, w
None (Score:2)
Anything is possible. Practical though? (Score:4, Interesting)
What I found was that every 4KB sequence I randomly generated as part of a set was able to be located within the Mandelbrot set. Results generally were found in the nautilus or horse tail areas. I used several search methods.
1) Linear
2) Rectangular spiral clockwise (variable width, variable height)
3) Rectangular spiral counter-clockwise (variable width, variable height)
4) Zig-zag in 45 degree rotations and variable width and variable height)
5) Elliptical Spiral (clockwise and counter-clockwise, variable width, variable height)
I also experimented with added dimension which included iteration as a 3rd dimension when pattern searching allowing strings to extend across multiple iterations in cubical, spheroidical, etc... footprints. I had to abandon this effort due to computational complexity.
I also experimented with experimenting with adding bit layers as a 3rd dimension when searching, but this added even greater computational complexity than the previous method described above but certainly promised greater results.
The algorithms I used were exhaustive. So where a video motion vector search algorithm would abandon searching a given direction for performance reasons when the delta values in said direction were not yielding results, I searched every pattern type for every pixel for every parameter... etc...
Now, given my computational capacity and limitations of the time, the average seek time per 4KB string before finding a result was 1-2 weeks. All my code was heavily optimized to exploit memory as it was used on the architecture. Consider a combination of Michael Abrash style coding/optimizations with more specific knowledge of the specific CPU and memory it would run on. I also spent a great deal of money getting memory with 60ns response time (though I never measured better than 64ns on the scope). I also used a Micronics motherboard which at the time meant something as Intel had yet to enter the chipset market and as ECC memory was far too expensive... for pretty much anyone on that scale, and the memory controller was not yet part of the CPU, I needed a reliable and documented chipset to work with.
Ok, so here's the summary of all the results
1) It is worth investing greater effort in scientifically proving that absolutely any string can be found in the Mandelbrot Set given enough iterations, resolution, search creativity and of course CPU power.
2) It is believed (though through non-thorough experimentation, not mathematically proven) that any 4KB string can be found in the Mandelbrot Set
3) Once found, depending on the resolution and number of iterations provided as a coefficients for identifying the set, the data can be retrieved in a reasonable period of time (deterministic, though variable) on any device. The real computational cost was the search itself.
4) This method of compression has no value for video as the data sets are too large.
5) It is most common that longer string lengths searched for often requires higher resolutions and higher iteration counts to be found. This is common, not a rule. Depending on the search pattern, often strings with a more level histogram distribution could be found as bit patterns in lower resolutions and iterations.
6) The size of the strings found increased far more rapidly than the size of the coefficients required to represent the find. Meaning without employing additional methods such as entropy coding the
- resolution of the Mandelbrot image,
Re:Anything is possible. Practical though? (Score:4, Interesting)
You can take any irrational number and find any value in it. The problem is that you can't truly compress anything that way, your coefficients will be the same size or larger than any number you're trying to get as a result (the optimum compression can never be better than 2:1). If you're looking for "lossy" compression (basically, some values (but no more than x %) can change, you'd still have to look through the nearly infinite space for possible values and at best you'll get some compression close to whatever entropy allows.
To encode all data that's currently being stored or generated this way by humankind, your encoder would have to work through ~2000 exabytes.
The whole story makes it clearer (Score:5, Informative)
If you read the original stories in Dutch, the whole story becomes a lot clearer.
The system he had was enclosed in a box and you could initially see his "demo" of 4 movies in low quality. There were various claims on Sloot's part that it was only x size but nobody was allowed to look into or program the system. He was going around investors fishing for money to make it work at bigger resolutions for his lossless compression algorithm. When he croaked nobody found anything in regards source code or design documents.
It's also mathematically impossible to get the file compressions he got. At best it was a reference to a pre-programmed movie.
Re:The whole story makes it clearer (Score:5, Funny)
If you read the original stories in Dutch, the whole story becomes a lot clearer.
Personally, I found that reading the whole story in Dutch left me more confused and totally unsure how to take this. I don't read Dutch.
8 kilobyte move (Score:3)
But you need to download a 370 Mb CODEC for each one.
We already have this (Score:5, Funny)
It's not as crazy as it sounds (Score:3)
Two answers (Score:2)
2/ If it is lost then it's technically irrelevant even if historically interesting.
If it's possible, Google's AI will rediscover it (Score:2)
This sounds like a cross between Guetzli, an AI-optimized jpeg encoder, and Brotli, a zlib encoder with a predefined library. And Google loves looking for ways to compress YouTube videos.
Prior art as evidence - Scansoft/Nuance QMAX PDFs (Score:2)
2. We have a commercial entity called ScanSoft that also compresses PDFs
3. PDFs generated by ScanSoft are incredibly much smaller than any other PDFs
4. Scansoft PDFs use a compression algorithm called QMax
5. Nuance which now sells the Scansoft PDF products have access to the same technology
6. No other PDF technology in the marketplace has a compression algorithm that uses STANDARD decompression and unique COMPRESSION to achieve the ridicu
Art and science (Score:2)
Try a resolution from the vcd or vhs years.
Take time to look at every frame in great detail. Keep working on the data in each frame for a much longer time.
Use a good cpu and gpu to really work the math per frame and each next frame, all frames.
Then have an artist correct each frame by hand for movement or areas that should have not moved.
Alter any code as needed.
Try again.
Code could be optimized for a set background and a talking presenter.
$loot compression (Score:2)
Adam Clark was his other name by any chance? (Score:5, Interesting)
Adam Clarks Adams Platform:
https://www.itwire.com/opinion... [itwire.com]
http://www.smh.com.au/business... [smh.com.au]
Now you might think ok, this one was a scammer, but people vet those things, cant fool me twice, right?
http://v-net.tv/2015/10/09/unk... [v-net.tv]
5 years later VERY SAME "The company’s senior development team comprises: Adam Clarke" :)
Adam Clark, of Adam’s Platform Technology (2004) "transfer a 1.3 gigbyte video file to a 1.4 megabyte floppy disk." strikes again in another scam
Another one is Madison Priest's Zekko Corp: :D
http://www.bizjournals.com/sac... [bizjournals.com]
http://jacksonville.com/tu-onl... [jacksonville.com]
http://jacksonville.com/tu-onl... [jacksonville.com]
Magic video compression turned out to be buried cable
Want more video compression scams? Check out V-Nova Perseus - they promise 3x smaller files than h.264, but somewhat independent tests show 20% bigger files at same quality :) and the real kicker is Perseus is really just reencapsulated h.264 video with resize filter on top :D multi million dollar scam, they even scored one Sat TV network contract.
Re: (Score:2)
The real trick is storing the data in two-dimensional bits, so that each bit can store many thousands of times as much data. /sarcasm