A Distributed DivX Ripper? 32
RJ asks: "I know much about Java/C++ and
sockets programming and I'd like to use this knowledge to build a distributed program to rip a DVD into DivX. It will work by breaking the job into chunks, sending chunks to other computers to encode, then patching it back together.
Despite my searching efforts, I have been unable to find a decent resource to teach me how to program the DivX core to encode mpeg2 and re-join parts. I'm hoping that some readers of slashdot can point me in the right direction?"
I'm not an expert, (Score:3, Informative)
- Reading around 4 gigs of data in a fairly fast time: not much of an issue on a normal computer, but distributed computing might introduce issues with speeds of the network.
- Deencoding the mpeg2: this seems to me to be the biggest slow down. The problem seems to one of not having a fast enough decoder that is capable of good image output. Distributed computing would help, but it seems to me to be somewhat of a macgyvered solution to the real problem.
- Resizing, cropping etc: not much of an issue, but of course distributed computing would help.
- Mpeg4 encoding: Again, one of the major problems seems to be with fairly slow encoding alogarithims. Therefore, helped by distributed computing.
I'm sure setting up a beowulf cluster of computers or some other setup seems to be nice and all that, but unless you're reencoding several dvds a day, it seems like a normal athlon would probably do the trick, and working on the code for better decoding and coding of the various codec would seem to me to be a more efficient way of improving speed. I know that I can reencode a dvd at half real time with 1-pass divx 5 encoding and flaskmpeg, and a 1/4 real time with 2-pass encoding, and I've only got a Athlon 1.4. So unless you are reencoding Das Boot quite a bit, it seems to me like one normal high powered consumer computer would be plenty sufficient.Re:I'm not an expert (Score:1)
Please, it's "decode," not "deencode."
Like my stupid high cshool programming teacher who yankee-ily kept saying "disenable," when it's really "disable," you don't have to include BOTH positive and negative prefixes -- only one.
Anyway, you're pretty much right. Though I think the original intent was a bit more distributed than a cluster (wan), since such a thing (with oh, just a couple athlons) is beyond most would-be rippers, I'd assume.
However, if you had a good sized pool of cohorts, and they all had cable modems that can sustain 100k+ between them boh ways, then the networking part seems more feasible than one might at first think. But it's not something I'd release and let and old Joe 56k try, of course, but I wouldn't (personally) consider it McGyver-ish.
Distributed encoding... (Score:3, Interesting)
I ran into a snag.
It seems the encoder I was using at the time (bladeenc) was inserting silence at the end of each mp3, to keep it to spec. What I can imagine is that even with DVD encoding, you'd need a "master" that would give out file chunks to the worker bees. But -- it would have to be intelligent enough to know when you wanted a new keyframe, and split up the
In other words, you may as well just build one heck of a fast machine, and try to get 30-40fps encoding out of it, rather than try to put together something to distribute it and encode it. That's my 2 cents, and I may be wrong....
Dedicated audio encoder (Score:1)
It seems the encoder I was using at the time (bladeenc) was inserting silence at the end of each mp3, to keep it to spec.
Dedicate one machine to running LAME [sulaco.org] on the audio, and you won't have this problem.
Re:Distributed encoding... (Score:1)
The solution is to cut your .wav files to the correct size, and all will be fine.
dvd::rip has a cluster mode (Score:5, Informative)
Watch this post get modded up, and not my qualified response [slashdot.org] to the From Coder to Game Designer [slashdot.org] question. Humbug!
Anyway, as brought up in the last Ask Slashdot remotely similar to this one (Archiving DVD's with Linux [slashdot.org]), dvd::rip [exit1.org], which is a Perl+GTK front-end to transcode [uni-goettingen.de], has a fairly insecure cluster mode [exit1.org], whereby it will split up the video transcoding task among however many machines you can coerce into doing it, and rip and mux the audio with the video on the host machine.
Sounds like just what the doctor ordered. Now someone go mod up that other answer of mine. Please?
Re:dvd::rip has a cluster mode (Score:2)
However, I've found that the XviD [xvid.org] codec is fast enough to encode 2:1 on my P3 866. That is, it takes 2x as long to encode as the movie lasts - it gets around 12fps. DiVX 4.02 put out slightly better quality, at 4-6fps on the same system. I haven't done anything with DivX 5.xx yet, so I can't speak for that.
Er? (Score:2, Funny)
Re:Er? (Score:2)
You are technically right, the DMCA doesnt exist outside the USA, but laws just like it do.
Distributed Encoding (Score:1)
why use sockets ? (Score:2)
in PVM or MPI. I do not know if
"mpi_allgather" and "mpi_allscatter"
would stand an 2gb array like found on DVDs,
but at least this would put several 1M$
beowulves I know of to a somewhat useful
purpose (besides cracking
sniffs, of course), instead of boring
quantum chemical computations or climate simulations.
Re:why use sockets ? (Score:2, Interesting)
This already exists: Vidomi (Score:5, Informative)
This answer your question?
Re:I have a better idea (Score:3, Insightful)
Did you read the same Ask Slashdot that I did? There's nothing in his question that even remotely hints that he's copying someone else's DVD instead of doing this to DVDs he already bought.
Maybe that's what he's doing, but you're really jumping to conclusions. It's sort of like when someone asks, "Where can I buy a screwdriver?" and you fly off the handle that maybe he should stop stabbing people with screwdrivers.
Re:I have a better idea (Score:1)
Ever heard of google? Search before you ask! :)
Re:I have a better idea (Score:2)
I don't think it's fair to compare this to Napster. The non-infringing use of this these tools is far greater than it was for Napster. This audio equivalent of these kind of tools would be mp3 and vorbis encoders, not trading software like Napster.
You don't seem to understand that some people (e.g. me) really don't like those shiney discs. They get scratched, possibly lost, and once you have a few hundred of them, they are inconvenient to physically manage. Compared to files on a drive array, they are as hard to use as stone knives and bear skins. I don't even play my new audio CDs once; they get ripped and ogg encoded, and then they go into a box, where I don't know if I'm ever going to access them again. (Time will tell, I guess.) And no, my resulting collection is not shared.
Eventually it's going to be like that for my movies too (although at present, the storage demands are still a little high for me, and I also think DivX is too lossy for action scenes). This technology has very substantial non-infringing use, and the comparison to Napster is unjustified.
Breaks compression? (Score:1)
Load balancing (Score:2)