Increasing Video Detail Using Super-Resolution?

Cecil Esquivel asks: "I'm looking for ways to increase the quality of video by using super-resolution algorithms which use the visual information across multiple frames of video to increase the resolution of individual frames. I have found very little on the web that can do this effectively for the entire length of video. There is commercial software, VideoFOCUS, which produces hi-res stills from video, but doesn't seem to have a product for producing hi-res video from video. There is a thesis from Duke U. which is 6 years old, monochrome only and is mostly proof of concept.) Anybody out there have more information or is willing to help me develop some software that can do this? Darwin/Mac OS X solution that can work with QuickTime DV, preferred." Typically, super-resolution uses image samples generated from low-resolution and high-resolution samples of the same source, which is then converted into source independent information that can be used to increase detail for other low resolution sources. Has anyone seen programs that use super-resolution techniques for increasing the resolution of your typical digital video clip?

Some deinterlacers do similar work already

[mcgroarty]

Some deinterlacers use the InterCine or Messe filters to approximate the data missing in alternate frames. There are other algorithms as well -- most interpolate the existing data to guess what would have been in the missing area, and some do look at previously known data for the missing area as well as analyzing moving ranges in previous/coming frames to guess where edges or patterns would continue.

Looking at the state of deinterlacing technology and some of the "detail enhancing" resizing filters would be a good area for study.

I'd *love* to see this used to help correct data errors in video streams as well. A DirecTV receiver with this built in would be cake++.

Re:Some deinterlacers do similar work already

[Directrix1]

The duke U thesis has C source code to implement this. I don't understand what the problem is.

Re:Some deinterlacers do similar work already

[vadim_t]

Just try to read the article. "...which is 6 years old, monochrome only and is mostly proof of concept". Turning this into something usable would take a good skill and quite a lot of time that he very possibly doesn't have.

Re:Some deinterlacers do similar work already

[Directrix1]

Bah, just process the RGB channels seperately and recombine. And also write image importers/exporters (or use appropriate codec) to get the images in there. Even a lowly person with a CS degree can do that ;-).

Re:Some deinterlacers do similar work already

[ces3001]

I emailed the author... he said that getting the settings and parameters right was an "art-form in of itself." There are also significant optimization issues with this algorithm. It really would need to be reworked from scratch... but serves, I suppose as a point of departure.

multiple sources?

[chocolatetrumpet]

This might be a slightly related note -

I know many web sites have a video available in multiple formats; i.e. real, mp4, and sorenson, or whatever.

I'd like something that takes the different streams and assembles for me a high res final product, based on the idea that there are going to be different details available in each compression scheme... does this make sense? Does it sound like a reasonable idea?

Re:multiple sources?

[t]

Maybe, maybe not. The problem is that most video compression schemes depend on the DCT. Thus the artifacts and such are similar, in other words those different formats are somewhat correlated. If the correlation between the artifacts is stronger than the signal then all you will be doing is amplifying the artifacts.

Think of a badly compressed video with lots of blocky shapes, combine several like that and you'll probably end up with something resembling a brick wall.

I've thought that a better method wou

Next generation imaging technology from X-10

[VisorGuy]

Just imagine, you too could have a Beowulf cluster of X-10 cameras...

Spatial vs. Temporal

[Wavy]

I imagine increasing the spatial resolution for a frame of footage decreases the temporal resolution of the set of frames. My gut feeling is that you would see more after/pre-image than in the source.

Re:Spatial vs. Temporal

[ArmorFiend]

I think there's a sort of heisenberg uncertainty principle for video. Of course I have no proof either, just, as you say, a "gut feeling".

Re:Spatial vs. Temporal

[Mr Z]

Kinda, but not quite. You are bounded in your available information content. Total information is given by "spatial information * temporal information." If you increase one, you decrease the other, at least if you hold the bitrate constant.

You're right that it's like Heisenberg's Uncertainty Principle, in that its formulation is similar. (delta-X * delta-T == constant, so as you make delta-X smaller, delta-T gets bigger.) The difference is that Heisenberg's Uncertainty Principle is based on a single

Re:Spatial vs. Temporal

[MoOsEb0y]

Yes, typically there would be. However, if you look at the Duke U thesis, it uses MPEG's own motion vectors to determine when there's motion video (vs. a still scene) and compensate for this. Thus, what it will accomplish, is making still shots look a LOT better (and more compressible!), and in the fast-moving scenes it will do very little (but you won't notice it that much, either).

If my programming skills were better, I've be VERY interested in seeing if this can be done with MPEG-2. Noise originating from the original video has always been my primary gripe with those kinds of sequences. if this could be compensated for and exported to MPEG-4.... MMM that would be tasty.

Re:Spatial vs. Temporal

[NanoGator]

"I imagine increasing the spatial resolution for a frame of footage decreases the temporal resolution of the set of frames."

Not necessarily, though it'd require two passes.

In the first pass, you generate a high res image by using motion tracking to figure out how far the camera turned. Then, using that motion, it can read the sub-pixel data. That's semi easy to do, it's been done before. Before long, you have a high-resolution image.

Then, you do the second pass where you take that high resolution image and paste it on top of the low resolution footage using the motion tracking data to move the new pixels around in the right position. As long as the motion tracking data is reasonably accurate, then you could theoretically create a higher resolution video without losing temporal resolution. It's not clear to me, though, that there wouldn't be situations where that would break down.

Man I hope I expressed that fairly clearly. I've got a little experience with digitally painting video to change the details of a scene.

Wow..

[dr ttol]

That sounds pretty interesting. Typically any software that deals with movies tend to reduce quality or at most keep it the same. Is this the first time software can actually be made to dramatically increase the quality of a movie, even if the source sucks?

The script... is $#!+.

[yerricde]

Is this the first time software can actually be made to dramatically increase the quality of a movie, even if the source sucks?

Image processing systems have contained "noise reduction" processes for a long time. However, no software can save a bad script.

Surely this should be evident

[Anonymous Coward]

There is commercial software, VideoFOCUS, which produces hi-res stills from video, but doesn't seem to have a product for producing hi-res video from video.

You can't figure out why that is?

Re:Surely this should be evident

[benjamindees]

You can't figure out why that is?

No. Why don't you explain to us the magical quality of the universe that prevents us from doing that.

If you can enhance a single frame, you can enhance all the frames and re-encode an enhanced video.

algorithms which use the visual information across multiple frames of video to increase the resolution of individual frames.

Isn't that what the poster asks for?

Why this is possible

[0x0d0a]

It looks like you're claiming that this isn't possible. Technically, you're right --- you can't necessarily get bits from the air. However, you're ignoring the fact that video contains a good deal of redundant information. Video codecs try to avoid including redundant information this, but only do so in the simplest ways.

Suppose I show three consecutive frames of an eggbeater. Now, if you knew nothing about this video, if the frames had no correlation between each other, and were just each random stati

Use VideoFocus

[nomel]

from the videofocus website:
http://www.salientstills.com/product/pro duct_intro .html

Increased editorial choice: VideoFOCUS digitizes every single frame. You choose exactly the right shot from a video sequence to capture the perfect moment.

So...just run it through video focus, make an avi with all the frames, add the original audio track, then compress them using some video compression...you have yourself a high res video! :)

I'm guessing something will look strange, like funny blurs, or background motion or something. Who knows. I don't think it would do good with fast moving objects in the video...probably blur it real bad.

It's worth a try though. :)

Re:Oh yea. I've seen this.

[MoOsEb0y]

Okay. Let me debunk you. What you are describing is impossible. You can't get more "hidden" detail out of any image. What the poster of this article is describing, is using additional detail from nearby frames in a video sequence to gain additional pixels. In effect, a higher resolution image is created because the amount of data is there. It just needs to be juggled properly.

This principle applies to astronomy. If you want to find faint galaxies, you simply increase the exposure time for your video device

Re:Oh yea. I've seen this.

[hawkstone]

I realize the root was probably a troll (the AC, not you). Also, the submitter was technically asking about nearby frames. Realizing that, however, I'll respond anyway, because I feel the technique is at least worth a mention:

There is nevertheless a form of super-resolution which works on standalone single frames. It depends on what you mean by "additional detail". Normally when magnifying images you would use nearest-neighbor, or better yet, bilinear interpolation. But a magnification using these met

Found some more information

[MoOsEb0y]

Ok, after googling for a little bit (I will never MSN anything!), I found a site who has done just what you were describing with MPEG video.

http://www.ai.mit.edu/~brussell/research/sres/data / [mit.edu]

Anyways, it seems that without proper filtering, the output looks REALLY weird. (look at they guy walking in a circle in front of the quilt) It seems that the motion vectors from the MPEG get taken in as part of the detected edges somehow! Thus, this would be most useful for uncompressed analog video as an input.

VideoFOCUS may already do that

[Tyrdium]

VideoFOCUSLite is a limited edition of Salient Stills VideoFOCUS software. It is an affordable way to enjoy the full capabilities of Salient Stills patented image enhancing features. It does NOT offer several features of the full version including : DV capture, multi-session video capability, various workflow enhancing floating menus, free-hand elliptical and inverting masking tools, advanced preview management, improved video deinterlacing, ability to create a new video movie from a selection of video fram

This isn't much help, but....

[nzyank]

When I worked at HP (R&D) we had some pretty cool stuff for doing very similar things. Most of it came from Purdue or Palo Alto and then we made it actually work. All patented as fast as we got them working and sometimes sooner. Betcha you're going to run into that in real life. Even if you think of something 'novel' the patent's probably already filed or granted for what you're wanting to do.

Covered on slashdot last year

[grundy]

Here [slashdot.org]. Granted this is about creating high resolution stills, but read the ideas in the /. comments.

What kind of video?

[poptones]

To what are you applying this? If you are trying to make high rez from bad video caps or a DVD, that's a whole 'nother animal than astronomy.

I've been working on this for some time and have had a decent amount of success using multiple captures of digital broadcast (broadcasts are still generally encoded on the fly, so each stream will be different). This can also be applied (with great success) to analog captures of broadcast video. Just sum each capture with equal weighting and you'll be surprised how mu

Visar

[Elm Tree]

What you're looking for can be done by VISAR as displayed here [exn.ca]
Although, as it's apparently only available though Intergraph I'm sure it's rather costly.