Increasing Video Detail Using Super-Resolution?

Cecil Esquivel asks: "I'm looking for ways to increase the quality of video by using super-resolution algorithms which use the visual information across multiple frames of video to increase the resolution of individual frames. I have found very little on the web that can do this effectively for the entire length of video. There is commercial software, VideoFOCUS, which produces hi-res stills from video, but doesn't seem to have a product for producing hi-res video from video. There is a thesis from Duke U. which is 6 years old, monochrome only and is mostly proof of concept.) Anybody out there have more information or is willing to help me develop some software that can do this? Darwin/Mac OS X solution that can work with QuickTime DV, preferred." Typically, super-resolution uses image samples generated from low-resolution and high-resolution samples of the same source, which is then converted into source independent information that can be used to increase detail for other low resolution sources. Has anyone seen programs that use super-resolution techniques for increasing the resolution of your typical digital video clip?

Some deinterlacers do similar work already

[mcgroarty]

Some deinterlacers use the InterCine or Messe filters to approximate the data missing in alternate frames. There are other algorithms as well -- most interpolate the existing data to guess what would have been in the missing area, and some do look at previously known data for the missing area as well as analyzing moving ranges in previous/coming frames to guess where edges or patterns would continue.

Looking at the state of deinterlacing technology and some of the "detail enhancing" resizing filters would be a good area for study.

I'd *love* to see this used to help correct data errors in video streams as well. A DirecTV receiver with this built in would be cake++.

Re:Spatial vs. Temporal

[MoOsEb0y]

Yes, typically there would be. However, if you look at the Duke U thesis, it uses MPEG's own motion vectors to determine when there's motion video (vs. a still scene) and compensate for this. Thus, what it will accomplish, is making still shots look a LOT better (and more compressible!), and in the fast-moving scenes it will do very little (but you won't notice it that much, either).

If my programming skills were better, I've be VERY interested in seeing if this can be done with MPEG-2. Noise originating from the original video has always been my primary gripe with those kinds of sequences. if this could be compensated for and exported to MPEG-4.... MMM that would be tasty.

Re:Spatial vs. Temporal

[NanoGator]

"I imagine increasing the spatial resolution for a frame of footage decreases the temporal resolution of the set of frames."

Not necessarily, though it'd require two passes.

In the first pass, you generate a high res image by using motion tracking to figure out how far the camera turned. Then, using that motion, it can read the sub-pixel data. That's semi easy to do, it's been done before. Before long, you have a high-resolution image.

Then, you do the second pass where you take that high resolution image and paste it on top of the low resolution footage using the motion tracking data to move the new pixels around in the right position. As long as the motion tracking data is reasonably accurate, then you could theoretically create a higher resolution video without losing temporal resolution. It's not clear to me, though, that there wouldn't be situations where that would break down.

Man I hope I expressed that fairly clearly. I've got a little experience with digitally painting video to change the details of a scene.