A/V Data Collection Using Linux?

Simon D. Levy asks: "I am consulting in a biology lab that collects data on fruit-fly mating behavior, and we would like to migrate our data collection to a Linux box. Our primary concern is being able to digitize incoming audio signals (at 8kHz sampling rate), but we would also ideally like to be able to monitor the flies' behavior using a digital camera connected directly to the computer. Crucially, we need to be able to isolate the audio component of the signal, as well as having access to an API (C/C++ would be nice) that allowed us to start and stop the recording. Any experience that anyone has had with this sort of problem would be much appreciated. We haven't bought anything yet, and are looking to build this system (CPU, video camera, A/D cards) from the bottom up. Thanks!"

Maybe not under Linux

[trollbot]

In my experience, digital camera support under LINUX has been next to non-existent. Either the USB is flakey or the necessary software is too buggy (read: pathetic) to use.

As much as it pains me to say it, I think a quality WINDOWS XP system may be more up to the job. Don't get me wrong - I love LINUX - but this is just one (of many) areas that could do with improvement.

When is a troll not really a troll?

[Anonymous Coward]

When it tells the truth.

An XP machine would actually have better support available for all the types of devices and functionality that you need for your data collection.

Fruit Fly Porn?

[jechoe]

Hey, 'bout time we saw some insect action on the Spice Channel!

Re:Maybe not under Linux

[joekool]

why exactly would you be using a usb camera(almost always crappy) as opposed to a a reasonably cheap, well supported capture card, and a cheap camera? oh, and avifile [sourceforge.net], for the requested C api for controlling the recording.

Look within...

[bscott]

If you're part of a university, this project sounds tailor-made for a senior project by some EE whizkid. Just draft some geek and bribe him with promises of interships, and give him an unrealistically short deadline.

If you're in the private sector, it's a totally different story. You must summon the civic-minded core of your being and seize the opportunity to help the next generation. Give some deserving, hardworking American student the real-world experience that will provide them with an edge in the competitive job market of tomorrow. In return for the design and construction of your whole system, you will provide an invaluable internship opportunity that will give a real shot in the arm to their resume! And you'll hardly charge them anything.

Either way, you get your gadgetry, and everyone's happy.

Medical software

[fille]

I used to work for a firm that made medical software. They had a system where the signals of a bunch of brainprobes *shrudder* and high-quality video were streamed to a central server. It was quite impressive. I remember they used MPEG-2 hardware compression and a custom driver. Maybe you can find an MPEG-encoder with Linux support? I think the vendor of the encoder provided some C++ code to make the thing work.

Yes, but how do THEY feel about this?

[Futurepower(R)]

Once, just once, I would like to see a fruit fly turn to the camera and say, "Can't you see we need some privacy?"

Re:Yes, but how do THEY feel about this?

[red_dragon]

Why stop there? Just put an X-10 wireless camera in a corner somewhere and they won't notice...

Video4Linux, OSS/ALSA

[ikekrull]

This stuff is pretty trivial under linux with even the most budget hardware.

Take a bt848 card and a camera and you have good quality video in, easily accessible via Video4Linux with a large base of existing video manipulation tools that let you pipe the video data around, manipulate it in realtime, compress/decompress in realtime etc. etc.

OSS/ALSA make it easy to record sound, and again there is a huge range of open source projects that should be either directly applicable or easily modifiable for your purposes.

There is some specialised hardware for hardware compression, though depending on your needs an Iomega Buz or similar can be had for very few dollars and provides hardware MJPEG compression, and is well supported under Linux.

Its not hard to do this stuff under Windows either, but for the sort of project you have, Linux is ideal. You should be able to get something up and working in a couple of days.

Re:Video4Linux, OSS/ALSA

[ikekrull]

Whats wrong with BTTV's grabbing facility?

I used it to great effect to produce high quality claymation.

Try OpenCV

[glueball]

There is a email group called OpenCV that is sponsored by Intel that will allow you to do what you want with some decent software. I don't think they have anything for sound but rather just imaging and linking with a camera under various OS's

Search the yahoo groups for OpenCV.

More detail on OpenCV and its scope

[myelin42]

OpenCV is Intel's Open Source Computer Vison Library [intel.com]. It's a decent-sized library that helps out with image processing tasks. It's for image processing though - you still have to work out how to capture the video yourself. For that, use V4L (see other comments). One of the included example apps ('CAMSHIFT') includes code to do video capture, but only works under DirectShow on Windows.

If you want to do stuff like extract image features or track moving objects in the video stream, it'll be perfect.

The code is available from the 'opencvlibrary' SourceForge project [sourceforge.net] under a BSD-style license.

Re:More detail on OpenCV and its scope

[glueball]

OK, fine, it's for processing. But there are a lot of questions just like the "how do I capture under Linux" going through the list.

If the person is lost, I'd just send him to where other people doing the same thing hang out.

Totally doable in linux.

[Mysticalfruit]

Instead of using a usb based webcam, I'd go with a video capture card personally.

Under every OS, webcams suck...period. If this is something where your going to take the MPEG stream and make a VCD archive out of it, your going to want something that creates a quality MPEG stream.

For apps for video check out http://www.exploits.org/v4l/

As for the sound, "try dspspy".

Hope that helps!

I would think . . .

[acceleriter]

. . . that there are already a number of people here on Slashdot with experience in conducting video and audio documentation of, um, mating behavior. Applying that experience to fruit flies might take a little work, though.

You might start here [x10.com] for equipment.

National Instraments

[trixillion]

LabVIEW a little pricy, true. But the lab will be able to change their software and analysis on a whim. It is a very powerful programming suite and well suited for this type of application. If you ask, you will probably find that they already have copies of it around, or that half the labs in the building do.

Beware Sound Cards

[samjam]

Beware sounds cards, 8KHz is often not quite 8KHz and it might matter; esp if you wish to correlate recordings made simulaneously from two cards, or from a sound card to a video card.

You may find that 30 mins of recordings from each card are not the same length.

You may prefer to take continuous small blocks of samples and timestamp them yourself, but I don't really know if this would help.

For a real sound card expect to pay over a hundred dollars; but you may find it cheaper to add a trimmer capacitor to the crystal oscillator circuit on the sound card and calibrate yourself.

This is the problem.

[SuiteSisterMary]

"Hi, we do something. We're probably already doing it. We've made a decision to switch platforms. Now that we've decided to do that, we should figure out if the platform we've chosen can actually do what we need it to do."

Audio/Video under Linux

[da0g]

The audio aspects are fairly easy. Any decent sound card will suffice. The Audio API permits you to buffer input/output... But you probably want to use a pre-allocated non-swappable memory block to make sure nothing is lost when transferring to disk. Also, I'd suggest going with a multi-threaded approach.

Be aware that specific sound cards may limit your audio frequency sampling rates. Furthermore, these sampling rates may not be too accurate, and may vary somewhat based on the characteristics of the card, the manufacturer, the current temperature, the phase of the moon, etc.

The Linux kernel runs at 100 Hz. So you are looking at 10ms slices, which may impact your temporal resolution issues.

For video, webcams are useless. (Low resolution, low frame rates, painful interfaces.) If you want still pictures, try gphoto. For motion pictures, you will need some sort of NTSC (American TV standard) video capture, from a video camera. Given the extreme bulk of this video data, you will need some form of compression during the capture operation: JPEG, MPEG2, etc. There are a number of Linux based solutions. The matrox marvel G200/G400 (NOT the G450, NOT the Millennium) is one that I've played with, and has fairly good Linux support.

Be aware that these forms of compression (JPEG, MPEG) will limit your visual frequency range in the frame (field). Rapidly changing data will be lost. (For example, alternating dark/light bars every other pixel.) In practice, this means that minor-ghosting is filtered out. But for your experiments, working with fruit-flies, this may have negative consequences.

Your simplest solution might be to obtain a Replay or TiVo unit, or even a VCR, and use that for the video recording. Possibly interfacing the remote-control into a Linux-based software application.

Drop me a line if I can be of further help... ([e-mail])

components

[joekool]

well, two basic parts: hardware and software
first of the hardware is pretty independant of the os, just pick the highest quality capture card you can afford that is supported in linux, add a reasonable camera, sound card, and a nice modern machine of any kind. I use a bttv card, which works well for me, but of course it depends on your exact needs, and the camera that you hook up to it. As to software:
SDL [libsdl.org], mostly used for game programming, but has some of the capabilites needed for capturing, sound recording, etc.
v4l [exploits.org] the basic component of all video type stuff under linux-you can see the list of crap it supports there, but it doesn't really have facilities for actually capturing to something like mpeg.
avifile [sourceforge.net] everything you want in a capture API, will let you output to all kinds of formats.
mplayer [mplayerhq.hu] I have heard that they support capturing now, but haven't used them for that, but is what you will be using for playing back the files you capture.

My recommendation would be to use the VCR [stack.nl] project, and one of these [freshmeat.net] for audio. VCR uses avifile to record the video in your chosen format, and will record the audio also, but if you want seperate files for audio and video, it is simplest just to use the mic in with another program.

Let me know if you have problems--it's remarakably similar to what is already done to record television programming, with higher quality requirements, so you should be able to take advantage of all the PVR projects out there.
Or you can let me do it-send me the requirements, 1500$, and I'll send the box back a week later
;-}

C++ API

[dnxthx]

(shamless plug warning). If you're looking for a multithreaded C++ API for video capture and display (along with hooks for Intel's Image Processing Library or IPL), check out SPU-Toolbox. (http://www.spu-toolbox.org).

try it.

[BenTheDewpendent]

get a cheap test bed together.
a video card with a tv tuner and inputs
a sound card. a fairly heavy install of your fav distribution and start tring what you need todo.
you can talk about it til your blue in the face.
you can think about it until your head expoldes.
or you can do it and see what you come up with on your own and then migrate over to it once its done.

you can do anything with linux. its just how much time do you have to write the code you need to make it do it... and linux has lots of audio tools. linux has lots of and lots of programing tools. linux has some video tools too. i dont see whats slowing you down.