How Do You Organize Your Experimental Data?

digitalderbs writes "As a researcher in the physical sciences, I have generated thousands of experimental datasets that need to be sorted and organized — a problem which many of you have had to deal with as well, no doubt. I've sorted my data with an elaborate system of directories and symbolic links to directories that sort by sample, pH, experimental type, and other qualifiers, but I've found that through the years, I've needed to move, rename, and reorganize these directories and links, which have left me with thousands of dangling links and a heterogeneous naming scheme. What have you done to organize, tag and add metadata to your data, and how have you dealt with redirecting thousands of symbolic links at a time?"

Here

[Xamusk]

I store them in first posts.

Totally disorganized

[countertrolling]

Whenever I need to find anything, I use "Command-F"

Organize it with style

[Anonymous Coward]

Organize your data like I organize my bedroom: Everything on the floor.

Look, how big is your desk? 8 square feet? How big is your floor? Several hundred square feet? If you can see all of your stuff, then you can access it instantly. Organized Chaos.

Now, if you'll excuse me... I think something's moving around in my trash can.

Go for NoSQL!

[JamesP]

OK, subject is the short answer, here's the big answer

Since experimental data usually doesn't have the same structure for all experiments, you may try something like this:

at the deeper, most basic level organize it using JSON or XML (I don't know what kind of experiment you do, but you would put lists of data, etc)

Then you store this in a NoSQL db (like CouchDb or Redis) and index it the way you like, still if you don't index you can always search it manually (slower, still...)

four directories

[arielCo]

$PRJ_ROOT/data/theoretical
$PRJ_ROOT/data/fits
$PRJ_ROOT/data/doesnt_fit
$PRJ_ROOT/data/doesnt_fit/fixed
$PRJ_ROOT/data/made_up

Re:Use databases!

[grub]

If the dude works at a research institute of minor size and up, they should have IT staff who can do that initial setup for him.

Re:four directories

[morgan_greywolf]

Oh, come on! Who let the climatologists in here?