Ask Slashdot: Software To Organise a Heterogeneous Mix of Files? 254
BertieBaggio writes "I am a medical student at the end of an academic year trying to get my notes organised. I'm looking for a software document organisation system to organise a mix of text notes, journal articles, diagrams and scans. Ideally such a system would permit full-text and metadata search, multiple categorisations (eg tags), preserve the underlying files and be cross-platform (Linux/Windows/OS X). While I'm not averse to paying for such a complex solution, ideally the software would be FOSS so that extension or migration are possible if necessary. Desktop search (eg Google Desktop) probably does 90% of what I want apart from multiple categorisations, which is the feature I'm most interested in. Searching turned up a similar question over at 43folders which pointed me in the direction of Papers and DevonThink, but these are OS X only and seem to be aimed more at academic paper organisation. What recommendations does the Slashdot community have for categorising and organising a heterogeneous mix of files?"
Quick Answer (Score:5, Informative)
Zotero - is awesome - Firefox plugin
Re: (Score:2, Funny)
+1 informative but I have no mod points :(
Re: (Score:2)
I really like Zotero for web research - and if everything comes from the web, it can be great. What I don't like is that, aside from Zotero, there's no real good way to access the files from other software (for example, opening Adobe Acrobat and searching through the tagged files in Zotero to find that PDF that I know I have). In order to access stuff, I need to go into Zotero as my file manager first. Then there's the backup issues, data migration issues, and what happens when my Zotero database is over
Re: (Score:3)
dude, you make some good points but DVDs have been obsolete for like 10 years. A 32Gb thumb drive goes for like 30 bucks these days.
Re: (Score:2)
What is the longest you've had a thumb drive last?
Zotero is hamstrung by it's limits. (Score:2)
I agree that Zotero is awesome. I just wish it wasn't tied so closely to Firefox.
there is an alpha stand alone version of zotero now but that's not what I really want. it's one that works will all browsers and shares a common database.
But my major gripe with Zotero is that it does not work with Pages.app. The zotero folks seem to get inexplicably indignant when asked why they don't support footnotes in pages.app (other than the useless RTF conversion method). They say it's because there is no public API, but any doofus can examine the XML and it's obvious how to extend it from a few m
Re: (Score:2)
I agree that Zotero is awesome. I just wish it wasn't tied so closely to Firefox.
Use it as a plugin in your word processor [zotero.org]?
From The Atlantic's James Fallows (Score:2)
Personal Brain [theatlantic.com].
Gunna hate this BUT (Score:2)
does what you need and then some.
Re: (Score:2)
Sharepoint is utter crap if it cannot understand the file you are putting on it. Which incidentally would be anything not MS Office.
Re: (Score:2)
Sharepoint would do what you are looking for but the advanced features will cost you.
Re: (Score:3)
It has a search function, but it's almost completely useless. I can even put the exact file name I'm looking for and it won't even be in the top 10 results.
The only advantage Sharepoint has over a simple shared file directory is some crude revision control and the ability to create calendars.
Re: (Score:3)
Heh, the only way I've ever been able to tolerate having to use any version of Sharepoint is to open a document library in IE, and then click on some dropdown to change it to explorer view, and then create and right click on a folder and select explore in a new window. Then it opens up in File Explorer, where I bookmark/favorite it so I don't have to deal with the atrocious "information blackhole" Sharepoint web interface, and I can easily drag / drop / delete entire folders using the File Explorer interfa
Re: (Score:2)
Re:Gunna hate this BUT (Score:4, Interesting)
As a SharePoint admin for three years, I can definitely say, without any kind of reservation, that it is utter crap.
Now don't get me wrong, the idea of SharePoint is great. But it is badly designed (the users can't find any document they need) and badly implemented (loosing data is unacceptable).
If you need a document management system, I advice anyone to use SharePoint for a few months and then switch to another system. You'll appreciate your new system so much more that way...
Re:Gunna hate this BUT (Score:5, Interesting)
Re: (Score:3)
Can't find something? Poor design of sites/libraries/lists.
For a simple at home Sp implementation, a couple of doc libraries with appropriate metadata would allow for easy searching.
Oh, and SP Foundation can run on Win7. Doesn't strictly need Server2008.
If you're well tied into the whole MS
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
I am also a Sharepoint server admin and I would never recommend it to anyone.
As has been said already, it has a lot of really good ideas, all executed terribly. Search is so important, yet Sharepoint is very bad it. Yes you can drag and drop a whole hierarchy of files to add them to sharepoint, but woe to you if one of those files has a name that Sharepoint does not like http://blogs.msdn.com/b/joelo/archive/2007/06/27/file-name-length-size-and-invalid-character-restrictions-and-recommendations.aspx [msdn.com]
Re: (Score:2)
I just threw up a little in my series of tubes.
Re: (Score:2)
Sharepoint is hardly FOSS, though. If you want that kind of solution, DSpace is a more logical starting point.
Re: (Score:2)
How about OWL? http://owl.anytimecomm.com/ [anytimecomm.com]
Yes you need to install a webserver on your system but it really does work very well.
OS X - Plain old search (Score:2, Interesting)
I have a Mac and it's not the greatest OS, but I love the search. I search all my old emails and a horde of other documents all the time.
I'm sure other computers can do this just fine, but I was never satisfied with a desktop search implementation until OS X. And I used to be a search index consultant.
Re: (Score:2, Informative)
I second that. You can add metadata (Spotlight comments) to any file, and that metadata is searchable, so if you're thinking of tagging files, that's easy.
an age old solution (Score:4)
Sounds like you're describing a directory tree. Search with grep, or any similar utility. Put files in multiple categories (appropriately named directories) using ln. It's cross platform, timeless, and seems to do what you describe. I feel like I'm missing something though.
Re:an age old solution (Score:4, Insightful)
Re: (Score:2)
No, but Directories/Folders as low level storage (the files have to exist somewhere) and a layer on top of that which can provide metadata tagging, searching, views etc. is. KDE4 with Nepomuk (http://nepomuk.kde.org/) seems to provide much of what the OP asked for.
Re: (Score:3)
Directories/Folders are not a substitute for documents tagged with meta data
Why not? It works for me. It's pretty easy to have a script parse, e.g., your MP3s ID3 tags and link them to the appropriate directory.
Not too mention you can't create views
That's what 'find' is for.
Re: (Score:2)
Because that's not a proper document management solution.
Nevertheless, it can work quite well, depending on your needs. I manage large numbers of files that way, with separate index files handling the metadata. Grep is very fast for simple keyword and pattern searches, even on index files with ~100,000 records, so there's no reason even to use a DBMS (I don't need complex queries). Scripting for special purposes is simple and straightforward, and backups are a breeze, using nothing but rsync. No special-purpose software needed, beyond what comes with the OS.
To ea
Re: (Score:2)
The file manager may not be great but if it isn't up to the task then that really needs that the file manager needs an update.
All sorts of sometimes mutually exclusive bits of software that try to organize one bit of information or the other really doesn't cut it.
If you have a good "primary key" that's generally useful most of the time, you can "get by" even with just a simple file manger. In truth, most n00bs probably aren't industrious enough to really make a good multi-key approach work. It would require
Re: (Score:2)
Directory for each meta data tag. Directory for each view.
Directory FOR EVERYTHING!
I kid, I kid.
thebrain dot com (Score:2)
Emacs Org-Mode (Score:4, Interesting)
Emacs Org-Mode [orgmode.org]. I've learned a little Emacs syntax just to use that package after I've being a Vim user for over 15 years.
Re: (Score:2)
Emacs Org-Mode [orgmode.org]. I've learned a little Emacs syntax just to use that package after I've being a Vim user for over 15 years.
A bit more: Org-mode allows to define text documents with smart headings and lists. You can insert links, equations, store file attached to a heading. It is cross-platform and you can export your documents to, among other options, html or latex-pdf. You can flags items as TODO or attribute a "done" time or a "todo" time.
Re: (Score:3)
Org-Mode files are plain text, but there is extensive support for inline tables/spreadsheets and images and even code blocks. For example, you can keep data from an experiment in a table, do some analysis for which one would normally use Excel, and then plot and display a graph based on the data in one document. That takes care of a lot of files right there, because many tasks can be incorporated into the org file organically.
Another great feature is the Org-Agenda which defaults do displaying date-based in
Does a good one actually exist? (Score:3)
Many times she has spoken to me of the failures of the information-organization software that they have tried. Some would reach a certain capacity and choke, others had terrible OCR, and so on. In fact she even asked me about building a better application to do that; but I was too busy trying to put food on the table to take on a large project that probably would not pay off for a couple of years at least.
If there is an application that does this well, I would like to know about it too. (One person has already mentioned Evernote, but that's a "cloud-based" application for some unknown reason, and I would have privacy concerns.)
Re: (Score:2)
Evernote notebooks don't have to be synchronized to the cloud, you can have local-only notebooks if you have sensitive data. Just make sure to back it up; I use their script engine to perform regular dumps to a network backup location.
Software To Organise a Heterogeneous Mix of Files (Score:2)
There's an app for that. [wikipedia.org]
Re: (Score:2)
Also, to elaborate on your specific requirement for "multiple categorisations", and so that I might save myself from a "smart-ass" mod, here's a possible suggestion: http://www.tagsistant.net/ [tagsistant.net] It's a tagging file system. You didn't specify which operating system you were on, but this works with Linux/BSD. Not quite mature, but I could see it potentially going places. At the very least, the idea of implementing tags directly in the filesystem might trickle up to extfs or NTFS or hfs+ eventually.
Re: (Score:2)
The OP specified cross platform compatibility. Linux/BSD is probably not going to be enough. However, I'd imagine that it would be easy enough to get it running under OSX. I'm not sure how easy it would be to get this running on Windows though.
Personally, this looks like something I'm going to have to keep my eye on as my main OSes are Linux and FreeBSD.
OneNote (Score:2)
This is actually what OneNote [wikipedia.org] -- the oft overlooked/maligned offering from MS -- is designed to do, and it does it pretty well believe it or not. Technically it's aimed at collaboration, but there's no reason it can't work equally well for self-organization.
Re: (Score:2)
This is actually what OneNote -- the oft overlooked/maligned offering from MS -- is designed to do, and it does it pretty well believe it or not. Technically it's aimed at collaboration, but there's no reason it can't work equally well for self-organization.
Looks like that would be a good fit, except for the requirements that it be open source and cross platform that is.
Re: (Score:2)
Right, except those are ideals, not requirements.
Re: (Score:2)
OneNote is available for Windows, Mac and iPhone. The iPhone version is curretly free.
OneNote is part of Office which is available for a 60 day free trial.
OneNote is the best thing since sliced bread for what this guy wants to do. It is especially useful if you need (want) sync'd data on two or more devices.
Evernote (Score:3)
+1 iusethistoo (Score:2)
+1 iusethistoo
Evernote [evernote.com] has clients for OSX, IOS, Android, Blackberry, Windows Mobile, Palm, and the web. OCRs images you put in it. Syncs between your devices transparently. All clients are free but there's a max of like 60M/mo transfer; if you want to exceed that it's $45/year.
I haven't put alllll my data in it and probably never will but for reference material and general notes? It's kinda where everything goes now.
Re: (Score:2)
Evernote saves to the cloud, no backup worries.
You are foolish [businessinsider.com] if you trust your data to the "cloud". Please understand: THE CLOUD IS NOT, BY ITSELF, A REPLACEMENT FOR BACKUPS.
Re: (Score:2)
Everything in your Evernote account is also replicated to your machine, in a set of files you can include in a local backup if you wish.
Google had a solution.... (Score:2)
Re: (Score:2)
It was ahead of its time. It's gone the way of the personal flying car, not the buggy ...
Mendeley Desktop (Score:2)
OpenKM (Score:3)
OpenKM will do most of what you want. We've deployed it for clients who have been happy with it. It does not preserve the underlying filesystem, but you can upload a ZIP file of documents.
It's a tomcat app - that used to be heavy-duty - if it is today depends on what kind of machine you're using.
A system to organize files (Score:2)
Would be a file system. Run something like Beagle for full text/metadata search. Use hard links to keep a single file under multiple folders.
EndNote (Score:2)
They used to be (Score:2)
called secretaries, now they want to be known as administrative assistants.....
I'm old school. (Score:2)
I use VisiCalc because I'm down like that...
Re: (Score:2)
OP here (Score:4, Interesting)
Many thanks for all the informative replies so far. I've had a quick glance at Evernote, thebrain, Nepomuk (I'm loving KDE4 so far after switching a week or so ago), OpenKM and FreeMind and these seem promising. I've still to look at emacs' org-mode, and when I do I will try to put my vi prejudices aside ;-) Some of the other suggestions are rather good but aren't really what I'm looking for as they are either fully cloud-based (eg Google Docs, Wave) or one platform only (eg Sharepoint) or too expensive (hire a secretary :-P).
I like the idea of some of the "roll your own" ideas, eg directories + hard links, serving from a web server or wiki. The problem is as I progress though the medical degree, I am likely to have decreasing amounts of time to tinker with things if they have shortcomings; and to be honest they probably will as I am unlikely to have thought through the problem fully! Plus third-party solutions will definitely have substantially more polish than anything hastily dreamt up by me!
A shared wiki for my cohort / medical school / country may be an option on top of whatever comes out of this discussion, but I'd like something personal as... ah, let's say medical students have wildly varying standards of what is acceptable for notes ;-)
My supplementary questions for anyone still wanting to chip in:
Thanks again for the helpful replies, Slashdot. You continue to impress me - I doubt I would have gotten such a useful variety of responses elsewhere. I hope this discussion is useful to other folk looking for something similar.
Re:OP here (Score:4, Interesting)
Finding specifics versus finding sets (Score:2)
I am a researcher. I want to add my vote for "file system." The less interaction you do with most of this material, the better off you'll be. For me, important or useful material goes into a reference manager. Those files get tagged in the reference manager. At this point in my career--only four years in--that's just under 600 articles with accompanying pages of notes. Other stuff goes into folders based on broad categories. I don't do any tagging on these because find-by-content always does the job just fine. Avoid the extra work. You're not paid to be a secretary. And most of the organizing won't pay off, will become an end in itself.
Thanks, it's good to be aware of the payoff of pre-organising things (a la putting in frameworks in programs that will never need it) before embarking on something like this. You also reminded me of something I wanted to put in my follow up: The reason I wanted multiple categorisations or tags is that some things do fall under more than one category. I'm a medical student, so I'll give you an example: Malaria. Does that come under:
Re: (Score:2)
(ie where was that paper by Doe et al from 2007 on the effects of foo in vivo)
Don't you have a medical librarian for that sort of thing?
I'm a lawyer and I never once have regretted tossing out any piece of paper that can easily be located in the library or on Westlaw. If you save all that stuff, you will spend more time saving it than you spend using it and, even then, it will be harder to find it when you need it than it would be if you just went back to the library.
The most important knowledge winds up in your head or can be found in a few important reference works. For everything
OneNote (Score:2)
Why not use OneNote with either a local folder, a personal server or even a cloud-based Skydrive? It will gobble up everything you have and allow you to search, organize, etc.
Re: (Score:2)
Why not use OneNote with either a local folder, a personal server or even a cloud-based Skydrive? It will gobble up everything you have and allow you to search, organize, etc.
Unfortunately OneNote seems to be Windows-only, and I'm spending ever-decreasing amounts of time using Windows. It's undoubtedly a good suggestion for everyone who does though :)
Re: (Score:2)
You seem to have missed Zotero [zotero.org] (it's up there in the comments somewhere) which is a FOSS plugin for firefox. It keeps an offline database, and for nearly any site (e.g. journals) you click one button on the URL bar and it downloads the citation including full pdf so you can read it whenever. It will also let you perform full text searches of your database, and can be configured to perform OCR on scanned documents. Best of all, it's trivial to make bibtex (or many other formats) bibliographies.
I use tha
Re:Zotero (Score:2)
You seem to have missed Zotero [zotero.org] (it's up there in the comments somewhere) which is a FOSS plugin for firefox. It keeps an offline database, and for nearly any site (e.g. journals) you click one button on the URL bar and it downloads the citation including full pdf so you can read it whenever. It will also let you perform full text searches of your database, and can be configured to perform OCR on scanned documents. Best of all, it's trivial to make bibtex (or many other formats) bibliographies.
I use that in combination with TiddlyWiki [tiddlywiki.com] for personal typed notes not associated with a journal article/textbook, and Xournal [sourceforge.net] for annotating documents and taking notes with my tablet computer. When annotating documents (textbooks, journal articles) just configure xournal as your pdf viewer and you'll be able to save every annotation you make. TiddlyWiki has a ton of plugins to do whatever you need, including a GTD [nathanbowers.com] (Getting Things Done -- it's a book) variant that's probably comparable to Emacs Org-mode, LaTeX math [mcelrath.org] (I wrote that one -- use it every day), and many more.
The one drawback to all this is that I have no way to automatically organize my handwritten notes from xournal. Though they're computer files, my organization for them is horrendous. I still fantasize about some kind of hybrid mutant of TiddlyWiki, OCR (that can magically read my handwriting and equations), and xournal that would let me do all this on a pen-based tablet...
You're right, I didn't include Zotero in that comment! Having looked at it before, my impression of Zotero was that it is a (very competent) reference manager / source collection. Is it more than this? TiddlyWiki sounds familiar too - this is a subject I've dallied with before, so perhaps I came across it previously :-). I'll have another look at Zotero and TiddlyWiki and see if I can get it to do the categorisation thing I'm looking for that I mentioned in my other comment [slashdot.org]. Cheers!
Re: (Score:2)
Zotero does categorization. You have a familiar hierarchy of folders (topics). A single reference can appear in multiple folders. (It's your organization, not the filesystem -- each thing appears only once on disk). It does not automatically create categories for you, however. But frankly, I don't think any software could do that in a way I would agree with... You can further attach arbitrary files and notes to things you put in your Zotero database.
Re: (Score:2)
http://www.branchable.com/ [branchable.com] (or acutally ikiwiki) is a wiki that you can work on offline. And there's vim-outliner if you don't like emacs orgmode.
Re: (Score:2)
Offline mode is one of the things Evernote uses to distinguish the paid version from the free version
Re: (Score:2)
Try out KnowledgeTree (Score:2)
"Papers" and other metadata thoughts (Score:2)
A friend recommended this to me: http://www.mekentosj.com/ [mekentosj.com] I've played with it a bit, and it's very academic-focused.
There's probably a need for a more general metadata-integrated information management tool, that makes use of Mac OS X facilities for metadata definition and management. Do Linux and Windows 7 have similar OS level facilities to support metadata creation and management?
A key consideration is the ability to store metadata in the file, rather alongside. Some EXIF tools support this for photo
the cloud (Score:2)
OCR,pdf annotations,text/html,pdflatex + recoll (Score:3)
If you insist in keeping your notes (think before how often you will look at them, then the above tools may work for you.
(recoll is based on xapian and works very well if you have big static archives which you don't need to index often.)
Shoebox 2.0 (Score:3)
Save yourself - go on a date.
Xena (Score:3)
You might also want to look at an xml normalizing tool like Xena [sourceforge.net] - automagically converts all your docs, files, etc into open formats whose content can be searched by open tools.
If you're this close to the end of the term... (Score:2)
... quit dicking around trying to find software and start studying. Use whatever break you have before your classes start again to get your organization system in shape. Frankly, you're about nine months late on this quest and your faith in computers and your own tagging abilities are, if not misguided, pathetically touching. Good luck with your medical career.
Two projects to look at (Score:2)
How to use Devonthink Pro (Score:3)
This isn't your ideal solution—as you've noted, DTP is currently OS X only—but it does work pretty well for me, especially when I'm thinking about a general topic and need to find information on it. I even wrote a post [jseliger.com] about the similarities between Joyce's method of composition / finding material and how Johnson uses DTP.
Tracker (Score:2)
Tracker ( http://projects.gnome.org/tracker/index.html [gnome.org] ) is relevant to your needs. It stores relational information about your data. So for example, if you have an id3 tagged mp3 album, it will store the artist as an entity x, and the album as an entity y, and it will store that each of the mp3 files are a part of y, by x. In the same way, it stores authors and publishers as entities for pdf's. You can address this data store in the ordinary `desktop search' way (there are some search GUI's available), but
Shouldn't Nautilus be able to do this? (Score:2)
Database (Score:3)
Via RDBMS, I've done similar things with info I had to use extensively. I parsed the files, tracked the paths, classified sections (as best a machine could), added category tagging, word indexing, etc. I'm not called "Tablizer" without a reason.
Note that it was for personal use, but could be extended to others with a fancier UI and less abbreviations.
KT, Alfresco, OpenDocMan, various other DMSs (Score:3)
You might consider exploring http://www.knowledgetree.com/ [knowledgetree.com] (commercial open source) or another DMS like the following:
Alfresco - Open Source Enterprise Content Management (CMS) ... - alfresco.com ... - openkm.com ... - epiware.com ... - ademero.com
OpenKM - powerful, easy to use, web-based scalable electronic
Epiware - Document Management Solutions for Everyone. A powerful
Document Management Software - Your Search for Document
http://www.opendocman.com/ [opendocman.com] (PHP)
http://en.wikipedia.org/wiki/Category:Document_management_systems [wikipedia.org]
Dolphin (Score:3)
KDE's Dolphin file manager, coupled with Akonadi and Strigi (built-in, and seamlessly integrated) does everything that you are asking for. It runs best on Linux, but there are KDE Windows and Mac ports. Of course, that means that you must install all of KDE on that Winbox or Mac.
Note that in the past there was much criticism of Akonadi due to resource usage, but that has been taken care of for at least two major version numbers (KDE 4.5 and 4.6). Let us know how it works out for you, you are really going to enjoy the tools that KDE and Dolphin offer for file management and organization.
Wikipedia, "reference management software" (Score:3)
Comparison of reference management software
http://en.wikipedia.org/wiki/Comparison_of_reference_management_software [wikipedia.org]
Storing/Indexing/Tagging vs. Searching/Finding (Score:3)
Re: (Score:2)
Re: (Score:2)
In the Open Source world, DSpace is probably the document management system to beat.
Re: (Score:2)
Re: (Score:3)
Weird, certainly. You could sort, search and store documents in the way described using Gopher or WAIS long before HTML was even invented.
Re: (Score:2)
There's nothing wrong with "organise."
Re: (Score:2)
Learn how to spell in English, not just American.
Re: (Score:2)
now you just need a handy front-end :-)
- unless you enjoy the subtle beauty of the SQLplus command line.
Re: (Score:2)
Oh, yes, this. Brings back fond memories of a troll extraordinaire.
Re: (Score:2)
[citation-needed]
Re: (Score:3)
Re: (Score:2)
It's ok to take seriously. It'll be done by 2150, when the Daleks will invade the Earth. (The Daleks are too smart to invade a disorganized planet.)
Re: (Score:2)
Indeed, I found Zotero extremely valuable to manage papers (especially because you can add them directly from the web site with a single click), but I don't use it for anything else (my LaTeX files are under version control and organized using traditional directories, my notes are mostly in Tomboy [except for those more complex which I do in LaTeX, and of course those I do on paper], my mails are on the mail server [which I don't access through the web interface if I don't need to], any self-written program
Re: (Score:2)
No actually it went from 'good' to 'excellent'.
Re: (Score:2)
library scientist
Eh????