Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
Education Software

Building and Maintaining Large, Collaborative Databases? 32

Posted by Cliff from the software-for-large-scale-jobs dept.
hherb asks: "We are in the process of building and maintaining a free pharmaceutical reference database, in order to liberate medical decision support systems from vendor driven databases. For that purpose, we need some way of allowing multiple authors to contribute to a large number of data records - most of them small...too small for CVS. We need version tracking as well as authentication of authors. We need to tag every bit of information enterrred with information about source reference, author, peer review result, and so forth. I had a look at existing version tracking software. like CVS and Subversion, and I did not have the impression that any of them would suit our needs. Does anyone have ideas for *free* software solutions that we can use?"
This discussion has been archived. No new comments can be posted.

Building and Maintaining Large, Collaborative Databases? More

Building and Maintaining Large, Collaborative Databases?

Comments Filter:
  • I am assuming you are looking for a "Free as in Beer" solution. If one of the standard VCS solutions will not work for you, you are stuck writting your own ontop of MySQL or something like that to handle the small records you are looking for.

    I guess I don't understand why CVS won't work for you, I assume you create a record for each drug/whatever and then track them... So you end up with a flat directory structure that holds all of 1,000,000,000 records. If CVS won't work for you maybe you should look i

    • What I don't seem to have made clear enough that this is about highly structured data that needs to be modified by hundreds of health professionals with little to no knowledge from a multitude of different platforms.

      Hence, we need a web interface and cannot use a more suitable tool like a XML editor which would enforce compliance of entered data with a DTD.

      For medicolegal and other reasons, we need to be able to backtrack which junk of information has been modified by whom and when.

      Currently, as you can
  • Wikipedia [wikipedia.org] is a very nice example of a collaborative, moderated, reference site.
    • Second the recommendation to use a Wiki. There are Wiki engines that support user authentication and history. Various backend storage mechanisms are also supported, from flat text files to full-blown relational databases. Take a look at the Wiki Clones Directory [python.net]
      • I'm not sure I agree. But I do share your enthusiasm for the Wiki concept. And the best way to understand that enthusiasm is to read (or rather use) the authoritative book on the concept [amazon.com].

        (Warning! A lot of remainders vendors have cheap copies of this book, even though it's still in print. The CDs on these copies are screwed up, but are usable if you know to convert the Perl scripts from Macintosh format!)

    • Beyond Wiki, I would also suggest everything2.com. The engine for that site is at everydevel.com. everything2 is a project of Nate and Hemos, among others, who developed most of this site. The engine seems flexable in my use of it, and provides a mechanism through which similar pages are linked based on user search patterns. Adding a little for source authentication should be minor. Best of luck.
  • Looks like you are looking for some kind of front end to the database. Fairly simple, but you will probably need to write it yourself if noon-techies will be doing the entry. Frankly, I don't see why you need to poll the Slashdot community on how to write a simple front-end to a database.
    • This is about collaborating on highly structured data. We can't require users to run specific software, it must run via web interface. In fact, currently we are using PostgreSQL as backend and a simple web interface as frontend.

      The problem is the version tracking - assigning authorship to the differences between records - since no matter how normalized a record (table row) is, it might always have a number of authors. OTOH, this tracking/auditing must not obfuscate the record text (inline tagging not feasi

      • The problem is the version tracking - assigning authorship to the differences between records - since no matter how normalized a record (table row) is, it might always have a number of authors. OTOH, this tracking/auditing must not obfuscate the record text (inline tagging not feasible)

        Alas, not as simple as it looks.

        Unless I'm missing something, it should be simple. I know there are other solutions out there but one solution I would suggest is like so:

        • User enters/modifies information
        • User submits

  • DIY (Score:2)

    by pmz ( 462998 )
    If the records are so small, just create your own database schema that allows keeping a whole copy of each version of each record along with appropriate metadata tags (version, author, etc.).

    This isn't very difficult, and anyone with data modeling experience can pull it off. As far as free software goes, just use PostgreSQL or MySQL and a PERL or PHP or JSP front end. Problem solved.
    • We ARE using Postgresql and PHP currently. But the problem is NOT solved.

      Just copying the whole record for autit purposes does not help, since we need to assign authorship to the changes, not to the whole record.

      • Re:DIY (Score:1)

        by pmz ( 462998 )
        Just copying the whole record for autit purposes does not help, since we need to assign authorship to the changes, not to the whole record.

        You could store a whole new copy of the record along with the set of people responsible for that version. Each author entry can have an associated revision comment. That would probably be enough of a "paper trail" that the group of responsible people is small enough for an investigation to flush out the remaining details.

        It seems that if the record is small, then th

  • PostGreSQL (Score:3, Informative)

    by stanwirth ( 621074 ) on Monday May 12, 2003 @03:04PM (#5938276)

    You might want to try an open-source relational database engine like PostGreSQL [postgresql.org]

    This means, of course, you'll have to create your own schemata for logging version control and records added, but versioning additions to the database -- and user/connect information -- can be tracked with a combination of triggers and timestamps in a separate table. These are fairly standard techniques [goucher.edu] in generating an audit trail [cosyn.co.nz] for tracking relational database changes and enforcing data integrity constraints. While the syntax varies from database engine to database engine--PostGreSQL is a good (free) place to start, and if the spirit moves you (or if the DB becomes too large) the syntax differences do not preclude your moving to, say, a development edition of IBM's DB2 UDB [ibm.com] or even (aack!) Oracle [oracle.com]. All three run rather nicely under SuSE Linux, and are said to run quite nicely under RedHat, as well.

    It probably wouldn't harm you at all to develop this database with a relational database, and test it under different engines-- in fact, it would likely make you highly employable in the very near future.

    • We are experienced in writing standard database applications with standard audit trails (for example our health record system at http://www.gnumed.net). In fact, as you can see from the drugref.org website, we are using a PostgreSQL based drug database at present.

      However, the granularity of versioning and audit trailing typically used in relational daabases does not suit our needs, see my coments to other replies.

      As far as "being employable" goes: I don't. I do employ people, including doctors, since I am

  • Zope (Score:2)

    by Scaba ( 183684 )

    It sounds like you're really talking about content management more than version control. Maybe Zope [zope.org] and Plone [plone.org] will do it for you? Out of the box it gives you versioning, authentication and a decent database. The database isn't the most scalable, but you can extend it to use any number of database and database-like backends, like MySQL, PostgreSQL, Berkeley, and various pay-for dbs.

  • http://hardhats.org/

    The VA has the VistA project, which covers pretty much every aspect of medicine. The application is based on 30 year old technology known as M (aka MUMPS, which evolved into "Caché".) The source code is not GPL, it is available because of the Freedom Of Information Act (FOIA). And IIRC I believe RMS said a long time ago that the VA FileMan code was GPL-compatible.

    The normal audit trail for record addition/modidifcation sounds like it would cover your needs. But more likely, th
    • The VistA source is GPL compatible because it is Public Domain.

      There are parts of the VistA system which are not included in the FOIA release. The poster's primary interest (medication instructions) would appear to be one of those bits. The bits which aren't covered are primarily information copyrighted by other organizations (like the AMA).

      If you aren't already an experienced M programmer, I would recommend having a large bottle of your favorite painkiller handy for when you review the code, it is pret
      • Fortunately, the AMA is supposedly developing an open source EMR system. (Or is it AAFP? Yeah, it's AAFP) Whether this sort of data will also be open is anyone's guess at this point (also whether or not the eventual license will be cleared by OSI).
  • From your description and your web site, what you need is not "version control" or "source code management" or "configuration management" software (which is what cvs or subversion provide), but a "content management" system, which contains editorial workflow, publish/unpublish capability, probably the ability to generate static pages, be a web services provider, etc.

    Try one of the following for size:
    Tiki [sourceforge.net]
    phpWebSite [sourceforge.net]
    OpenDocMan [sourceforge.net]

    Your workflow requirements seem to be a lot more sophisticated compared to mo
    • Yeah but I would look at going to a deeper level of CMS that has workflow built in.

      eZ Publish [ez.no] from ez.no, and a bunch of others listed at OSCOM [oscom.org]

      But I would stay well away from some of the new stuff. There are plenty of CMS's out there without a project needing to use something that is new and is a duplicate of an existing product.

  • Use File Loading Processes (Score:3, Interesting)

    by justanyone ( 308934 ) on Monday May 12, 2003 @03:37PM (#5938584) Homepage Journal
    I work for a large multinational bank in Chicago. We are aggregating data in a data warehouse and have both lots of sources and lots of data.

    The way we cope with this problem is that each data source is given a code (it's usually just the filename). We have a Perl program parse these files (they're comma delimited ascii, tab delimited, DBase IV, etc., and some even human-readable reports), and load the database with the contents. Each record includes the source ID, for easy attribution / tracking.

    We keep each file version for a while. Each file has a business date so if they want to clobber a previous version of that data, they get to do so.

    This could keep your troubles to a miminum. Write a parser and have a file-upload site that lets people upload data. Define a group of people if you want. They should only be able to add/replace/delete their own data by the nature of the file. Each group can only create a certain filename or the group id is in the file.

    This way, many can share database updates via a batch run where updates are tracked and possibly even approved before committing them.

  • CMS (Score:1)

    by Samus ( 1382 )
    Sounds like you need a light weight content management system that at the very least has an audit trail associated with it. That isn't trivial but you don't need a rocket scientist to build it for you either. Just use one of the free relational dbs and in a week or so you could have a pretty functional system.
  • Do you plan to compete with databases like Merck Index?
    How are you going to select your contributors? How you are going check for the correctness of their data input?
    How do you plan to solve the problem with synonymous keywords in the search? [Good old- fashion Chemical Abstract database published realy extensive indexes on general subject keywords - apart from indexes on the substances, chemical core structures etc. You will need to do this and it is a lot of awful work]

    The main problem is you will need a
    • Our scope goes beyond the Merck Index.
      Correctness of data is checked by peer review.
      We have literally hundreds of volunteering qualified physicians and pharmacologists to perform data entry and peer review - any hands make any work small.

      The medicolegal aspect of it is exactly the reason why we need a finer granularity of database auditing than most projects do.

      Our intended "customers" (our work is free in every aspect!) are health professionals wanting to use independent medical software and decision sup
  • You should take a close look at open source TWiki (www.twiki.org) -- from what you've said, I think it might be perfect for you. I personally have been using it with large groups of people for several years.

    TWiki is a Wiki variant that adds RCS for every page that is editable, the ability to have access control lists for different pages, yet like Wiki has an "edit this page" button, and is very accessible to those you are not familiar with HTML. TWiki also has a large base of "plugins" that allow you to so
  • It sounds to me like you need more of the Database (probably relational) than just a version management system. Since you mentioned of different authors working on the (potentially) same pieces of the information - you will need one with decent support for locking and transaction isolation levels. PostgreSQL is probably the right candidate from the OpenSource domain and from proprietary ones - I would say Oracle. You might need to look into some additional features such as how your database will be searched

  • Bugzilla (Score:3, Informative)

    by The Bungi ( 221687 ) <thebungi@gmail.com> on Monday May 12, 2003 @05:05PM (#5939535) Homepage
    Some people have recommended a Wiki... I'd go with Bugzilla. Maybe with a little bit of Perl hacking you can modify it so that it looks (and behaves) the way you want.

    I have a modified version of Bugzilla running on Windows (W2KAS/IIS) that is being tested for something vaguely similar. It works great. The hard part was getting it to work on IIS with ActivePerl - if you're running Apache/Linux or BSD it won't be nearly as big a chore. I didn't even have to touch the MySQL schemas.

  • Wiki? (Score:1)

    by oz_ko ( 571352 )
    Have a look at http://www.wikipedia.com. It's a collaborative encyclopaedia that sounds very much like what you are after. It's really just a very big wiki site.

    Also has version control features, peer review, tags etc etc.

    HTH.

    Oz

  • As was mentioned previously, the easiest solution sounds like a redesign of your existing database. But it sounds like you'd want [at least] three specific tables:

    drugs:
    drugID (unique, autoincrement, long/int)
    drugName (text)

    users:
    userid (unique, autoincrement, long/int)
    name (text)
    securityLevel (if you want multiple levels, if not everyone's the same level)

    data:
    dID (same as above)
    uID (same as above)
    date (timestamp)
    added (text)
    deleted (text)
    rating (float)

    Then each addition is run through the UNIX command

  • Content management systems (CMS) not CVS... (Score:3, Informative)

    by aquarian ( 134728 ) on Tuesday May 13, 2003 @09:52PM (#5951083)
    What you want is a content management system, or CMS. These do exactly what you're talking about. There are a whole slew of them out there, free and not free. Furthermore, there are some general web services toolkits with good CMS modules. Find one that comes closest to meeting your needs, then modify it to get exactly what you want. Some that I've used are Zope [zope.org], OpenACS [openacs.org], Redhat CCM [redhat.com], OpenCMS [opencms.org], MMBase [mmbase.org], and Vignette [vignette.com].

    Of all of these, I like OpenACS the best, mostly because of its developer community. There are a lot of great people involved, and there's a high signal to noise ratio on the developer forums. Even though OpenACS probably has the least of what you're looking for, it might be the easiest to develop. OpenACS runs on top of Postgres or Oracle, and is written in Tcl.

    Redhat CCM is basically a Java rewrite of the original OpenACS. Its CMS modules are supposedly more mature. It runs on a Redhat version of Postgres, and I think Oracle too.

    Zope is a whole lotta product, and probably has most of what you're looking for. However, I find it kind of murky, difficult to figure out. YMMV.

    These three are the most promising in terms of developer community. This is a bigger undertaking than it might seem at the outset. You'll need all the help you can get, and getting involved with these communities will spare you from trying to reinvent the wheel.

    Of course, I'd love to have you guys use and extend the OpenACS toolkit, and share your efforts with the community!

Slashdot Top Deals

Our business in life is not to succeed but to continue to fail in high spirits. -- Robert Louis Stevenson

Close