Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Data Storage

How Would You Archive Mounds of Genealogy Data? 73

Posted by Cliff from the storing-your-family-tree dept.
dexter riley asks: "Hello, all. My mother, a librarian, historian and genealogist for over twenty years, died about a year ago. She left a huge amount of genealogy information, culled from books, magazines, and the internet, mostly in the form of typewritten, photocopied, and printed pages. My main goals are: Preservation - converting the documents into a compact format that can be easily copied and transferred to others; and Indexing - making it possible for someone else to easily find the documents referring to a particular person, family, place, or document type (like land, marriage, military, birth or death records). To this end, I would like to convert her work into a format that can be stored digitally and scanned for keywords, to make it easier for others to use this information for their genealogy projects later on. What tools do you recommend for handling a project of this size?"
" I'd estimate there are at least 10,000 pages of documents in all. Much of it is organized by binder into family groups, but a lot of it is unorganized, loose paper. Besides being an irreplaceable resource for any future genealogists in my family, there are other researchers working on related lines that may find some part of this data useful. At the very least, I would like the satisfaction of keeping some part of her work from being lost for a few years more.

Here's a general list of things that I've determined I would need:
  • Scanners: What flatbed scanners would you recommend for fast, high-resolution scanning of documents?
  • Image formats: What lossless image formats would you scan your original documents into?
  • OCR software: Although OCR is not perfect, would you recommend using it to allow keyword searching to the original document? If so, which software would you suggest?
  • Document Indexing: In addition to OCR, are there other tools (document tags?) that you would use to help classify and organize images and other digital documents?
  • File organization software: Ultimately, many thousands of text and image files will be generated. Since I don't want to just convert a paper mess into a digital mess, what tools would you use to organize related image and text files?
Did I miss anything in the above list? Any suggestions you all might have would be hugely welcomed."
This discussion has been archived. No new comments can be posted.

How Would You Archive Mounds of Genealogy Data? More

How Would You Archive Mounds of Genealogy Data?

Comments Filter:

  • That is SOME work... (Score:4, Informative)

    by McSnarf ( 676600 ) * on Monday July 11, 2005 @10:48AM (#13032837)
    1. Check the Document Management Continuum !
    http://www.archivebuilders.com/whitepapers/index.h tml [archivebuilders.com]
    2. Get two reasonable scanners that work with whatever software you choose. One with a document feeder (can be monochrome). Modern office MFPs work fine. The other one is a cheap flat bed scanner with color for anything the big one won't process.
    3. Doc prep and Indexing will take much longer than the scanning - and unlike OCR, are a lot of manual labour. Expect a couple of weeks, minimum, especially if you have't got an indexing scheme in place.
    4. Use TIFF G4 and PDF (OCRed text over the images).
    5. Profit.

  • Dead tree (Score:3, Insightful)

    by keesh ( 202812 ) on Monday July 11, 2005 @10:53AM (#13032865) Homepage
    Dead tree lasts longer than computer media. Ask anyone who has ever tried to get data off twenty year old tapes...

    • Re:Dead tree (Score:5, Insightful)

      by bill_mcgonigle ( 4333 ) * on Monday July 11, 2005 @11:03AM (#13032973) Homepage Journal
      Ask anyone who has ever tried to get data off twenty year old tapes...

      After I give them a dopeslap for not keeping their data current.

      I mean, my first Mac had a 40MB hard drive, but I still have all the data from it - it's become easier and easier each generation to copy all my old data forward.

      Granted, there's always the odd lost-tape found behind a cabinet, but that's someone who didn't have a good data retention plan in place and didn't care about that data too much.

      There will always be a need for forensic recovery, but compared with just a few years ago, almost all the casual users I know keep all their data on a hard drive. The floppies and ZIP's are gone. Some of it is on CD-R, but that's the new backup media, not current storage.

      Now getting them to do a good backup so I don't have to go rescue their drives with dd_rescue - somebody let me know how to do that!
      • After I give them a dopeslap for not keeping their data current.

        When your starting point is a Mac that has a 40MB hard drive, your perspective is understandable.

        Some of us have/had data on old 8" floppies or 5.25" floppies, back from the days when a 5MB hard drive was $2,000 and the size of a Dell server.

        It wasn't so easy to keep data current in those days. If you were lucky, you had a serial port, and could tranfer to a later generation machine. But disk formats were not standardized the way they are t
        • When your starting point is a Mac that has a 40MB hard drive, your perspective is understandable.

          For me it's not, but going forward the easy-to-read data model is standard. So _today_ you have no good excuse for keeping your data in a non-digital manner. Heck, I can hook up an IDE, SATA or SCSI drive via PCI controller, USB controller, or Firewire controller all with ease. Tell that to the MFM drive I still have sitting in the corner to dump.

          (If you or anyone knows a cheap service to grab data off of

    • Re:Dead tree (Score:4, Insightful)

      by 4of12 ( 97621 ) on Monday July 11, 2005 @02:53PM (#13035325) Homepage Journal
      Dead tree lasts longer than computer media.

      Frightful.

      I was looking at 100 year old newspapers from a small town in AL about 5 years ago. Yellowed, brittle, crumbly. Half of those old newspapers were useless to amateur pawed geneologists who were slowly contributing to their demise by even attempting to turn the pages.

      Then, there's the tons of valuable paper records that get passed to random descendents of record keepers (eg, Grandpa has a bunch of records of marriages, births, baptisms from some old church from 60 years ago that doesn't exist).

      If you make dead tree records I'd recommend making multiple copies that get distributed to different people in different places, preferably on acid-free paper.

      The biggest enemies of geneological records, IMHO, are

      • uncaring descendents chucking out a bunch of "junk",
      • their antecedents who never even bother to tell them or, better, write down who the hell is in those old photos, etc., and
      • the odd house fire that consumes everything that can burn.
      • The biggest enemies of geneological records, IMHO, are:

        Good choices, I'd add:

        • the jerk who has a complaint with the court/tax collector/local officials/adjacent land owner and burns down the courthouse
        • the jerk who steals original records for his "collection"
        • the record keepers that stores "that old stuff" in abysmal conditions, e.g. the basement that floods every 20 years...
    • Believe me, 20 year tapes are easy, if you used the QIC format. Just give Tandberg Data a call.

  • Organization (Score:4, Insightful)

    by poopdeville ( 841677 ) on Monday July 11, 2005 @10:54AM (#13032877)
    Are you sure there's actually a mess? Since your mom was a librarian, it seems to me that she would know how to organize this information. Go through it and make sure the information isn't structured before you start messing around.

    You might also want to ask the guys from rotten.com if they'll let you see the code behind the nndb.

    • ...see the code behind the nndb [nndb.com].
    • Good point! But although Mom was a librarian by profession, she was a packrat by nature. A lot of the information is organized in binders by family, but there is a lot of looseleaf stuff in bins that doesn't have any apparent structure. Much of it may even be scrap paper, but it's not a call that I'm qualified to make at this stage of the project.

      I definitely want to find a document management software/structure that will let me maintain what order exists, while making it possible to add structure to th
  • You expect her to do all of this herself... quick if you don't have a younger sister get an intern!

    Oh... You will probably find a wiki sort of thing these easiest to deal with (sort of a DB front end for dummies)

  • Mormoms can help (Score:5, Informative)

    by HowlinMad ( 220943 ) on Monday July 11, 2005 @10:58AM (#13032917) Homepage Journal
    The Mormon faith believes in tracing humans back to Adam and Eve. They have a hug geneaology library in Salt Lake City. There are several programs available that you can pu the information in, and submit it to them. They will keep it forever, and other can research it as well.

    • Re:Mormoms can help (Score:1, Interesting)

      by Anonymous Coward
      Not to turn this into a discussion on religion and whatnot, but it's probably worth mentioning that any dead relatives you turn over to the mormons will most likely be posthumously baptized into their faith. Being agnostic myself, I don't care one way or another, but some people do. I have a friend who is more into genealogy than I am who does research at one of the local LDS centers. He claims he is hounded everytime he goes there for his information (but won't give it for this very reason).

      This articl
    • Although I'm sure the Mormons would welcome a fixed, formalized family tree, most of the information I have is far lower-level than they might be interested in. There's a little data like, "Jehod begat Ezekial begat Fred", but most of it is like "John Smith owned 12 acres in Norfolk County in 1728." Information that by itself doesn't provide a definitive lineage, but is more like a circumstantial lead that a private investigator might follow; information that might help someone whose ancestor was married

      • Re:Mormoms can help (Score:3, Informative)

        by HowlinMad ( 220943 )
        You would be very surprised. My mother is big into this stuff. You can put in any kind of information about a person. They want it. When looking for ancestors, that kind of information is very handy, as it tells the story of where they where, went, etc. That help you to uncover where to look next. You can include it all, picture, land purchases, you name it.

      • Re:Mormoms can help (Score:3, Interesting)

        by menscher ( 597856 )
        Although I'm sure the Mormons would welcome a fixed, formalized family tree, most of the information I have is far lower-level than they might be interested in. There's a little data like, "Jehod begat Ezekial begat Fred", but most of it is like "John Smith owned 12 acres in Norfolk County in 1728."

        Although they might not be able to take that level of detail into their central database, I think they would welcme info like that at a more local level. At the local levels, they often have small libraries of

    • If you're not particularly keen on porting family history into a church's docs, and - if you're based in the right country - you might want to try the genesreunited [genesreunited.co.uk] sites. They can use the same format (GEDCOM) as the Mormons (they wrote it) and also allow searching around for matches in other peoples trees.

  • Mormons? (Score:4, Informative)

    by bill_mcgonigle ( 4333 ) * on Monday July 11, 2005 @10:58AM (#13032919) Homepage Journal
    Maybe it's best to consider outsourcing it. Groups like the Mormons do this kind of work as one of their missions.

    This link [familysearch.org] may or may not be useful.
  • The lds church [lds.org] does a lot of family history and genealogy work. They have family history centers around the world staffed with volunteers that work to help people preserve their genealogies and family history. Locations can be found here [familysearch.org] and other resources that might be helpful are found at Cyndi's List [cyndislist.com] which compiles a list of family history resources.

  • Google (Score:4, Interesting)

    by Phillup ( 317168 ) on Monday July 11, 2005 @11:13AM (#13033083)
    Give it to google, let them do it.

    You'll have it forever, and anyone will be able to pull it up.

    (Too bad they don't offer this service... yet.)
    • Once the data is scanned, letting google do the indexing (either online, through Desktop Search [google.com], or through a Search Appliance [google.com] isn't a bad idea.

      If there is significant enough value, pawning off some of the work to Google or the Internet Archive or something similar isn't a bad idea. In particular, many libraries and Universities already do this kind of work.

  • Tools (Score:3, Interesting)

    by ka9dgx ( 72702 ) on Monday July 11, 2005 @11:16AM (#13033123) Homepage Journal
    Image formats: It appears that TIF is currently the gold standard in terms of archival storage of documents. JPEG2000 will be the way to go, once it becomes commonplace.

    Document Indexing/File Organization: A Wiki is the proper tool for this job, in my opinion. It makes it very easy to edit, and hyperlinking is instictive. You can easily attach documents to pages, you can usually export the whole thing as a directory tree. Most Wiki software also keeps track of all of the versions of a page, so you can worry less about making bad mistakes.

    I've used both MoinMoin [wikiwikiweb.de], which is a traditional web based Wiki, and WikiDpad [sourceforge.net], which is an IDE environment for Windows that does Wiki-like things. Both of these programs are open source, Python based applications.

    You also might want to check out ThumbsPlus by Cerious Software [cerious.com], which stores thumbnails of images in a database (including SQL backends), along with keywords and user fields. It can help you as well.

    --Mike--

  • You didn't indicate any question about what archival-quality digital storage medium might be best. Remember CD rot? Magnetic tape/floppy wear & tear? Hard drive crashes? That's why my personal choice is magneto-optical. Available in 5.25" and 3.5" sizes, the disks are removable like floppies and have protective shells. The disk itself is mostly polycarbonate ("Lexan") and the data layer can only be altered if the temperature rises near the Curie Point for the media (protect from fire, obviously!).
  • The Fort Wayne, Indiana library has an amazing genealogy section. From the website:

    The Fred J. Reynolds Historical Genealogy Department of the Allen County Public Library was organized in 1961 by the library director for whom it was named. The department's renowned collection contains more than 300,000 printed volumes and 314,000 items of microfilm and microfiche. This collection grows daily through department purchases and donations from appreciative genealogists and historians. Because of the collectio

  • * Scanners: I would go with something basic. I'm a debater for a high school squad, and last year when we decided to digitize literally thousands of pages of evidence, we used one of these HP 5550 [hp.com] It's great, cheap (Only $300)and USB 2.0, the only thing I would say is an absolute requirement. * Image formats: I would use tif. They can be huge and higer resolutions, but scanning at 1bit (Black and White) seems to keep things under control. You also could try adobe's pdf, but then you are locked
  • Other posters have already suggested you check whether the papers are already in some kind of useful order. Whatever you do, preserve them. To you, they are the originals and points of reference against which all errors fall to be compared.

    As for digital format, I suggest you consider plain (ASCII) text. It's likely to outlast more complex formats.

    I've tried some specialised genealogy software, and it looks like a mixed blessing. It usually forces the data into pre-ordained types, and quite a lot of

  • Google says search, don't sort! (Score:3, Interesting)

    by jimbro2k ( 800351 ) on Monday July 11, 2005 @01:24PM (#13034376)
    Meaning that you don't necessarily need to organize the data, just be able to search it quickly.
    If you agree with that philosophy, then, after you have it all in ASCII, just do a full text index of the data (which makes sense if the data is rarely or never updated) and it is quick to pull out anything you need.

  • Fire. (Score:3, Funny)

    by Chess_the_cat ( 653159 ) on Monday July 11, 2005 @01:30PM (#13034440) Homepage
    And plenty of it!
  • For Genealogy data, the de-facto standard file format is GEDCOM [rootsweb.com]. Originally created by the LDS Church (the Mormons), virtually any decent genealogy program will support the format.

    While it's a somewhat ugly text-based, flat-file format, it does permit organization of information in ways that will be useful to genealogists and researchers.

    • While it's a somewhat ugly text-based, flat-file format, it does permit organization of information in ways that will be useful to genealogists and researchers.

      Even better, pretty much any genealogy software can read and write GEDCOM files. Think of GEDCOM as the CSV of the genealogy world. It even has rudimentary facilities for storing multimedia files in an encoding similar to Base64. I don't know if the various software packages commonly support it, though. (My wife's the family genealogist. I

      • I don't know if the various software packages commonly support it, though.

        I think many of the recent ones do. (Of course, there are still the products that don't really support GEDCOM at all, but they're the minority.)

        Then there are a lot of hybrid solutions (like used by phpGedView, for example) which will support only "external" media.

        I suspect that, eventually, they will all support the embedded media.
  • Although I am sure that there are better models now, this is the scanner I am still using:
    http://avdcs.com/prod/cards/fb1200.html [avdcs.com]

    Although you won't usually print anything at higher than 300(ink jet) or 600(laser) dpi, I recommend that for archival purposes to go ahead and scan at full-color 1200dpi from within Photoshop or something.

    I would make sure to backup all your originals onto a DVD DL (Dual Layer) drive as you can now get the drives for about $50 and they can use CDRW, DVD+/-RW or DVD DL disks a

  • First things first (Score:3, Informative)

    by GCP ( 122438 ) on Monday July 11, 2005 @02:55PM (#13035358)
    Others are suggesting answers to your scanning needs, and your mother may have already done this, but just in case:

    The most important thing you can do with a pile of data is turn it into useful information. I know that's what you're trying to do. In the case of genealogical data, the most important is not just an electronic version of paper sources, but something built from those sources: an electronic family tree with facts (birth date/place, death date/place/cause, graduations, marriage date/place, immigration, military service, etc.) attached to each person and SOURCES attached to each fact, and all of it exported to GEDCOM format.

    For each source, good software will let you include transcripts (not scans but transcribed excerpts that aren't too big) along with the bibliographic reference that tells others where they could find the information for themselves.

    I probably have as much of this sort of data as you have, but most of it has been "processed" by extraction. That leaves the rest as backup that I seldom need to refer to. Since 99% of the information (by VALUE, not by bits) has been extracted into a very useful form, the rest is referred to so rarely that paper is just fine (as long as there is another copy stored elsewhere).

    I'm not saying that having electronic copies of all of the original paper sources wouldn't be better. It would, but mostly for ease of backup. It's just that the most valuable thing to do (in my opinion) is the extraction and assembly of a really rich information structure (family tree, facts, sources, transcriptions of sections of sources) in a standard interchange format (GEDCOM).

    After doing so, the additional value in having electronic, searchable copies of the original documents gets much smaller, so if I were you I'd make sure the information extraction project was done first before embarking on the scan/OCR/indexing project. After you do the former, you may decide that the latter isn't worth the effort.

    • The one problem with GEDCOM is it does not currently support lots of notes and other extraneous information. My father uses a program called Legacy Family Tree. ( http://www.legacyfamilytree.com/ [legacyfamilytree.com])

      This allow you to attach notes to each individual thus as much extra information as you want. You can also attach images, scanned docuemnts, pictures etc...

      the program is for windows, but is very inexpensive ~$20.00.

      this would allow you to organize all the information with all relations and family tree informat

  • GEDCOM (Score:3, Informative)

    by D.A. Zollinger ( 549301 ) on Monday July 11, 2005 @03:19PM (#13035639) Homepage Journal
    A lot of what you are looking for has already been figured out by others who do not want to be locked into a single format, vendor, or system. There are many geneology programs out there, such as Gramps [gramps-project.org] or Brothers Keeper [bkwin.net] who use a standardized file format, GEDCOM, that allows information stored in those specific programs to be transferred to other programs. This allows for easy upgrades in software, as well as the possibility of moving from one package to another as the information can be archived to the GEDCOM file, and then read in again once the new software has been installed.

    As for hardware and other software, I would suggest you use what is familiar to you, or is compatible to the software package you have chosen.
  • http://www.familysearch.org/ [familysearch.org]
    It is run by the Church of Jesus Christ of Latter-Day Saints. Future access will pretty much be a given and it will help others their family history.
  • Make sure this wealth of information ends up in GEDCOM, which is pretty much the de facto standard for exchanging genealogical data electronically. Any genealogy software worth its salt can read from and write to this format.
  • If you care about this paperwork at all, have it photographed to 16mm or 35mm roll film. Some companies will generate digital version as they create the film, or you can pay someone to scan the film afterwords. Film is the only reliable long term storage media that allows you to reduce storage space requirements. It also does not suffer from "tech rot" as all you need is a light and a lens to view it. From there you can build your digital collection safe in the knowledge that if you keep your film in a fire
  • I know you asked for flatbed scanners, but if you are seriously going to feed in 10,000s of pages, an actual document scanner would be better. Even the document feeders available on many flatbeds are deoptimal--they are slow and jam all of the time. Scanners which are meant to be used by workgroups to actually scan in documents typically dont curl the paper over itself, so are both quicker & jam less. But they are more expensive.

    I am extremely happy with my Canon Canon DR-2080C [canon.com]. Note: It is the onl
  • Archive it the same way most organizations still do: RAIDed (possibly networked) hard disks with (offsite) backup to tape. Probably still the cheapest & one of the most dependable ways to store all of this.

    As for organization, just come up with a simple, sane way to name things (like /smith/binder1/page1.tiff and /loosepage/page3.pdf (TIFF and PDF being two damn-fine formats for you to use--TIFF for lossless archives & PDF for the graphics+OCR-ed text) & rely on your indexing to actually make

  • I think the best way for you to use this information is to first develop an interest in genealogy yourself. Genealogy is an art as much a science. What you see as only mountains of raw data the genealogist sees as potential relationships between any one piece to any other. Some examples:

    Census Forms. If you come across a series of census forms, how are you going to organize them? Are they by year? By location? By primary line? By associated families? (Which may be the only connection between the primary

    • The best description I have ever read about genealogy is that it is like a court case in which you are trying to assemble facts and proof to make an argument.

      This is very true, but there are few genealogist who are working according this principle, and even fewer software packages that are supporting this way of working. It seems that most programs are only about recording the conclusions of your research.

      Even the GEDCOM format is not really designed according to these principles. A good genealogic pro

  • into a format that can be stored digitally

    Hot on the heels of this... I believe the linked thread resolved that digital formats are doomed to fail because they are unreliable over long term (ie > 5 or 10 years). Anywho, the thread might help you with the storage and backup aspect :)

    • *grrrr* Forgot the link:

      LINK [slashdot.org]
    • You're confusing media and format. A particular piece of digital storage media (eg. a hard disk, or a DVD) may not be reliable over 5 years. That's why most people are smart enough to keep copies of valuable data on more than one physical piece of media. (ie. multiple DVD backups, or IDE hard disks in a RAID formation).

      Once a newer/better/more reliable storage medium comes around, it takes an hour or two (usually) to transfer the data to the new media.

      Formats however are fine. GIFs, TIFFs and JPGs
  • Digital preservation is a pretty hard task, mostly because we haven't yet collectively acknowledged that our society now relies on a digital memory that is extremely fragile.

    Given that this problem is quite hard to solve in the long term (although much easier if just for the short term), it would probably be better to donate the material to an organisation with the resources and longevity to secure it.

    Over the long term, you will have to migrate storage media every few years. You will also have to migrat

  • GEDCOM (Score:2)

    by N8F8 ( 4562 )
    GEDCOM [familysearch.org] is the standard file format for geneology information. There are plenty of products that import or use this format. GEDCOM was developed by the LDS Church (Mormons). They also have a free program to manage your geneological data called Personal Ancestral File [ldscatalog.com]. Personally I use PHPGEDView [phpgedview.net] to manage my family geneology data (view here [lowing.org]
  • I have struggled with this same issue with having lots of (physical and digital) documents & photographs, and I want a way to store and track it. I have multiple "buckets" of info to store, from genealogy to projects to general tech dump areas.

    I use ScanSoft's (www.scansoft.com) PaperPort ($100), which is about as cheap as you can get for document management on a decent level... it has basic OCR software built-in, although they sell OmniPage for about $150 (I think), for better OCR capabilities.

    T
  • Many have pointed out that digital formats can become obsolete - the main thing to remember is that digital data can MORE EASILY be moved to new media, file formats, whatever. So the data doesn't become obsolete, at all (for example, old data on Apple II => csv => Access => MySQL is technoligically trivial). GEDCOM is a good format to use for the genealogical data, IF you have the time to index it that way in some program and output to GEDCOM. If not, just preserve it for some family member who w

Slashdot Top Deals

He has not acquired a fortune; the fortune has acquired him. -- Bion

Close