Slashdot Log In
Will There Be Historical Records from the Digital Age?
Posted by
Cliff
on Tue Apr 10, 2001 12:59 PM
from the stuff-to-think-about dept.
from the stuff-to-think-about dept.
magarity asks: "NPR's Morning Edition today aired a segment on the Medici Archive Project where every letter sent and received by the ruling Medici family of renaissance-era Italy is being stored. The interviewer, Bob Edwards, casually joked that it was a good thing the Medicis didn't use email or else all this history would have been lost. It is easy to predict that at a similar distance in the future little will be known about our time period. After all, it is already problematic retrieve 25 year old data from 8 inch floppies, simply because the reading mechanisms are hard to find even if the media has retained the data. The same thing will happen to CDs in 50 years. How should the dawn of the digital age be recording itself for history, especially casual correspondence that gives insight into day to day life?"
"The Medici Project concerns itself with the rulers and given the recent report of US Congress members not making use of email one assumes they are still using good old long term archivable paper. Will the President and Congress in 2030 or even 2020 feel the same way? The main problem being digital records are so much more easily tampered with compared to old paper. It's not as easy to do carbon dating or other such tests with a bunch of bits. Remember: the victors always have and always will rewrite history as much as possible."
Related Stories
[+]
Technology: We're In Danger of Losing Our Memories 398 comments
Hugh Pickens writes "The chief executive of the British Library, Lynne Brindley, says that our cultural heritage is at risk as the Internet evolves and technologies become obsolete, and that historians and citizens face a 'black hole' in the knowledge base of the 21st century unless urgent action is taken to preserve websites and other digital records. For example, when Barack Obama was inaugurated as US president last week, all traces of George W. Bush disappeared from the White House website. There were more than 150 websites relating to the 2000 Olympics in Sydney that vanished instantly at the end of the games and are now stored only by the National Library of Australia. 'If websites continue to disappear in the same way as those on President Bush and the Sydney Olympics... the memory of the nation disappears too,' says Brindley. The library plans to create a comprehensive archive of material from the 8M .uk domain websites, and also is organizing a collecting and archiving project for the London 2012 Olympics. 'The task of capturing our online intellectual heritage and preserving it for the long term falls, quite rightly, to the same libraries and archives that have over centuries systematically collected books, periodicals, newspapers, and recordings...'" Over the years we've discussed various aspects of this archiving problem.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Rosetta Stone writings and primers (Score:2)
Think about Egyptian culture. We wouldn't have a clue without the Rosetta stone. It wasn't enough that they left writing and markings that have lasted thousands of years. We needed a tablet with the same message in several messages to figure out what they were trying to say.
So what you really want in your storage is a long term package, no moving parts or power supply, some generic and easily understood interface, and a primer that cannot be misunderstood.
Also, for those thinking we can just have plain ascii text, it's not that simple. Ascii is an encoding scheme. You have to have something in the primer to tell the reader how to decode the data and then what those letters and words mean, and so forth. In 2000 years we invented Latin, French, German, English, but modern German speakers would find Old High German hard to comprehend.
This gets worse as time goes on. It's already hard to explain feudalism to people, try explaining the Roman Republic's governmental structure. Now, try explaining American Democracy in 500 years.
It's not just the media, it's the culture. And a primer is how you get them able to follow enough of the conversation to get a grip on it.
Don't you know? Historians == IP thieves! (Score:2)
Of course there will! (Score:3)
They will say:
Thank you, Ministry of Historical Perspective!
Will anyone care... (Score:2)
In 50 or 100 or even 500 years will historians be able to access what we have done today? I hope so but I don't really know.
Re:Some thoughts (Score:2)
We don't have less long-term storage (Score:3)
Check out the LongNow Library Project (Score:2)
http://www.longnow.org/10klibrary/library.htm
Donald E. Knuth's real reason for TeX (Score:2)
This is no coincidence, because Knuth's main oeuvre, a several volume work on computer science, has already a related aspect:
Computer science changes very fast and Knuth decided to include just those parts of computer science that have settled and that might have reached a maturity that would make them unlikely to get radically changed in the future. Hard task. And indeed that stuff he put into his three released volumes is highly mathematical, because such stuff is typically evolved enough, but still he did not really manage it, so the RISC architecture for example pushed him to update his machine language MIX.
At some point, when Knuth got some copies of his TAOCP, he was frustrated enough because of the typographic quality getting worse. So he decided to take some time off to develop a system that turned out into TeX (who else than a professor can take 10 years sabattical to do such :-)
To shorten the story:
Knuth developped TeX, the programm that assembles boxes into lines, lines into pages, pages into documents. Developped Metafont, the programm that takes the mathematical description of font families (= a meta font) and renders them into bitmaps. He developped the computer modern fonts in Metafont format. Plus he invented a system called literate programming, that allowed to derive programming code and documentation from the sources.
All this, has been released in form of five books:
This means, that even in hundered of years, everyone with those 5 books, something like a computer, and the ability to read mathematical texts plus the computer science knowledge to implement a Pascal like language, will be able to reconstructs Knuth's whole system!!!
If at that point .tex sources are available
(at least as printed listings!),
they will be able to hack device drivers for
their then common output devices and to be
able to print all of Knuths works in original
typographical quality!
That is real deep reason for Knuth's TeX - longevity of information.
Re:I only see (Score:2)
No better. Metal gets corroded by water (worse yet: saline water), melted by fire, cracked by cold etc.
Besides rock, which has proven pretty good throughout the ages, there's one thing that could hold up the promise, and that's mineral paper [stanford.edu]. (Aka, asbestos paper.)
Karma karma karma karma karmeleon: it comes and goes, it comes and goes.
Re:Christ, were you even awake in science class? (Score:2)
Hardly affordable metals aren't they? I'm talking something remotelly accessible, not gold-plated disks to be sent outter space...
Where YOU awake in economics class?
Karma karma karma karma karmeleon: it comes and goes, it comes and goes.
Orwell (Score:2)
Doubleplusungood! Thought Police! Here! I have found a crimethinker! He must be an agent of Emmanuel Goldstein, spreading misinformation!
Put Doctor K with his brother in the Castle!
__
Re:Yes, the Louvre is a piracy bunker! (Score:2)
This might be a good thing (Score:2)
Digital rot of our records, I mean.
Think about it -- what do we have to pass on to future generations of the past 20-30 years? Boy George, N'Sync, Lyndon LaRouche, Hare Krishnas, Monica Lewinsky, Rush Limbaugh, Al Gore, Rob "CmdrTaco" Malda...
It might be a good idea for ALL these things to slowly melt away ...
"Beware by whom you are called sane."
answer: (Score:4)
The Rulers (Score:2)
Currently I'm about to pick up a used Super-8 projector to show some films that are in great shape.
Also just got a 1930's Burroughs adding machine for $15 from a hamfest that, with a few drops of oil and cleaning is in 'like new' condition and will probably be in working condition hundreds of years from now if kept in the right environment (room temp, low light and humidity - basements, attics, garages and sheds are hell on that stuff).
Re:Does anyone care (Score:2)
I'm not worried about what records will survive and won't survive from our era. The Romans, the Greeks, they didn't worry about such things. They worried about what legacy they would leave for the future (fat lot of good it did them) which is what kind of world they were leaving for their children. This is far more important, IMHO.
I have the archives,in M$Word 2.0,uh never mind (Score:2)
If its not the medium (read an 8" diskette lately? How about a 14" 5MB cartridge? How about a reel of mag tape?) its the software (M$ Word documents formats were deliberately sabotaged to force people to migrate to the newer versions. [I don't know anyone who actually needed M$ Word '97 until they found that they had to upgrade when M$'s biggest clients who'd got their copies for dirt.])
There will be thousand year old documents and last week's flimsies and nothing in between. Just an Orwellian silent testimony to greed and obsolence planned and otherwise.
But that said. have we said or written down anything worth keeping?
micro-printing (Score:2)
Etch text, not binary codes.
The future can read this with a computer or magnifying glass.
only copied stuff is "saved" (Score:3)
Even paper distintigrates, albeit in centuries.
Only a tiny fraction of stuff is copied now or then.
Re:Historical value of recent archives (Score:2)
We need a standard for long term storage (Score:4)
LTSS 2.0 might have whizbang new file formats and storage media which supports 100 times as much information density, but it must be compatible with version 1.0.
LTSS 1.0 could support WAV, MP3, GIF, TIFF, Text/ASCII, Text/Unicode, HTML version whatever, and perhaps even Java for interpretation of abirtrary file formats. The media, CD-R, or perhaps one of the writeable DVD formats when they mature.
-josh
Re:Simple answer: "No." The reason should scare yo (Score:2)
Take the Ems telegram for example, seriously altered, and sparked a war between France and what would become Germany. Of course, we now know that it was altered, but at the time no one knew what happened.
If you think that people altering documents for their own good is anything new and ruins good historical records, you need to wise up and take a history class. This is nothing new, and we still have a good idea of what happened.
Institutions... (Score:2)
Kevin Fox
--
Not what you store it on, but *where* you store it (Score:2)
If we want to save data we need to make redundant copies, in a form that is resistant to electromagnetic radiation (say, microetched in carbon, silicon, or other stable element), and put it into a heliocentric orbit 1 radii behind or ahead of the earth's orbit (this way it's not in a trojan point, which could result in collision damage, but is still in a 'mathematically likely' place).
Most of the corrosive factors would be left behind on Earth, and the data would be stable for the long haul. Alternatively, we could put data on the moon, where it would be stable until a meteor hit it or covered it up, likely tens or hundreds of millions of years, and if we put several down, they'd last longer.
Hmm, maybe a big micro-etched monolith buried just under the surface...
Kevin Fox
--
Re:Is it necessary? (Score:2)
Re:only copied stuff is "saved" (Score:3)
Slightly OT: Check out Hillis' 10,000yr clock (Score:2)
media matters too much (Score:2)
For a virtual world we ought to separate the infos from the media. We could store data and execute programs some computers and use the majority result. See Askemos [askemos.org] how this will work.
Once we are at it, we might find that files are worse than paper for another reason. We better had "write once" files. - If reusable paper were better that nomal paper, we would have it in the stores. Enough cycles of invention went over it already.
Isn't this scary (Score:5)
It reminds me of the Arnold Swarzen...(?) movie, "The Running Man". He's a police helicopter pilot who refuses to shoot unarmed people involved in a food riot. The powers that be manipulate the video tape evidence to make it appear that he massacres the people instead. People are shown the tape and cry for his death in a game show type fashion until some revolutionaries are able to show the real tape by hacking into the communications channel.
The temporality of public records has very serious implications for our social structure. If the only record of your speeding ticket is an entry in a database, what happens when a glitch makes you a drunken sloth who doesn't pay child support. If the entry showing Bush's drug convictions get deleted, will there be no other record. Trust me on this, email is a politician's dream. Everything from here on has plausible deniability.
Re:Legacy Databases (Score:2)
The day-to-day information that we produce is the stuff that makes genealogists go nuts. It's the stuff that leads to books like "Roots". Biographies of people who, to themselves, seemingly did nothing with their lives, yet looking back ath them a hundred years later we see how extraordinary they were.
Should -everything- be saved? No. Personal correspondance with friends and family should. (and hell, I have -every- piece of email that I've received at work over the last year saved. Talking roughly 500MB or so of gzipped archives (which balloon to about 1.5G)).
How much of it would you want to read? (Score:2)
Moore law says: don't read originals, copy them (Score:2)
As a matter of fact, each time I get a new computer, I copy all the stuff from the old one, and it takes only a fraction of the space. The 40MB of my first (Atari ST) hard disk are there. The 160MB of my first Mac hard disk (120MB left after I copied the Atari hard disk onto it) are there. And so on.
The real issue is binary formats that have been forgotten. For instance, I have source code of programs I wrote in GFA Basic (a Basic for the Atari ST, in case you wonder.) But emulators come to the rescue there. Today, I can run Atari programs faster than on the real machine.
Some thoughts (Score:4)
One problem with archiving digital communications is the volume. One of the problems that were found during the many Clinton investigations was, when e-mail was subpoenaed, separating the wheat from the chaff. All the mail was backed up onto tapes, which weren't very well marked. And the first searches were done on subject lines. Quite a bit of relevant mail was missed, and turned up years later when people actually sat down and read every message.
The National Archives (here in the USA) is worried about preserving data. The various software and hardware formats used over the years make it difficult to track and retrieve the data. NASA has spent a fair amount of money moving old planetary exploration data from tapes to optical disks, and then to CD. My father worked on a project at DMA (now NIMA) to do the same thing there.
Paper tape (Score:3)
Sometimes the answer to your question about how do we do X with technology can be found by remembering the history of technology. In this case, what might be a better long-term storage medium than magnetic or optical media is good 'ole paper tape. Now, some research should probably be done to increase both the durability of the tape material and the density of information stored on it, but it is the best solution I can think of, and probably the easiest to decipher by archaeologists of the far future.
--
SecretAsianMan (54.5% Slashdot pure)
Automated chaff generation doesn't help (Score:3)
No kidding. I'd hate to be in Deja/Google/whoever's shoes, trying to archive useful data, in face of terabytes of "Nude Asian Teens" email generated -- literally -- completely automatically at the click of a mouse button. Especially since the most useful spam filtering methods (outright router blocks, keyword triggers, a bullet to the head of the marketing agent) are frowned upon by nice people.
Paper libraries have a "volume" problem because the media itself takes up so much space, and must be carefully stored. Digital libraries have a "volume" problem because any old jackass can easily create fifty times the amount of information that's worth keeping, and it must be winnowed out by a human.
Just my rant today (cleaning out another twelve spam emails).
It's Not Just Digital -- Microfilm Sucks as Well (Score:3)
There's a good review [nybooks.com] of a Nicholson Baker rant against Librarians in general for their sins of deliberately pulping the paper records of the past 130 years and replacing them with decomposing and badly executed microfilm facsimiles.
It seems that Vannevar Bush's infatuation with microfilm was shared by many in the WW2 OSS community, and this seems to have led to a misguided attempt to replace papers and books with microfilm in the interests of "efficiency".
Re:Natural selection (Score:5)
I've worked on research projects whose primary source was day-to-day accounting records of a small business running in Egypt during the 11th century. The records were preserved in part because they were at the bottom of a trash pile. The records gave us a huge amount of information about everything from transport methods to the ability of the state to collect tax. Most of the 'important information' from that period which people though was worth preserving revolves around which ruler stomped which other ruler's butt. Our 'trivial information' gave us a lot of stuff which we knew nothing about before, stuff which helped explain why ruler X had the economic wherewithall to stomp ruler Y's butt and, well, more interestingly, what it was like to live under ruler X or Y.
The same applies today. Yeah, a record of what your family ate for dinner for the past two weeks is truly trivial. But what it will say about daily life, the transport of food, diet, cooking technology, food storage & a whole lot more about life in the early 21st century might be invaluable to some historian in a thousand years.
Your 'trivial information' is someone elses data goldmine and vice versa. One of the things I really like about computers is they allow you to keep a lot of personal shit you might otherwise have to trash because it gets bulky. The chances that I'll hang onto all my mail & all my parent's mail and all my grandparents mail is pretty good when it fits onto a CD rather than choking up my small apartment with boxes. The chances that some future historian will get to read ordinary everyday mail rather than just the mail of presidents and kings in a thousand years is getting better.
On optical media. (Score:3)
The real reason that old magnetic tape is hard to read now is that it was never a great format in the first place. The stuff falls apart. My last employer had an old HP reel-to-reel machine for reading data on tapes from a company we had purchased, but the tapes were so old that the chemicals on the tape itself turned to dust and fell off. This is not a problem with optical storage. Optical storage also has the option of being dedicated in very small spaces, unlike the van sized tape players of old.
Life is also not a big issue with optical media, because just as the books of the Medici's were recopied over and over into new languages and on better bindings, so can data be quickly copied from old optical media onto newer formats.
The answer to this question is pretty obvious... (Score:5)
The analogy would be to read emails from, say, the white house in 200 years. Do you think the white house is saving their emails? You bet. Do we have lots of examples of (from the general public) letters from 200 years ago? Certainly not as many as there will be emails in the future. Usenet archives, digital backups stored in basements, most emails are being stored two or more times at two or more places. I don't quite understand why someone would think that just because it isn't on paper, it isn't going to keep. We are going to have far more emails stored in the future than we will know what to do with.
As society we think of ourselves as individuals to be pretty important, but lets face it, for the vast majority of us, no one is going to care in 150 years. With that in mind, the digital age is storing far more records than ever before and the future holds a new paradigm of historical record. I almost lament that I wasn't born 150 years after the advent of the digital age where high resolution movies will look as good 1000 years from now as they do today.
-Moondog
This is a known problem (Score:4)
Likewise, various people are trying to shut down the MAME ROM sites, but a lot of the hardware ROMs are deteriorating now and many of those games, which represent a golden age of creativity and a technical wonder of resource usage, will be gone forever. Kinda makes you sick, doesn't it?
Is it necessary? (Score:4)
While I'm all for archiving data for future historical analysis, I think it's fairly certain that IM logs, "how's it goin?" e-mails, and detailed transcripts of #40yearoldsinglebaldguys will not be very useful to historians in three hundred years. Yes, they tell about our culture and practices, and yes they might be interesting, but we don't need all of it to extrapolate those conclusions. There is simply no room to store the vast quantity of information generated on the Internet on a daily basis, and considering the fact that 99.998% of it is of little value, I think that we can safely do without it.
Things are still floating around from the old days. We have Usenet archives from the 80s, and text files from even earlier. We can learn a lot about the culture based on those. Things that grab the public consciousness tend to around. They get mirrored, printed out, saved on disk, etc.
Does there need to be a giant warehouse that contains vacuum-sealed printouts of every wise thing said on the internet?
No. No, there doesn't.
Re:Does anyone care (Score:3)
The problem with planning for the future is that it is hard to know today what will be important tomorrow. Perhaps the insignificant trolls on Slashdot will be of great import in the future (and, no, I'm not referring mainly to Jon Katz articles). Who woulda thunk that an accounting ledger from ancient mesopotamia would be of any interest 2500 years later?
Re:Is it necessary? (Score:5)
Of course the flip side of this is that it's not always possible to tell who will be considered interesting in the future. In many cases, the most interesting use of archives is to look at the work of interesting people while they were working their way up and weren't of broad enough interest to attract major attention. Nobody knew that a 25 year old patent examiner named Albert Einstein was about to become a scientific star, but because we have his personal letters we can find out what he was doing scientifically and personally.
You never know if the next great author might be posting his early, great works to some fan e-mail list because he can't get his foot in the door at a major publisher. Maybe the next great debator is getting started in flamewars on Slashdot. Maybe the next great OS designer is getting into arguments with established academics on USENET. Oh, wait, that already happened, and we can only read the argument [www.dina.dk] because somebody though to archive it. Maybe the next great philosopher who will be mostly ignored for 100 years is already publishing his early thoughts somewhere on the web. You can't always tell what will be valuable to the future until well after the fact, so preserving as much as possible is still a really good idea.
A truly wonderful example of this kind of thing are the early works of JRR Tolkein. The early history of the Silmarillion is absolutely fascinating and a wonderful example of the development of a literary theme. That's a work that wasn't published for over 50 years after it was started, but some of the earliest drafts still exist. Because those drafts are available, it's possible to see how it developed. Will the same thing happen when authors write everything in Word and write over old versions every time they change anything? How about if they're still very careful about keeping copies of early drafts but the formats change so much that they can't be read anymore?
Re:Some thoughts (Score:3)
So I'm sitting down and thinking, but I still don't understand their position. I can appreciate both the importance of ASCII text and its accessibility (hell, I still use lynx to browse the web), but I can't understand why you would restrict yourself to only text.
Consider the following:
On July 20, 1969, Neil Armstrong was the first man to walk on the surface of the moon.
--versus--
On July 20, 1969, Neil Armstrong was the first man to walk on the surface of the moon. Here is a picture, in an open, documented graphics format.
There's just too much history that's more than just pure text. I can understand trying to make as much material as possible available as text, but you can't let such a decision allow you to exclude relevant materials that're more than just text.
Re:Is it necessary? (Score:3)
While the histories, news articles, and official documents of a given era are very important and informative, it is also necessary to the personal accounts from the people involved in the society at the time to help provide perspective, and to help identify biases in the 'official' accounts.
Considering how valuable even the pedestrian of documents are from e.g. 3000 BC, I imagine that today's equivalent will be of equal value to historians in the 7000 AD.
-----
D. Fischer
It's not the media, it's the SOFTWARE. (Score:4)
Even two or three hundred years from now, a reasonably skilled technician or at worst a team of them will be able to dig up a CD mechanism from somewhere, fix it up and get it reading data. CD mechnisms are like Ford's Model T -- only much more common -- and let's face it, there are still a reasonable number of Model T's running around to auto shows, and there isn't nearly the historical incentive to keep a Model T running that there is to ensure that there will always be a CD-ROM reader running somewhere.
And it's likely that if most people are like I am (I value my data and my work) they will continue to migrate data to new formats as they emerge.
The bigger question isn't media, but sofware. I'm very confident we'll be able to get our files from ISO9660 discs, but I already have a bunch of WordStar and old MacWrite/MacPaint files I can't open and it's only been a decade. We'll be able to retrieve the raw data, but will be actually be able to interpret and make use of it?
P.S. I still have an old Siemens 8" floppy drive, single-sided, hard sector. About five years ago I still had an old floppy controller with an odd WD chip on it that could talk to it using OS-9. No way to talk to it with my Linux box, though...
White House Email (Score:4)
Apparently, George W. was an inveterate user of email right up until the inauguration. At that point, he sent a farewell missive to his correspondents [slashdot.org], in effect saying he could no longer use email because all such correspondence would be a public record and he didn't want his private musings made public.
So, no, many important communications will not be retained, unless someone is placing a wiretap on the president's phone.
Simple solution to digital recording (Score:3)
Re:On Bitrot (Score:4)
The digital age gives us great hope for preservation of everything, because we can copy sounds, images, motion and even DNA structures with perfect reproduction. But it will only be through the careful preservation of this information that future generations will be able to access it
If anything, and you can consider this a dig at DMCA if you like, it will be the number of copies of these artworks that will permit them to be preserved. Consider this: there is only one Mona Lisa -- if she fades, we can only guess at what her colour was. But there are millions of copies of Wing Commander IV. It's a relatively simple task to go through a few thousand of these, extract from each disc what data hasn't rot through, and compare it to the others. Combine that with huffman coding and CRCs and we can quickly reconstruct the original with perfection and certainty. You can't say that of the Venus DeMilo. And unlike other generations' copied mediums, we can trust the intermediary -- the cold, heartless eye of the scanner and OCR soft -- not to misspell anything or make up shit. Bemoan the need for proprietary copyrights if you like, but the digital age's perfect reproducability is the factor that will decide its permanent etching in the databases of the future.
Simple answer: "No." The reason should scare you. (Score:4)
Digital records are favored by our corrupt, foreign-dominated Federal tyranny for one very simple reason:
It's terrifyingly easy to alter them, or to dispose of them entirely.
This is frightening, but true: As the well-known conservative George Orwell observed in his great novel 1984, "He who controls the past controls the future." The "Party" in 1984 devoted itself to doing exactly what the Clinton regime did: They went through all historical records, altering, falsifying, modifying, deleting.
No one will ever know what the Clinton death count really was. No one will ever know what really happened. The "records" are malleable. You can trust no information that comes from the government, because it's all been "massaged" and "fixed up".
Will there be historical records? Not in any meaningful sense: There will be something that looks a lot like such material, but it will be a work of pure fiction.
Goodbye, America. We were great while we lasted.