Will There Be Historical Records from the Digital Age? 251
magarity asks: "NPR's Morning Edition today aired a segment on the Medici Archive Project where every letter sent and received by the ruling Medici family of renaissance-era Italy is being stored. The interviewer, Bob Edwards, casually joked that it was a good thing the Medicis didn't use email or else all this history would have been lost. It is easy to predict that at a similar distance in the future little will be known about our time period. After all, it is already problematic retrieve 25 year old data from 8 inch floppies, simply because the reading mechanisms are hard to find even if the media has retained the data. The same thing will happen to CDs in 50 years. How should the dawn of the digital age be recording itself for history, especially casual correspondence that gives insight into day to day life?"
"The Medici Project concerns itself with the rulers and given the recent report of US Congress members not making use of email one assumes they are still using good old long term archivable paper. Will the President and Congress in 2030 or even 2020 feel the same way? The main problem being digital records are so much more easily tampered with compared to old paper. It's not as easy to do carbon dating or other such tests with a bunch of bits. Remember: the victors always have and always will rewrite history as much as possible."
Rosetta Stone writings and primers (Score:2)
Think about Egyptian culture. We wouldn't have a clue without the Rosetta stone. It wasn't enough that they left writing and markings that have lasted thousands of years. We needed a tablet with the same message in several messages to figure out what they were trying to say.
So what you really want in your storage is a long term package, no moving parts or power supply, some generic and easily understood interface, and a primer that cannot be misunderstood.
Also, for those thinking we can just have plain ascii text, it's not that simple. Ascii is an encoding scheme. You have to have something in the primer to tell the reader how to decode the data and then what those letters and words mean, and so forth. In 2000 years we invented Latin, French, German, English, but modern German speakers would find Old High German hard to comprehend.
This gets worse as time goes on. It's already hard to explain feudalism to people, try explaining the Roman Republic's governmental structure. Now, try explaining American Democracy in 500 years.
It's not just the media, it's the culture. And a primer is how you get them able to follow enough of the conversation to get a grip on it.
Don't you know? Historians == IP thieves! (Score:2)
Of course there will! (Score:3)
They will say:
Thank you, Ministry of Historical Perspective!
Will anyone care... (Score:2)
In 50 or 100 or even 500 years will historians be able to access what we have done today? I hope so but I don't really know.
Re:Some thoughts (Score:2)
We don't have less long-term storage (Score:3)
Check out the LongNow Library Project (Score:2)
http://www.longnow.org/10klibrary/library.htm
Donald E. Knuth's real reason for TeX (Score:2)
This is no coincidence, because Knuth's main oeuvre, a several volume work on computer science, has already a related aspect:
Computer science changes very fast and Knuth decided to include just those parts of computer science that have settled and that might have reached a maturity that would make them unlikely to get radically changed in the future. Hard task. And indeed that stuff he put into his three released volumes is highly mathematical, because such stuff is typically evolved enough, but still he did not really manage it, so the RISC architecture for example pushed him to update his machine language MIX.
At some point, when Knuth got some copies of his TAOCP, he was frustrated enough because of the typographic quality getting worse. So he decided to take some time off to develop a system that turned out into TeX (who else than a professor can take 10 years sabattical to do such :-)
To shorten the story:
Knuth developped TeX, the programm that assembles boxes into lines, lines into pages, pages into documents. Developped Metafont, the programm that takes the mathematical description of font families (= a meta font) and renders them into bitmaps. He developped the computer modern fonts in Metafont format. Plus he invented a system called literate programming, that allowed to derive programming code and documentation from the sources.
All this, has been released in form of five books:
This means, that even in hundered of years, everyone with those 5 books, something like a computer, and the ability to read mathematical texts plus the computer science knowledge to implement a Pascal like language, will be able to reconstructs Knuth's whole system!!!
If at that point .tex sources are available
(at least as printed listings!),
they will be able to hack device drivers for
their then common output devices and to be
able to print all of Knuths works in original
typographical quality!
That is real deep reason for Knuth's TeX - longevity of information.
Re:I only see (Score:2)
No better. Metal gets corroded by water (worse yet: saline water), melted by fire, cracked by cold etc.
Besides rock, which has proven pretty good throughout the ages, there's one thing that could hold up the promise, and that's mineral paper [stanford.edu]. (Aka, asbestos paper.)
Karma karma karma karma karmeleon: it comes and goes, it comes and goes.
Re:Christ, were you even awake in science class? (Score:2)
Hardly affordable metals aren't they? I'm talking something remotelly accessible, not gold-plated disks to be sent outter space...
Where YOU awake in economics class?
Karma karma karma karma karmeleon: it comes and goes, it comes and goes.
Orwell (Score:2)
Doubleplusungood! Thought Police! Here! I have found a crimethinker! He must be an agent of Emmanuel Goldstein, spreading misinformation!
Put Doctor K with his brother in the Castle!
__
Re:Yes, the Louvre is a piracy bunker! (Score:2)
This might be a good thing (Score:2)
Digital rot of our records, I mean.
Think about it -- what do we have to pass on to future generations of the past 20-30 years? Boy George, N'Sync, Lyndon LaRouche, Hare Krishnas, Monica Lewinsky, Rush Limbaugh, Al Gore, Rob "CmdrTaco" Malda...
It might be a good idea for ALL these things to slowly melt away ...
"Beware by whom you are called sane."
answer: (Score:4)
The Rulers (Score:2)
Currently I'm about to pick up a used Super-8 projector to show some films that are in great shape.
Also just got a 1930's Burroughs adding machine for $15 from a hamfest that, with a few drops of oil and cleaning is in 'like new' condition and will probably be in working condition hundreds of years from now if kept in the right environment (room temp, low light and humidity - basements, attics, garages and sheds are hell on that stuff).
Re:Does anyone care (Score:2)
I'm not worried about what records will survive and won't survive from our era. The Romans, the Greeks, they didn't worry about such things. They worried about what legacy they would leave for the future (fat lot of good it did them) which is what kind of world they were leaving for their children. This is far more important, IMHO.
I have the archives,in M$Word 2.0,uh never mind (Score:2)
If its not the medium (read an 8" diskette lately? How about a 14" 5MB cartridge? How about a reel of mag tape?) its the software (M$ Word documents formats were deliberately sabotaged to force people to migrate to the newer versions. [I don't know anyone who actually needed M$ Word '97 until they found that they had to upgrade when M$'s biggest clients who'd got their copies for dirt.])
There will be thousand year old documents and last week's flimsies and nothing in between. Just an Orwellian silent testimony to greed and obsolence planned and otherwise.
But that said. have we said or written down anything worth keeping?
micro-printing (Score:2)
Etch text, not binary codes.
The future can read this with a computer or magnifying glass.
only copied stuff is "saved" (Score:3)
Even paper distintigrates, albeit in centuries.
Only a tiny fraction of stuff is copied now or then.
Re:Historical value of recent archives (Score:2)
We need a standard for long term storage (Score:4)
LTSS 2.0 might have whizbang new file formats and storage media which supports 100 times as much information density, but it must be compatible with version 1.0.
LTSS 1.0 could support WAV, MP3, GIF, TIFF, Text/ASCII, Text/Unicode, HTML version whatever, and perhaps even Java for interpretation of abirtrary file formats. The media, CD-R, or perhaps one of the writeable DVD formats when they mature.
-josh
Re:Simple answer: "No." The reason should scare yo (Score:2)
Take the Ems telegram for example, seriously altered, and sparked a war between France and what would become Germany. Of course, we now know that it was altered, but at the time no one knew what happened.
If you think that people altering documents for their own good is anything new and ruins good historical records, you need to wise up and take a history class. This is nothing new, and we still have a good idea of what happened.
Institutions... (Score:2)
Kevin Fox
--
Not what you store it on, but *where* you store it (Score:2)
If we want to save data we need to make redundant copies, in a form that is resistant to electromagnetic radiation (say, microetched in carbon, silicon, or other stable element), and put it into a heliocentric orbit 1 radii behind or ahead of the earth's orbit (this way it's not in a trojan point, which could result in collision damage, but is still in a 'mathematically likely' place).
Most of the corrosive factors would be left behind on Earth, and the data would be stable for the long haul. Alternatively, we could put data on the moon, where it would be stable until a meteor hit it or covered it up, likely tens or hundreds of millions of years, and if we put several down, they'd last longer.
Hmm, maybe a big micro-etched monolith buried just under the surface...
Kevin Fox
--
Re:Is it necessary? (Score:2)
Re:only copied stuff is "saved" (Score:3)
Slightly OT: Check out Hillis' 10,000yr clock (Score:2)
media matters too much (Score:2)
For a virtual world we ought to separate the infos from the media. We could store data and execute programs some computers and use the majority result. See Askemos [askemos.org] how this will work.
Once we are at it, we might find that files are worse than paper for another reason. We better had "write once" files. - If reusable paper were better that nomal paper, we would have it in the stores. Enough cycles of invention went over it already.
Isn't this scary (Score:5)
It reminds me of the Arnold Swarzen...(?) movie, "The Running Man". He's a police helicopter pilot who refuses to shoot unarmed people involved in a food riot. The powers that be manipulate the video tape evidence to make it appear that he massacres the people instead. People are shown the tape and cry for his death in a game show type fashion until some revolutionaries are able to show the real tape by hacking into the communications channel.
The temporality of public records has very serious implications for our social structure. If the only record of your speeding ticket is an entry in a database, what happens when a glitch makes you a drunken sloth who doesn't pay child support. If the entry showing Bush's drug convictions get deleted, will there be no other record. Trust me on this, email is a politician's dream. Everything from here on has plausible deniability.
Re:Legacy Databases (Score:2)
The day-to-day information that we produce is the stuff that makes genealogists go nuts. It's the stuff that leads to books like "Roots". Biographies of people who, to themselves, seemingly did nothing with their lives, yet looking back ath them a hundred years later we see how extraordinary they were.
Should -everything- be saved? No. Personal correspondance with friends and family should. (and hell, I have -every- piece of email that I've received at work over the last year saved. Talking roughly 500MB or so of gzipped archives (which balloon to about 1.5G)).
How much of it would you want to read? (Score:2)
Moore law says: don't read originals, copy them (Score:2)
As a matter of fact, each time I get a new computer, I copy all the stuff from the old one, and it takes only a fraction of the space. The 40MB of my first (Atari ST) hard disk are there. The 160MB of my first Mac hard disk (120MB left after I copied the Atari hard disk onto it) are there. And so on.
The real issue is binary formats that have been forgotten. For instance, I have source code of programs I wrote in GFA Basic (a Basic for the Atari ST, in case you wonder.) But emulators come to the rescue there. Today, I can run Atari programs faster than on the real machine.
Some information has to be destroyed. (Score:2)
The amount of information (archives) that a state amasses, is simply astounding, and thats just the the bits that goes into the archives; at least 90% of all paperwork is scrapped even before that.
An example; I helped a scholar do some statistics on black market crime during and after the the war;
He examined a single, lower court, in the period from 1940-1953. "Only" 8000 cases went through this court, but just the verdicts alone, averaging 3 pages per case, amounted to 25000 pages, bound into fifty, 500 page tomes. Each of these cases, would also have generated a "file", containing eg. police interrogations, wiretapping records, anonymous letters, forensic evidence, case evidence, court orders, affadavits, etc. A really conservative estimate would be, that each case, would have generated at least 20-40 pages, meaning that just this single court, in a few years, could have archived 100.000 - 200.000 pages. A totally impractical thing to do. Therefore these files were "cleansed", before being archived.
If all those papers that public institutions produces were preserved, we would be swamped in archives. Some stuff simply has to go.
Old-style paper archives has physical storage problem. Modern "bit-based" archives should in theory, be less burdenend by this. (200.000 pages should fit handily into a single cdrom.)
But on the other hand, modern information systems makes it so much easier to generate, and preserve information. (just think of many gigabytes of information a single company has on its servers)
How many emails is sent every day? 5-10-20 millions? If just a fraction of these, say 100.000, were preserved every day, think how many freaking million emails that would be during a short period of 50 years. But more importantly, how many (and which) emails would posterity need, to say something about our time, and the social pattern behind the phenomenon; email?
The main problem with digital archives, is the same as with paper archives; You can't, and shouldn't try to preserve everything.
I don't doubt, that over time, even the majority of that information selected to be preserved, will be lost, due to bit-rot, war, fire, carelesness, natural disasters etc, during the next 1000 years. But even if just, a tiny, tiny, fraction of this is preserved, there would be "enough" information, about our time, for the historians to make a good overall picture.
A single, modern "Statistical Yearbook", probably contains more demographic information, than all medieval archives put together.
A modern public library, probably contains more works, and written information about the last 100 years, than have been preserved, from when man began to write, until the Middle Age. Still, a lot can be said about the Roman Empire, even though so precious little in writing has survived.
So to reduce future archives to a manageble size, the majority of information simply has to be discarded. Then it is more likely, that there will be funding, for preserving the rest in a proper way.
Consider the amount of time, money, blood, sweat and technology, that goes into carefully extract scrolls from the Pompeii site, and make them readeable, it should be a "trivial" task to recover any kind of non-encryptet data, no matter what digital media it resides on. However, the cost of doing so may not be trivial. Just think on how many data formats, future historians would need to reverse engineer, just to cover this last decade.
"Remember: the victors always have and always will rewrite history as much as possible."
How I have come to loathe this dogma.
Originally, it stems from the fact, that sometimes only one parts "history" survived from ancient times until today (Athens, Ancient Egypt springs to mind).
But the dogma really isn't true anymore; First, in democtratic countries, it is impossible for the state to directly control, what history is written. Secondly, after having dealt with the massive "memory" rewrites among former Waffen-SS soldiers, I can only conclude, that the loosers are just as eager as the winners to rewrite history; there has been a huge amount of revisionist "history books" written since the 2. WW. ended. From outright holocaust denial, to apologeic "Waffen-SS coffee-table books", where the W-SS soldiers are portraied as just a bunch of happy, anti-communistic boy-scouts, on a picnick in the USSR. Noone of them were ever nazis, or anti-semetic, they never saw any warcrimes (except those the russions made), blah, blah, blah. Total denial of facts.
So a better dogma would be:
"Remember: both the winners and looser always have and always will rewrite history as much as possible."
Historians know this of course.
Re:Moore law says: don't read originals, copy them (Score:2)
It would take a few seconds to copy the equivalent amount of data stored on 100 floppies but it wouldn't take a few seconds to copy 100 floppies. The distinction is important for archivists, who might have, say, a building full of 9 track tapes to convert, a process which could take years.
Re:Simple answer: "No." The reason should scare yo (Score:2)
Outlook to text (Score:2)
Some thoughts (Score:4)
One problem with archiving digital communications is the volume. One of the problems that were found during the many Clinton investigations was, when e-mail was subpoenaed, separating the wheat from the chaff. All the mail was backed up onto tapes, which weren't very well marked. And the first searches were done on subject lines. Quite a bit of relevant mail was missed, and turned up years later when people actually sat down and read every message.
The National Archives (here in the USA) is worried about preserving data. The various software and hardware formats used over the years make it difficult to track and retrieve the data. NASA has spent a fair amount of money moving old planetary exploration data from tapes to optical disks, and then to CD. My father worked on a project at DMA (now NIMA) to do the same thing there.
Paper tape (Score:3)
Sometimes the answer to your question about how do we do X with technology can be found by remembering the history of technology. In this case, what might be a better long-term storage medium than magnetic or optical media is good 'ole paper tape. Now, some research should probably be done to increase both the durability of the tape material and the density of information stored on it, but it is the best solution I can think of, and probably the easiest to decipher by archaeologists of the far future.
--
SecretAsianMan (54.5% Slashdot pure)
Re:Papyrus (Score:2)
There's actually some of this going on today.
I'm a bit fuzzy on details, but a few years ago I heard (from someone who worked in the field) about a project to resurrect old LANDSAT tapes from the 1970s. Someone figured out that the old data would make a great baseline for climate change studies, and the raw data could be processed in ways simply not possible 25 years ago.
The tapes were still around, stuck in a warehouse somewhere., To get them into a readable condition, they had to be slowly baked (in pizza ovens!) to drive out moisture they had absorbed, then scraped with a sapphire blade to...well, I forget why. Scrape off some gunk.
I believe they managed to dig some old recorders out of the scrapheap and get them working with the help of some old hands.
Wish I could recall more details, but that's all I know.
Tom Swiss | the infamous tms | http://www.infamous.net/
One Word: Echelon (Score:2)
Ever wonder what they do with all those communications? Maybe they can put them in escrow for 200 years :)
My Idea (Score:2)
Everything2 is great for recording encyclopedic sort of knownledge. What I'd really like to see is something that is designed just like Everything2, but instead it records *experiences*. Everybody writes experience and event nodes, and eventually we have a living history of everything that ever happened. Sure a lot of that will be irrelevant, but just think of all the correlations and connections that could be made. Sort of like 6 degrees of separation, but for real life events.
Automated chaff generation doesn't help (Score:3)
No kidding. I'd hate to be in Deja/Google/whoever's shoes, trying to archive useful data, in face of terabytes of "Nude Asian Teens" email generated -- literally -- completely automatically at the click of a mouse button. Especially since the most useful spam filtering methods (outright router blocks, keyword triggers, a bullet to the head of the marketing agent) are frowned upon by nice people.
Paper libraries have a "volume" problem because the media itself takes up so much space, and must be carefully stored. Digital libraries have a "volume" problem because any old jackass can easily create fifty times the amount of information that's worth keeping, and it must be winnowed out by a human.
Just my rant today (cleaning out another twelve spam emails).
The problem looms even closer than that (Score:2)
So forget this problem of losing our digital records as a society, what about losing my personal identity?
I still go back and look at physical letters of mine from 10+ years ago, but email from as recent as 1994 is hard to find. That frightens me, frankly.
the paradox of digital media (Score:2)
Digital archives that last forever... (Score:2)
But you have to get away from the mindset that seeks a "wearever" medium, everlasting standards, and indefinitely available hardware. That is the naive approach.
The word is "living archives". The archivists' work is never done.
The approach that works is just to regenerate all data from media that is wearing out, obsolescent media, and obsolescent standards - before it is in danger of being lost. This must be a constant process of renewal. Since the data is digital, and anyone with the slightest imagination would store redundant copies in physically separated locations, the process is lossless.
So when 3.5" diskettes become well established, and 5.25" diskettes start looking like orphans, you redub everything from 5.25" to 3.5". Then the same thing when CDs overtake 3.5" diskettes. And on and on (I seriously doubt CDs are forever in any sense of the word).
The trick is to know when the time is right each time. I won't minimize the problem. But the watchword is "be conservative".
Re:We need a standard for long term storage (Score:2)
Theoretical computer science does tell us some things which appear to be absolute. One of those is that "information is information is information".
The big difference between a standardised digital archive and microfiche is that the former is pure information. It will be automatically convertible to more sophisticated forms of storage in the future. Digitising microfiche archives is possible, but still requires lots of physical work which is only partly automatable.
Having said that, it is also possible to identify a big limitation in the proposed "LTSS" - the same thing which is missing from the web today - rich metadata (this isn't just about XML, btw). Do a search on the "semantic web" if you're curious....
Even print media have become "transient" (Score:2)
Twenty years or so ago, the Smithsonian museum had an exhibit about fiber optics that included a working model of Alexander Bell's "light phone" (it mechanically modulated a beam of sunlight) and his original lab notebook (borrowed from Bell Labs' engineering records). The notebook was still legible because (a) the paper was acid-free and (b) the ink was pigment-based. Even though I keep a notebook, it will not be legible in 100 years (perhaps one of my great grandchildren will be interested) because either (a) the high-acid paper will have decomposed or (b) the parts written with dye-based ink will have faded.
The fairly recent PBS documentary on the US Civil War was based in large part on letters and journals written by soldiers using (you guessed it!) acid-free paper and pigment-based ink.
Make tomorrow's history! Write letters and keep journals using acid-free paper and pigment-based ink -- if it's all that survives, it will be the authoritative material on the typical daily life!
Re:Nickel (was Microfilm baby...) (Score:2)
The Rosetta Disk [longnow.org]
AJ
Historians are historians (Score:2)
CD's fade away? (Score:2)
CD-R's and CD-RW disks record via a dye that changes color and reflectivity with heat from the laser. This dye can destablise under light and heat. So keeping your CD-R's and CD-RW's in a dark cool place would be a good idea. Also the more they are 'played' the shorter their lifespan might be. So make a backup copy of any CD-R/RW you want to keep. CD-R's might be more stable than CD-RW's.
It's Not Just Digital -- Microfilm Sucks as Well (Score:3)
There's a good review [nybooks.com] of a Nicholson Baker rant against Librarians in general for their sins of deliberately pulping the paper records of the past 130 years and replacing them with decomposing and badly executed microfilm facsimiles.
It seems that Vannevar Bush's infatuation with microfilm was shared by many in the WW2 OSS community, and this seems to have led to a misguided attempt to replace papers and books with microfilm in the interests of "efficiency".
Not only media problems... (Score:2)
Of course, the NYTimes, etc, have archive searches as a premium service, but there are just tons of media outlets that don't seem to archive, or if they do, don't seem concerned with letting people get at it. This seems like at least as much of a concern as degrading media: the organization and maintainence of archives in the FIRST place.
Digital preservation is a well-known issue... (Score:2)
The USA's Library of Congress [loc.gov] Preservation Reformatting Division [loc.gov] is digitizing many items for preservation, and you can be sure that they're concerned that the digital preservation will be at least as effective as the original (analog, paper, whatever) form.
One of the current projects of the Research Libraries Group [rlg.org] is data preservation [rlg.org]. The RLG is an international group formed originally by Columbia, Harvard, and Yale universities and The New York Public Library in 1975, with current members from academia, government archives, public and private sector historical organizations.
A google [google.com] search on digital data preservation [google.com] gives plenty more linkage to groups actively looking at the issues involved in digital storage.
Of course, there is still a huge volume of personal and corporate data that will no doubt degrade to dust. For that, we all need to take the approach [slashdot.org] of wiredog to keep our personal data accessible by refreshing the media as technology advances.
Naturally, since this is Slashdot, all of this has been already covered [slashdot.org]. This article [slashdot.org] was a particularly good treatment of the topic and was posted as a followup to an older [slashdot.org] Ask Slashdot.
Really, how different will it be if the future only has the preserved personal effects and communications of an insignificant fraction of the general population? Today, archeologists make a career out of extrapolating whole civilizations out of building foundations and shards of pottery.
So, with a little care, I'm confident that my own data will be happily accessible as long as I need it. After that, the future will take care of itself.
Re:Natural selection (Score:5)
I've worked on research projects whose primary source was day-to-day accounting records of a small business running in Egypt during the 11th century. The records were preserved in part because they were at the bottom of a trash pile. The records gave us a huge amount of information about everything from transport methods to the ability of the state to collect tax. Most of the 'important information' from that period which people though was worth preserving revolves around which ruler stomped which other ruler's butt. Our 'trivial information' gave us a lot of stuff which we knew nothing about before, stuff which helped explain why ruler X had the economic wherewithall to stomp ruler Y's butt and, well, more interestingly, what it was like to live under ruler X or Y.
The same applies today. Yeah, a record of what your family ate for dinner for the past two weeks is truly trivial. But what it will say about daily life, the transport of food, diet, cooking technology, food storage & a whole lot more about life in the early 21st century might be invaluable to some historian in a thousand years.
Your 'trivial information' is someone elses data goldmine and vice versa. One of the things I really like about computers is they allow you to keep a lot of personal shit you might otherwise have to trash because it gets bulky. The chances that I'll hang onto all my mail & all my parent's mail and all my grandparents mail is pretty good when it fits onto a CD rather than choking up my small apartment with boxes. The chances that some future historian will get to read ordinary everyday mail rather than just the mail of presidents and kings in a thousand years is getting better.
Re:Historical value of recent archives (Score:2)
I beg your pardon, but when I was doing research on the dietary habits in Early Modern France, someone's grocery list would have been of extreme historical value! Luckily we do have some petitions for aid written to city authorities in which the petitioners detail the household consumption of bread wine etc ...
Make a drive? (Score:2)
On optical media. (Score:3)
The real reason that old magnetic tape is hard to read now is that it was never a great format in the first place. The stuff falls apart. My last employer had an old HP reel-to-reel machine for reading data on tapes from a company we had purchased, but the tapes were so old that the chemicals on the tape itself turned to dust and fell off. This is not a problem with optical storage. Optical storage also has the option of being dedicated in very small spaces, unlike the van sized tape players of old.
Life is also not a big issue with optical media, because just as the books of the Medici's were recopied over and over into new languages and on better bindings, so can data be quickly copied from old optical media onto newer formats.
The answer to this question is pretty obvious... (Score:5)
The analogy would be to read emails from, say, the white house in 200 years. Do you think the white house is saving their emails? You bet. Do we have lots of examples of (from the general public) letters from 200 years ago? Certainly not as many as there will be emails in the future. Usenet archives, digital backups stored in basements, most emails are being stored two or more times at two or more places. I don't quite understand why someone would think that just because it isn't on paper, it isn't going to keep. We are going to have far more emails stored in the future than we will know what to do with.
As society we think of ourselves as individuals to be pretty important, but lets face it, for the vast majority of us, no one is going to care in 150 years. With that in mind, the digital age is storing far more records than ever before and the future holds a new paradigm of historical record. I almost lament that I wasn't born 150 years after the advent of the digital age where high resolution movies will look as good 1000 years from now as they do today.
-Moondog
This is a known problem (Score:4)
Likewise, various people are trying to shut down the MAME ROM sites, but a lot of the hardware ROMs are deteriorating now and many of those games, which represent a golden age of creativity and a technical wonder of resource usage, will be gone forever. Kinda makes you sick, doesn't it?
Re:Is it necessary? (Score:2)
Enter VMS, which automatically saves every version of a file, until you manually delete them. If Unix had not wiped out VMS, everybody would have every file they ever worked on.
Word actually does have a versioning feature which saves every version you worked on if you enable it.
My OS is going to have infinite versioning and journalling capabilities, so you can undo any change you ever made (not just on "file save" boundaries). When VMS was developed the typical hard drive was under 100 MB, and now that they sell 100 GB drives for a dime a dozen, we have the room to save everything. Why current OS have usage models which encourage people to delete everything is beyond me.
Is it necessary? (Score:4)
While I'm all for archiving data for future historical analysis, I think it's fairly certain that IM logs, "how's it goin?" e-mails, and detailed transcripts of #40yearoldsinglebaldguys will not be very useful to historians in three hundred years. Yes, they tell about our culture and practices, and yes they might be interesting, but we don't need all of it to extrapolate those conclusions. There is simply no room to store the vast quantity of information generated on the Internet on a daily basis, and considering the fact that 99.998% of it is of little value, I think that we can safely do without it.
Things are still floating around from the old days. We have Usenet archives from the 80s, and text files from even earlier. We can learn a lot about the culture based on those. Things that grab the public consciousness tend to around. They get mirrored, printed out, saved on disk, etc.
Does there need to be a giant warehouse that contains vacuum-sealed printouts of every wise thing said on the internet?
No. No, there doesn't.
Re:It's not the media, it's the SOFTWARE. (Score:2)
The bigger question isn't media, but sofware. I'm very confident we'll be able to get our files from ISO9660 discs, but I already have a bunch of WordStar and old MacWrite/MacPaint files I can't open and it's only been a decade. We'll be able to retrieve the raw data, but will be actually be able to interpret and make use of it?
Well, there are two issues here. One is keeping a readable copy of the software, the other is being able to run it. Since most software programs are used by large numbers of people, it seems likely that someone would have the foresight to keep a copy of the software to interpret the data along with the data itself. Running it also shouldn't really be a problem for future generations. Presumably, someone will have a copy of the specs for the architecture for which the software was written, and an emulator can be created. Of course, if the software's source is available, it would be even easier.
Also, reverse engineering a data format isn't that hard anyway. If you looked at the raw data of your MacWrite files, I'm sure you'd find your text in ASCII somewhere, possibly with embedded formatting information. Non-textual data is more difficult, but still possible, particular if you have some fragments of information about the data format to go on.
Remember delay tubes? (Score:2)
Perhaps we can still use the same technique to solve the data archiving problem: Just broadcast all our data into space. To read it, all we need to do is invent FTL drive, pop out to the right point in time and read the data as it goes by.
I'm sure we could find other uses for the FTL to help recover the R&D investment.
Re:Some thoughts (Score:2)
And while we're on Vietnam, where would Rage Against The Machine be without the monk setting himself on fire?
--
Re:Does anyone care (Score:3)
The problem with planning for the future is that it is hard to know today what will be important tomorrow. Perhaps the insignificant trolls on Slashdot will be of great import in the future (and, no, I'm not referring mainly to Jon Katz articles). Who woulda thunk that an accounting ledger from ancient mesopotamia would be of any interest 2500 years later?
But won't the government do this for us? (Score:2)
I thought that was what things like Echelon and Carnivore were for????
Is an 8 KB picture worth 8 KB of words? (Score:2)
No, the trick is that a picture is worth 1000 words. Since graphics usually compress worse than text (limited dictionary)
The latest wavelet compression techniques [jpeg.org] can compress a good-sized color image to 8 kilobytes, or the size of a thousand English words plus light markup.
Open, documented textual graphics format (Score:2)
On July 20, 1969, Neil Armstrong was the first man to walk on the surface of the moon. Here is a picture, in an open, documented graphics format
And the format is called ASCII art [everything2.com]. Just use this simple program [everything2.com] to convert your 1-bit .bmp format images to images made of standard ASCII characters.
So archive the IEEE/ISO standards. (Score:2)
You just don't want to accept random binary data that you would have to retain a reader for as well.
If binary is the problem, uuencode is the solution.
If proprietary formats are the problem, then documented, unencumbered formats such as PNG, JPEG, FLAC, and Ogg Vorbis are the solution. Just make sure to archive documents (such as ISO and IEEE standards) that can be used to create a reader.
So emulate the software. (Score:2)
The bigger question isn't media, but sof[t]ware. I already have a bunch of WordStar and old MacWrite/MacPaint files I can't open... will [we] actually be able to interpret and make use of it?
For older formats, you can always emulate the computer for which the viewer software was designed, or write a new viewer from the format documentation. For example, QuickTime 4 can open MacPaint files, and so can a short C program I wrote. Remember, if you want to archive something, make sure you have the format documentation (or the viewer software and the architecture documentation) so that future generations will be able to create a usable viewer. (IEEE and ISO standards are Good Things[0].)
About five years ago I still had an old floppy controller with an odd WD chip on it that could talk to it using OS-9.
So install Mac OS X (the successor to Mac OS 9) on your machine and read that floppy.
Oh, you were talking about that OS 9.
[0] GOOD THING is U.S. Trademark No. 75,516,347 registered to Martha Stewart Living Omnimedia LLC. (Look it up at TESS [uspto.gov].)
Good idea, with these changes: (Score:2)
LTSS 1.0 could support WAV, MP3
s/MP3/Ogg Vorbis/ [vorbis.com]
GIF
s/GIF/PNG/ [burnallgifs.org] because PNG is better documented and supports 24-bit color and alpha transparency. You partially address this with
TIFF
but s/TIFF/PNG/ because even without TIFF's LZW codec, TIFF is much larger than PNG and not as well standardized.
Text/ASCII
Non-European language advocates would complain.
Text/Unicode
Better. Thank you. This solves the script issue, but in what natural language would information be stored? How is it a valid assumption that future generations can read format specs written in US English of A.D. 2001 or in UK English of A.D. 2001?
HTML version whatever
Make sure it's run through W3C's HTML Validator [w3.org] if you want to archive it. MSHTML is a Bad Thing.
and perhaps even Java for interpretation of abirtrary [sic] file formats.
The Java(TM) langauge does not have the wealth of alternative implementations that the C language has. Both are nearly Turing complete (full Turing completeness requires unbounded storage) and equally fast when compiled to a native instruction set [gnu.org].
The answer is simple... (Score:2)
2315 AD: It would appear that the entire society was obsessed with "NAKED HORNY CHEARLEEDERS WET AND WAITING FOR U!!!!!!!!!!", "online casinos", messages from some person named "bounce@" and worshipped a diety called "Viagra". No wonder they vaporized themselves.
Re:Is it necessary? (Score:5)
Of course the flip side of this is that it's not always possible to tell who will be considered interesting in the future. In many cases, the most interesting use of archives is to look at the work of interesting people while they were working their way up and weren't of broad enough interest to attract major attention. Nobody knew that a 25 year old patent examiner named Albert Einstein was about to become a scientific star, but because we have his personal letters we can find out what he was doing scientifically and personally.
You never know if the next great author might be posting his early, great works to some fan e-mail list because he can't get his foot in the door at a major publisher. Maybe the next great debator is getting started in flamewars on Slashdot. Maybe the next great OS designer is getting into arguments with established academics on USENET. Oh, wait, that already happened, and we can only read the argument [www.dina.dk] because somebody though to archive it. Maybe the next great philosopher who will be mostly ignored for 100 years is already publishing his early thoughts somewhere on the web. You can't always tell what will be valuable to the future until well after the fact, so preserving as much as possible is still a really good idea.
A truly wonderful example of this kind of thing are the early works of JRR Tolkein. The early history of the Silmarillion is absolutely fascinating and a wonderful example of the development of a literary theme. That's a work that wasn't published for over 50 years after it was started, but some of the earliest drafts still exist. Because those drafts are available, it's possible to see how it developed. Will the same thing happen when authors write everything in Word and write over old versions every time they change anything? How about if they're still very careful about keeping copies of early drafts but the formats change so much that they can't be read anymore?
Historical Data has Always Been Volitile (Score:2)
This stuff has always been volitile. We have a fraction of the historical data we would like to have from any time period. Yes, the letters of the Medici are still around and available, similarly the corresponsdence of the major players of our time will be archived (either electronically or in hard copy. Probably both.) The letters of the common man were as often discarded in times past as e-mail is today. Some of it will not doubt still be around (just as the data on many of those eight inch disks still survives on more modern media today), but the vast majority will be lost. This is fine, especially since there is a finite amount of data that historians can analyse anyway. Generally speaking it is nearly impossible to tell what will or will not be historically sigifigant from the point of event origins anyway. I would venture to say that considering the level of literacy in our culture today, and the varied data storage mediums available, historians will have far more data from our time than current historians have from anytime before World War II.
Of course there will (Score:2)
The trick will be recalling the data from those organizations.
DanH
Cav Pilot's Reference Page [cavalrypilot.com]
Broadcast it to Space (Score:2)
Oh, don't worry about it... (Score:2)
Right now, the NSA is reading and cataloging all of our private e-mails -- there will be records of everything we say for generations to come!
"Grandpa, what was a EULA?"
Feudalism (Score:2)
I strongly agree with AC's argument. But forms of government are a really bad example. How many Americans have any understanding of how their government works? Even those who have taken the time to study it (mostly naturalized citizens, who are required to know something about this stuff, unlike "real" Americans), mostly just read the Constitution and related documents -- which have roughly the same relation to actual government as physical chemistry has to cooking.
Feudalism is an even worse example. The word, in its modern sense, was first used by French revolutionaries, to describe the aristocratic regime they had just overthrown. (Before that, it was a legal term, applying to a certain kind of property law.) Since then there have been endless redefinitions of the term, all of them pretty conflicting.
A better example would be based on simple cultural icons. In 500 years, how many people will know that Neal Armstrong was a real person and Luke Skywalker wasn't?
__
It's not about what's lost.... (Score:2)
The archeological record always seems to improve with technology. From stone etchings to written scrolls to printed matter and photography and on into the 20th century, the more technology people had, the better record they left of themselves.
Re:Oh, don't worry about it... (Score:2)
Every wire inside your house has the potential of recording everything you do and sending it to an illuminati communication location to be stored. Hell, they've known everything well before the United States was formed.
FNORD!
NARA looks to be on the ball (Score:2)
To do so, they are using a new computer language called eXtensible Markup Language, or XML. It is a way of marking up electronic documents with easily understood tags instead of coding dependent on what will some day be obsolete software.
Naturally, NARA's main focus is the archiving of documents that are mainly of historical significance to Americans.
So what? (Score:2)
Re:Some thoughts (Score:3)
So I'm sitting down and thinking, but I still don't understand their position. I can appreciate both the importance of ASCII text and its accessibility (hell, I still use lynx to browse the web), but I can't understand why you would restrict yourself to only text.
Consider the following:
On July 20, 1969, Neil Armstrong was the first man to walk on the surface of the moon.
--versus--
On July 20, 1969, Neil Armstrong was the first man to walk on the surface of the moon. Here is a picture, in an open, documented graphics format.
There's just too much history that's more than just pure text. I can understand trying to make as much material as possible available as text, but you can't let such a decision allow you to exclude relevant materials that're more than just text.
Libraries do this all the time... (Score:2)
Basically, the five year plan means rotating the data from one media type to a new media type... waiting every five years. Although computers move from day to day, the method of data storage and retrieval remain approx. the same within a five year period. As long as the data is updated every five years or so, the data are always available. The price of keeping the data in this state of never-ending movement would be somewhat static, as once a new method of storage comes of age, and is a standard, it is pretty cheap. The real price comes from manpower. Which... could be solved by spending some time developing a software system that could be altered, on command, to handle the new media... Enter Linux!
I could keep going on and on about this beautiful system, but I grow weary of trying to remember all of this stuff, and typing it, and looking like I am still working on something useful!
Hieroglyphs (Score:2)
Besides, data formats are nothing, historians have decoded long forgotten scripts and languages which no-one speaks anymore. I think it will be comparatively easy to get at the files on a CDROM, 500 years from now.
They'll just put the thing into some sort of a 3-D scanner and work on the computer copy... "Oh lock these dents are 1's, and those are 0's and they write them in a spiral." Sure it may be tough work (file system, data formats), but they'll also have very sophisticated technology to analyze these things. They might just have to click on the "unknown media wizzard" and get all the files. ;)
Another problem is deteriorating media, but on a historic level I don't think it matters much. Current data recovery companies can do amazing things already: restoring hard drives from totally burned-out PCs, or restoring data which has been overwritten multiple times.
It's one problem to keep your data so you can readily access it in 50 years, but I think on the scale interesting for historians we have no problem at all.
Computers and History (Score:2)
The economy may be built on computers, but rest assured social record-keeping is not. For important documents, and permanent information, paper copies are still much preferred over their electronic cousins.
Think about the last time you read a novel on your laptop, instead of picking up the book. And the last contract you signed? It wasn't on those digital pads you find at Best Buy for signing receipts. Paper is still king, and it will be for years. It never gets obsolete, and it lasts just as long as anything else we have.
A new year calls for a new signature.
Re:Is it necessary? (Score:3)
While the histories, news articles, and official documents of a given era are very important and informative, it is also necessary to the personal accounts from the people involved in the society at the time to help provide perspective, and to help identify biases in the 'official' accounts.
Considering how valuable even the pedestrian of documents are from e.g. 3000 BC, I imagine that today's equivalent will be of equal value to historians in the 7000 AD.
-----
D. Fischer
It's not the media, it's the SOFTWARE. (Score:4)
Even two or three hundred years from now, a reasonably skilled technician or at worst a team of them will be able to dig up a CD mechanism from somewhere, fix it up and get it reading data. CD mechnisms are like Ford's Model T -- only much more common -- and let's face it, there are still a reasonable number of Model T's running around to auto shows, and there isn't nearly the historical incentive to keep a Model T running that there is to ensure that there will always be a CD-ROM reader running somewhere.
And it's likely that if most people are like I am (I value my data and my work) they will continue to migrate data to new formats as they emerge.
The bigger question isn't media, but sofware. I'm very confident we'll be able to get our files from ISO9660 discs, but I already have a bunch of WordStar and old MacWrite/MacPaint files I can't open and it's only been a decade. We'll be able to retrieve the raw data, but will be actually be able to interpret and make use of it?
P.S. I still have an old Siemens 8" floppy drive, single-sided, hard sector. About five years ago I still had an old floppy controller with an odd WD chip on it that could talk to it using OS-9. No way to talk to it with my Linux box, though...
And paper is safe? (Score:2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~ the real world is much simpler ~~
Re:Digital Photography (Score:2)
This is true of the hundred-year-old Bradyesque B&W's you mention, but the chemistry of color snapshots taken over the last fifty years makes them substantially less stable -- something to do with the organic dyes they use. Ever wonder why that old Kodachrome snappy of Grandpa from 1965 has that awful pink tinge? It'll only get worse, until eventually it's an unrecognizeable blob.
However, the older B&W stuff will just get a little yellowish. Or "sepia" if you prefer.
Deleting Archives (Score:2)
If I recall correctly, many attorneys are now advising clients to proactively delete archived email and other correspondence stored electronically, so that in presumed future legal actions the discovery process won't turn up incriminating evidence in the defendant's files.
The deletion, apparently, if prescheduled on all documents doesn't consititute obstruction of justice, whereas conscious destruction of only selected material may be construed as obstruction.
Part of the problem in maintaining a useful archive into the future is storage media, but a bigger part is the attitude that we should be afraid to allow our routine communications to be stored permanently.
Oh. And by the way, IANAL.
White House Email (Score:4)
Apparently, George W. was an inveterate user of email right up until the inauguration. At that point, he sent a farewell missive to his correspondents [slashdot.org], in effect saying he could no longer use email because all such correspondence would be a public record and he didn't want his private musings made public.
So, no, many important communications will not be retained, unless someone is placing a wiretap on the president's phone.
Get along, little datum! (Score:2)
Oh, and avoid the "no-copy" media.
Re:We need a standard for long term storage (Score:2)
I used to work for a company that built large plants that were expected to last 30+ years. We were greatly worried about how we were going to ensure that the original plans, operating procedures, maintenance records and the like were not just readable but usable and updatable down the road.
Basically the answer came back as - use only published, official, open standards. Even though they are not the greatest at least you can always reengineer the reader and the software because the designs are formally published (and not just as source code).
R.
Simple solution to digital recording (Score:3)
Re:On Bitrot (Score:4)
The digital age gives us great hope for preservation of everything, because we can copy sounds, images, motion and even DNA structures with perfect reproduction. But it will only be through the careful preservation of this information that future generations will be able to access it
If anything, and you can consider this a dig at DMCA if you like, it will be the number of copies of these artworks that will permit them to be preserved. Consider this: there is only one Mona Lisa -- if she fades, we can only guess at what her colour was. But there are millions of copies of Wing Commander IV. It's a relatively simple task to go through a few thousand of these, extract from each disc what data hasn't rot through, and compare it to the others. Combine that with huffman coding and CRCs and we can quickly reconstruct the original with perfection and certainty. You can't say that of the Venus DeMilo. And unlike other generations' copied mediums, we can trust the intermediary -- the cold, heartless eye of the scanner and OCR soft -- not to misspell anything or make up shit. Bemoan the need for proprietary copyrights if you like, but the digital age's perfect reproducability is the factor that will decide its permanent etching in the databases of the future.
Litigation kills history (Score:2)
The effect of this upon history is obvious. Originally, historians thought that the digital age would be great for doing history, because so much source material would be available. The growth in data warehouses and similar archives indicates that it's human nature not to throw anything away. (That, and my garage!) But now, to prevent the risk of exposure during email discovery, there won't be anything left.
What a shame.
Re:only copied stuff is "saved" (Score:2)
The stamped CD's will probably outlast paper records, but are only good for large-volume publications, not for the actual records that most interest historians. CD-R/RW and similar dye-based disks, properly stored, are probably going to outlive the technology to read them, but they are less stable than good paper.
Re:Oh, don't worry about it... (Score:2)
Simple answer: "No." The reason should scare you. (Score:4)
Digital records are favored by our corrupt, foreign-dominated Federal tyranny for one very simple reason:
It's terrifyingly easy to alter them, or to dispose of them entirely.
This is frightening, but true: As the well-known conservative George Orwell observed in his great novel 1984, "He who controls the past controls the future." The "Party" in 1984 devoted itself to doing exactly what the Clinton regime did: They went through all historical records, altering, falsifying, modifying, deleting.
No one will ever know what the Clinton death count really was. No one will ever know what really happened. The "records" are malleable. You can trust no information that comes from the government, because it's all been "massaged" and "fixed up".
Will there be historical records? Not in any meaningful sense: There will be something that looks a lot like such material, but it will be a work of pure fiction.
Goodbye, America. We were great while we lasted.