Long Now collects stories of the “digital dark age”. Originally these were kept in discussion boards, but we have moved this to our blog as of 02007 (Acknowledging that the present will in fact be known as the digital dark age, since all our digital data has no forward migration path.)
I thought that the first post in this category should be the story of some of our own data loss here at Long Now including the first version f this forum. We have learned a lot over the last decade, and are working now on some new architectures based on this learning.
Our problems stemmed from the fact that Long Now is a project-based institution that survives soley on on project grants. This means that we have little to no carryover infrastructure when there is a lull between projects. (we now have a full time sys-admin however)
So when we had a lull about 2 years ago we did not have any sysadmin staff for several months. This resulted in our servers getting compromised by hackers, age, and heat.
We had hackers using one of our open wikis for file trading gobbling up bandwidth, and we had a drive or two fail from excessive heat in a server (due to the fan dying).
The wiki problem was solved easily enough and no permanent damage was done. We learned our lesson about allowing public file uploads to our wikis though.
The drive problem was much more problematic. The drive was mirrored to allow for two copies of all the data should a problem occur. The problem was that the mirroring actually mirrored corrupted data leaving us with two drives with exactly the same worthless data on it. One of the things we lost was the first version of this forum.
We now do incremental backups on separate machines that allow us to revert to several different snapshots of the data moving backward in time. This way if we get corrupt data we can keep moving backward until we have a clean sample.
We also now backup much of our cultural resources (like our Rosetta linguistic database) to the Stanford servers thanks to Stanford librarian Mike Keller.
For the future we are now moving away from a server room all-together. We are moving all publically served data to a collocation facility. For the in house data we are now looking at distributing the in house server architecture under peoples desks. Each project workgroup at Long Now will get a Network Attached Server with 1Tb or more storage in it. Each workgroup will administer permissions on their NAS box. But we will only use half of the storage of each box for each workgroup. The other half will be a distributed RAID of sorts – backing up the rest of the offices data. This way no one box has all the data, and no one box can lose all the data.
We now have begun these efforts using Infrant’s ReadyNAS standalone RAID storage servers..
