Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Linux Software

Complete Filesystem Checkpointing? 36

Posted by Cliff from the filesystems-with-a-tau-axis dept.
polymath69 asks: "Living on the edge of Debian unstable means that updates sometimes break stuff, occasionally to an extent that is difficult to recover from. This got me thinking about treating the entire set of mounted filesystems as a transactional database. Mark state, try something which might be dangerous, test, and approve (commit) or panic (rollback). Obviously some filesystem support would be required, but with ext3 and reiserfs available, maybe the potential is already there. And such a system would need lots of disk space, but these days that's a demand easily granted. There's lots out there on process-level checkpointing, and even some stuff about system-level checkpointing, but all I've found on that was in the context of saving and restoring processes for a system freeze and restore. But I couldn't find anything on Google or SourceForge about doing this sort of temporary branching in the filesystem. Is this idea feasible? Is anyone working on it?"
This discussion has been archived. No new comments can be posted.

Complete Filesystem Checkpointing? More

Complete Filesystem Checkpointing?

Comments Filter:
  • I dont know if anyone is working on it, but it does sound like something microsoft tried with ME, where you can say wops everything is screwed up lets try and restore to yesterday. At first I though wow that kind of cool mainly because I do tech support for an ISP so I get calls like I hosed my computer fix it, so it would be easy to say just restore to last time I spoke with you. However that wasnt how it really worked, mainly what it did was eat up a lot more hard drive space then you would think it would and cause lots of problems with getting corrupted and what nut.
    Now maybe if you could have a delaid type raid mirror array, so the mirror gets updated once every 24 hours instead of every time the disk is accesed, then that may be something cool.

    • XP has system restore points, where it stores old drivers etc until you clean them up. Last time I did that I regained 2.5GB of disk space..

      Personally I think the answer for Debian is to stop using such a fragile packaging system, *shrug* :)

      (no, I don't mean switch to RPM, I mean things like not storing the entire package database for packages in one huge big file, and using that same file to keep track of what's installed. That's just asking to be nuked)
      • Keeping a separate package database in one huge (ASCII!) file is advantageous. By being separate from the individual packages, the dependencies are kept nicely so they can be easily referenced by the packaging tools when installing different packages. As an ASCII file, any sort of little error which might cause a problem is easily correctable, as opposed to other packaging systems.


        Regardless, this is not what the article submitter was talking about at all. He was talking about specific software packages in Debian unstable (bleeding-edge) that might have bugs that significantly affect the rest of the system. In such a case, a rollback process would be nice. This has nothing to do with the fact that the package database is stored in one big file.

        • Re:Hmmm. (Score:3, Insightful)

          by Fweeky ( 41046 )
          > Keeping a separate package database in one huge
          > (ASCII!) file is advantageous. By being separate
          > from the individual packages, the dependencies
          > are kept nicely so they can be easily referenced

          This is in no way an advantage of the implimentation format being one huge file. It could just as easily be one file per package, or one directory per package and multiple files referencing things like packing list, uninstall commands etc, plus a static, throwaway index file. It makes updates faster and a lot less disaster prone.

          > As an ASCII file, any sort of little error
          > which might cause a problem is easily
          > correctable, as opposed to other packaging
          > systems.

          That's great until your filesystem decides to truncate it to 0 bytes, and it's lots of fun looking for parse errors 40,000 lines into a file and guessing what might have been in the garbage that you find.

          > Regardless, this is not what the article
          > submitter was talking about at all.

          It is one of the dangers of Debian; upgrade to the latest unstable apt -> nuked package db, upgrade to the latest kernel and get fs corruption -> nuked package db. Find your ATA driver/chipset is buggy on handling large files -> nuked package db. Install a borked package -> nuked package db.
          • > Regardless, this is not what the article submitter was talking about at all.

            +1, Insightful.

            It is one of the dangers of Debian; upgrade to the latest unstable apt -> nuked package db,

            That's only a minor issue. Although I *have* had package DB corruption, it's usually been due to a daemon restart causing a sudden system freeze. And using dselect to rebuild the package database takes care of it.

            More serious is when an update breaks, say, Ethernet or Modem access and I can no longer connect to the Net to fetch repaired packages. That's when I'd love to be able to roll back to a known working configuration.

            • I too have had package db corruption, several times.

              > More serious is when an update breaks, say,
              > Ethernet or Modem access and I can no longer
              > connect to the Net to fetch repaired packages.
              > That's when I'd love to be able to roll back to a
              > known working configuration.

              portupgrade on FreeBSD creates a package of every port it upgrades before replacing it so the update can be rolled back if install breaks. Taring up everything on the packing list and leaving a backup package would be a lot simpler (and probably faster) to impliment than a transactional fs.
          • If your filesystem is truncating files, the packaging system is the least of your problems. Anyway, the very few times I have ever had a problem with parse errors were /easily/ resolvable. The line number is told in the error message, and you just go right to that line number in your favorite quality editor. The errors are pretty obvious.
            • > If your filesystem is truncating files, the
              > packaging system is the least of your problems

              This is what happens when you mount filesystems async on drives that do write behind caching and you get powercuts or crashes.

              Using one big fragile file for core system information is the same sort of braindamaged concept as a registry, or having /bin dynamically linked, or filling base system dirs with cruft.
    • There's a (somewhat) updated version of this in XP that lets you rollback to a previous existing state. However it's not perfect. The first time I installed XP Pro to test it at work I immediately grabed all the "critical updates" available. Well those "updates" ended up blowing out any and all TCP/IP stack functionality which rolling back could not fix. TCP/IP is considered a core component and is not included into the rollback information.

      Interestingly enough, XP is very selective on what information it does include into rollback information. It will revert to old drivers, old registry keys (not all), but it will not uninstall software. Durring my rollback attempts XP broke my Nero installation.

      Transactional filesystems would be fantastic I bet there are tons of little quirks to get around.

  • Chuq is working on it. (Score:4, Informative)

    by omega9 ( 138280 ) on Sunday February 10, 2002 @04:20PM (#2983307)
    Well, looks like this guy Chuq [chuq.com] is working on it [chuq.com]. He seems to be a kernal hacker that works for VERITAS.

    You can also find interesting filesystem info here [nondot.org]

    There's also work being done on TRAM [redwoodsoft.com] (Transactional RAM).
    • TRAM is just non-volatile solid-state. Slow, but not as slow as a hard drive. Bah.

      A cleaner solution to this person's problem would be more robust 'uninstall' functionality in Debian's package manager, to make it clean up its own mess. To 'rollback' a hard drive would not only be time-consuming, but it would require exclusive access to the hard drive for the duration of the rollback. Might as well just make a disk image once everything is running smooth, and rebuild it whenever there are significant changes.

    • Re:Chuq is working on it. (Score:3, Informative)

      by Tet ( 2721 )
      Well, looks like this guy Chuq is working on it. He seems to be a kernal hacker that works for VERITAS.

      Of course, Veritas have their own FlashSnap [veritas.com] product that does this for VxFS filesystems, and have just released it for Linux. It's a relatively pricey option, but it works well, and if you need this sort of functionality, the price is negligible.

  • You can do this in VMWare, but I'm not sure it's really the solution you want. :-)

    • Re:VMWare (Score:3, Informative)

      by dubl-u ( 51156 )
      I love, love, love VMWare for this. It's ctrl-Z for sysadmins.

      Lately I wanted to experiment with the various kernel-level security packages like LOMAC, LIDS, and SELinux. It was great to be able to build a default linux install on a virtual disk and then copy it three or four times to install the weird security stuff.

      It's even better for non-Unix OSes. A friend wanted help installing his Java web app on NT. I built a variety of virtual machines for testing, all using the VMWare "Undoable disk" choice. So when some weird registry key got screwed up by an Oracle installer, I just picked "Undo" and tried again!

      If you have to use crappy OS or packages that are inclined to break things and put crap everywhere, VMWare is a delight!

      (Yep, I'm just a happy customer.)

  • plan9's n/dump is this (Score:3, Informative)

    by DrSkwid ( 118965 ) on Sunday February 10, 2002 @04:41PM (#2983417) Journal
    the default increment is daily

    one can roll back the filesystem on a PER PROCESS basis with the yesterday command.

    In this way you can narrow down what's broken by for instance using yesterday's c library, or last week's , or last years!

    Also take a look at Venti

    From: Sean Quinlan To: 9fans Mailing list

    For those of you interested in the direction we are heading
    with respect to plan 9's file system, you might want to
    checkout our paper on Venti that will appear in the
    USENIX fast conference.

    http://www.cs.bell-labs.com/~seanq/pub.html#venti

    Venti is a block level storage server that replaces the optical
    juke box for a plan 9 file system. Some of the benefits include:
    coalescing of duplicate blocks
    compression
    no block fragmentation
    Also, we have switched from optical to magnetic disks as the storage
    technology. I know many of you already use magnetic disks to
    "fake" a worm, but for those of us using a optical juke box,
    the performance improvement is rather substantial!!

    seanq

  • Seems simplistic but... (Score:4, Insightful)

    by omega9 ( 138280 ) on Sunday February 10, 2002 @04:42PM (#2983421)
    It's a bit time consuming, but tar has been doing this job for me for a while. I finally (took me a long time, ok) created a working bootable linux CD that mounts my filesystems RO and tarballs the entire contents to a remote server. It takes about 1h20m to do a full backup.

    After it finishes, I test software on the existing system. If it breaks I restore, if it doesn't I play-test it for a while and if it keeps behaving I commit it to the next full "state preservation".

    The biggest drawback, of course, is that this sceme requires close to 1.5 hours of downtime for the backup and more if you need to restore. But for noncritical systems it works great.

  • TSM (Score:2)

    by nbvb ( 32836 )
    root@foobar# dsmc
    Tivoli Storage Manager
    Command Line Backup Client Interface - Version 4, Release 1, Level 2.0
    (C) Copyright IBM Corporation, 1990, 2000, All Rights Reserved.

    tsm> q server
    Node Name: FOOBAR
    Session established with server WAYBACK: Solaris 2.6/7
    Server Version 3, Release 7, Level 4.10
    Server date/time: 02/10/02 16:13:02 Last access: 01/30/02 00:00:56

    tsm> restore -subdir=yes -replace=yes /

    Consider it done.
  • Look at this article [microsoft.com], in the sections titled "System Restore " and "Volume Shadow Copy Service".

    Sounds like basically the same thing, no?

    - Steve

  • Backing up (Score:2, Interesting)

    by fava ( 513118 )
    What I do is use rsync [samba.org] to backup the company fileserver to a remote machine every 2 hours during business hours. Each update transfers about 10 to 20 MB over DSL/Cable. Then every evening I backup all the files that have changed over the last 24 hours into a seperate dated directory. The first dated backup consisted of the entire server (6 gb) and all the subsiquent backups are about 50 MB a day zipped.

    So I have both a snapshot of the current status of the server with 2 hours accuracy and the ability to roll back the server to any point with 24 hours accuracy.

    The best part is the company is paying for my Cable connection at home to do this.
  • Someone else mentioned this too, but can you do LVM in Linux easily? If so, set up to do snapshots before a big upgrade. A snapshot is a virtual copy of your data. When a file is added or changed only the extra info is written to a seperate place on the disk. So, you end up with two "copies" of the data but it doesn't use up a ton of disk space.

    This is great for backups too. You don't have to worry about files being open and it's a great way to get a point in time shot of your data in case of short backup windows.
    • Someone else mentioned this too, but can you do LVM in Linux easily? If so, set up to do snapshots before a big upgrade. A snapshot is a virtual copy of your data. When a file is added or changed only the extra info is written to a seperate place on the disk. So, you end up with two "copies" of the data but it doesn't use up a ton of disk space.

      That's just about the idea I was looking for; or, at least, half of it. (Thanks, Morzel, for the link to the white paper.) LVM seems to handle the "commit" half of the problem cleanly and efficiently (by just deleting the snapshot), but the "rollback" problem seems to remain. If I can't redesignate my "snapshot" as the new "mainline," then I don't have a painless way to do a "rollback."

      Others have mentioned using rdist or tar, which are not unreasonable ideas. But this is my home server, with more space than anything else I've got; so those solutions just require me to have another server, both bigger and more stable, and to which the same questions would ultimately apply. So, my ideal solution would be the all-in-one-box answer, if it can be done.

  • snapshotting.. (Score:2, Informative)

    by Anonymous Coward
    Snapshotting is what you really want for something like this. NetApp [netapp.com] has had this functionality available in their Filer appliances for a number of years - you can cd into a 'magic' .snapshot directory where hourly, daily, weekly, and monthly snapshots are kept.

    FreeBSD [freebsd.org] 5.0-CURRENT includes preliminary snapshot [mckusick.com] support for ffs.

    The Linux options aren't quite as good. The most promising new filesystem that could provide this functionality is tux2 [linux.org], where data is structured in a way that would make implementing this functionality fairly easy. There was a post explaining how it would work in the mail archives, but they seem to have disappeared.

    There is commercial option: MVD Snap [mountainviewdata.com]. Their fileserver is Linux based, and the code for their snapfs filesystem was once available during beta testing.

    • Re:snapshotting.. (Score:3, Informative)

      by Guy Harris ( 3803 )
      NetApp [netapp.com] has had this functionality available in their Filer appliances for a number of years - you can cd into a 'magic' .snapshot directory where hourly, daily, weekly, and monthly snapshots are kept.

      In fact, we've had that since we first shipped our machines. There's a paper on our Web site that discuss how this works, File System Design for an NFS File Server Appliance [netapp.com].

      However, although snapshot directories let you dredge up copies of files from snapshots in case you (or a program) screws up and trashes them, that's not a convenient way to roll back the state of the entire file system.

      We did implement that later (atop the same mechanism); see SnapMirror and SnapRestore: Advances in Snapshot Technology [netapp.com] - SnapRestore(TM)(R)(LSMFT) is the "roll back an entire file system to a snapshot" feature. (At times, all this SnapStuff makes me want to SnapTheNeckOfMarketing, but so it goes....) That paper doesn't discuss technical details to the extent that the other paper does, but it should be possible from the earlier paper to figure out at least some of how you'd do it.

  • Just use Linux LVM (Score:3, Interesting)

    by morzel ( 62033 ) on Sunday February 10, 2002 @06:42PM (#2983912)
    You need to get LVM [sistina.com] compiled in your kernel.
    LVM provides the ability to create snapshots of your volumes, so you can easily roll back if anything icky happens. Mind you that write perfomance when using the snapshot feature goes down: instead of one write operation, every write becomes a read/write/write operation slowing things down. And this happens for every active snapshot, so you really can't have too much active snapshots :-)

    Then again, if it's just for checkpointing (create snapshot), installing experimental stuff and then committing (delete snapshot) or rollback (restore from snapshot, delete snapshot), it should do the trick wonderfully.

  • AFS (Score:4, Informative)

    by William Aoki ( 392 ) on Monday February 11, 2002 @03:37AM (#2985617) Homepage
    AFS will do something like that, although not to the extent that I hear NetApp Filers can. Off the top of my head, there are two ways to do this with AFS. Both these methods require superuser access to your AFS cell, unless backups or replication releases are being done automatically.

    (CodaFS should be able to do this too. I haven't played with CodaFS enough to know if it offers any other way to accomplish checkpointing.)

    Method 1: backup volumes

    $ cd /afs/mycell/some/path
    $ kinit me/admin
    Password for me/admin@MYCELL:
    $ aklog
    $ vos backup some.path.avol
    $ kinit me
    Password for me@MYCELL:
    $ aklog
    $ cd avol

    do stuff with the filesystem...
    Oops! I need files that I modified or deleted!

    $ cd ..
    $ fs mkm avol.backup some.path.avol.backup
    $ cp avol.backup/little-lost-file avol/
    $ fs rmm avol.backup

    Many sites run 'vos backupsys' (generally before 'vos dump'ing volumes) every night to automatically back up all their volumes, and leave users' backup home volumes mounted under their home volumes, to provide easy access to yesterday's files without an administrator's help.

    Method 2: for replicated volumes

    $ cd /afs/.mycell/some/volume

    do stuff - uh-oh, I need a file back that I changed!

    $ cp /afs/mycell/some/volume/my/file my/file

    ok, finished with the changes. Commit them!

    $ kinit me/admin
    Password for me/admin@MYCELL:
    $ aklog
    $ vos release some.volume
    Released volume some.volume successfully
    $ kinit me
    Password for me@MYCELL:
    $ aklog

    Volume (for volume, read filesystem) backups work by saving the state of a volume at the time the backup command was issued. When changes are made to the volume, the original state is copied to the backup volume. The backup volume only takes as much space as the changes made since the last backup. Replication works by making read-only copies of a volume in one or more locations, as specified by 'vos addsite' commands. The copies are only updated when changes are 'released' from the read-write copy to the read-only copies. By convention, cell root volumes are mounted read-only on /afs/cellname and read-write on /afs/.cellname.

    I think that newer versions of Solaris will do checkpointing on UFS. I haven't adminned Solaris since 2.3 (the slooow SS20 with 2.8 under my bed dosen't count until I play with it some more), so I'm not familiar with the details.
  • It was called SnapFS. You can find traces of this in a LWN article [lwn.net]. I don't know if the projects still exists, go ahead and google .....
    • That is exactly the concept I'm looking for! Unfortunately it seems to be quite dead. Here's what I found with Google.
      • A sourceforge project [sourceforge.net] with no files uploaded, no description, and a few messages asking, "What happened?"
      • A glossary [kernelnewbies.org] which describes SnapFS as "A defunct experimental filesystem". There's also a broken link to linuxcare.com.au/projects/snapfs .
      • A diary entry [advogato.org] from somebody named Martin Pool. On 18 August 2000 he writes,
        I wrote up some documentation for my
        snapfs filesystem [linuxcare.com.au], which has been shelved for some months now. I get mail every so often asking what happened to it, so it seemed good to answer it once and for all. Perhaps I'll get back to it eventually.

      • And of course, Mountain View Data [mountainviewdata.com] (or the equivalent snapfs.org [snapfs.org]), which now owns the project and seems to have renamed it "MVD Snap". But there is very little information here, and nothing to download.
      Is it legitimate for SnapFS to go missing like this? The link you gave says that SnapFS is under the GPL. So doesn't it have to be available somewhere in a free and downloadable form?

      Thanks to everyone for all the help!

      • A little more SnapFS info...

        The Martin Pool/linuxcare.com.au project seems to be unrelated to the SnapFS in the original announcement, now owned by Mountain View Data. There's a README on the Pool project here [ultraviolet.org]. It does seem to be dead, which is OK since it didn't offer the full filesystem rollback capability. In addition, it was a true filesystem, unlike the other, which was designed to sit in another layer above Ext3.

        If anyone has an old tarball of the now-MountainView SnapFS, please let me know...

Slashdot Top Deals

For God's sake, stop researching for a while and begin to think!

Close