Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Linux Software

Reliability of Journalling Filesystems Under Linux? 66

Posted by Cliff from the checking-the-accuracy-of-your-journal dept.
chrysrobyn asks: "Every write-up I see about journalling file systems under Linux discusses efficiency (embedded) or speed (desktop/server). Have any studies been done on reliability? I've used Linux since Slackware 96 (and kernel 2.0.0), and put it on 9 or 10 machines over the years (Slackware on x86 and Debian on PPC), but I've never strayed from ext2. Always, when the uptime gets high, 20-50 days, the filesystems start to get minor fsck errors. Not that I repair the system and expect it to stay live, I just use the fsck -n to help me decide when a repair is in order. Since the same thing has happened on a variety of hardware (386-PII and every interface in between and 601 and 750 processors with Apple hardware), I'm leaning on blaming the ext2 filesystem for these, the slightest of problems. I typically keep my servers up for as long as possible because 95% of my hardware problems have happened during resets and cold power-ups. It's time for my every-other-year rebuild of my personal server, with another on its way, so I was hoping to incite some anecdotal Slashdot conversation on the journalling file systems available for Linux. Personally, I'm most interested in hearing about the file systems supported under Debian stable for ease of administration for this machine which is a 5 hour drive away from home. I've been around the block a few times, so I'm not fearful of patching the kernel with better patches, but I'm respectful of the work the Debian assurance teams have done."
This discussion has been archived. No new comments can be posted.

Reliability of Journalling Filesystems Under Linux? More

Reliability of Journalling Filesystems Under Linux?

Comments Filter:

  • If you are fscking a live filesystem (Score:4, Informative)

    by Pathwalker ( 103 ) <hotgrits@yourpants.net> on Friday November 15, 2002 @05:23AM (#4675713) Homepage Journal
    you have to expect some errors to show up from time to time, because the filesystem may change while fsck is running, and if so it will not be internally consistant.
    • If you read carefully he uses 'fsck -n' to *check* for errors without fixing it. Doing this regularly will show that even a perfectly fine running system will pollute an ext2 fs.

      I suspect that he is sane enough to init S before doing a *REAL* fsck.
      • Doing this regularly will show that even a perfectly fine running system will pollute an ext2 fs.
        Bollocks. All you'll get on a live filesystem is a series of false positives as the filesystem changes behind fsck's back. You won't trash it, but your results won't be reliable...

  • Few links for you (Score:4, Informative)

    by Rubbersoul ( 199583 ) on Friday November 15, 2002 @05:26AM (#4675717)
    Not knowing the answer to this myself I present to you a few links that may be helpful. Hope this helps.
    [bulmalug.net]
    This link has some good benchmarks of Ext2, ReiserFS and XFS.

    And here is a fairly good news group discussion [google.com] relating to what you are talking about.

    • Yes, I looked at a lot of benchmark studies a few months ago when I was overhauling my systems.

      In the end i figured that I would stay with ext2 for the time being.

      I'm not simply being reactionary here. Being an old-timer (since late '70s) as a sysadmin and sysprog, I have always placed a high value on good backups carried out rigorously and systematically, e.g. grandfather/father/son in daily, weekly, monthly etc cycles as required by the data turnover, even for desktop boxen.

      I've seen all too many people (often very technically savvy) lose important stuff by ignoring backups and trusting journalling systems.

      With my setup (on the basis of the benchmarks I read) ext2 outperforms any of the journalling systems, and I'll live with the risk of an occasional fsck on bootup.

  • /something/ is wrong here. (Score:5, Informative)

    by Eivind ( 15695 ) <eivindorama@gmail.com> on Friday November 15, 2002 @05:32AM (#4675721) Homepage
    ext2 is a quite stable fs. It is not journaling, so crashing at an inconvenient time can lead to an inconsistent fs, but other than that there is no reason why an ext2 fs should magically develop inconsistencies after 3-6 weeks of runtime.

    You should be aware that if you are running fsck -n on the fs while it is mounted in rw-mode, then it can and will report inconsistencies which are not real, simply because the fs has changed between the passes in fsck, something which it does not expect.

    For this reason, I suggest you try again with remounting the fs in ro-mode before running fsck -n. I am fairly sure you will find that your errors go away. Especially since you state this has happened on diverse hardware and presumably diverse kernels.

    That said I would recommend going with a journaling fs for that extra safety that comes from never getting inconsistent even if the power goes out at the worst moment. ext3 and reiserfs are both good, my preference would be ext3 for the simple reason that it can be mounted as an ext2-fs, which means that you will be able to read it with any old rescue-disk or whatever. Reiserfs typically requires you to redo all your rescue-disks, and make sure that your backup-restore-scheme handles it rigth.

    If the remounting in ro-mode does *not* make the reported errors in fsck -n go away, and you are somehow able to reproduce this, please report the bug to linux-kernel.

    • /something/ is wrong here.

      I think I may be able to point to the answer.

      You should be aware that if you are running fsck -n on the fs while it is mounted in rw-mode, then it can and will report inconsistencies which are not real, simply because the fs has changed between the passes in fsck, something which it does not expect.

      I'm doing my fsck -n's in RW mode. From the less file system experienced Linux user's perspective, I wonder what ext2 does when going from RW to RO that cleans up for fsck. I can understand the value of delaying some writes, but shouldn't that get flushed when the box is not active? Would fsck -n work on a RW mounted ReiserFS, JFS, XFS or ext3?

      I'm not being argumentative, this sounds like one of those typical Unix behaviors, but learning why may help me with other potential issues.

      • Fsck reports inconsistencies when checking a RW file system, because from fsck's perspective, it IS inconsistent. Things have changed that fsck didn't see, and it (correctly) reports it. Some file systems are designed to allow fsck to work on a RW filesystem (FreeBSD soft-writes, to a limited extent).

        Remounting RO "cleans up" for fsck, by flushing all pending writes to disk, and insuring that no new writes happen during fsck's run. Even on an 'idle' system, writes to the file system can and do occur (to update meta-data, if nothing else). If you REALLY want to run fsck on a RW file-system, you should at least use the "noatime" mount flag, and possibly "sync", and "dirsync" as well.

        You can also use LVM, and create "snapshots" of the live filesystem, which can then be checked for consistency offline, while the original filesystem chugs along.

        I agree that it is highly unlikely that the fsck errors you see are anything more than anomalies caused by your method of checking (unless you have some funky power source or radiation that is affecting a variety of systems). ext2 is considered to be quite stable (ie. wont become inconsistent over time on working hardware). Are these SCSI disks or IDE? I'd suspect the disk driver code before the filesystem code.
      • I'm doing my fsck -n's in RW mode. From the less file system experienced Linux user's perspective, I wonder what ext2 does when going from RW to RO that cleans up for fsck. I can understand the value of delaying some writes, but shouldn't that get flushed when the box is not active? Would fsck -n work on a RW mounted ReiserFS, JFS, XFS or ext3?

        Well, the reason whz zou get inconsistencies when doing fsck -n in rw mode is actually quite simple. Remember that fsck checks that the filesystem is consistent, this means for example: reference-counts are correct, directories all have a ".." and an "." entry, no block is both marked as being part of a file, and on the list of free blocks and so on.

        Many fs-operations are not atomic. For example, to remove a directory the kernel will first remove everything it contains (including "." and "..") and then remove the directory itself. Now, what happens if fsck happens to touch the directory *between* those two steps ? That is, after "." and ".." have been removed, but before the dir itself is gone ? You'll get an inconsistency.

        This is just one silly example, it's easy to think of others. You are rigth that the problems should be smaller if the filesystem has been inactive for a while so that most fs-operations are finished, but a total guarantee that everything is finished you don't get. Possibly running "sync" would be enough though, you migth try that.

        remounting in ro-mode helps because it ensures that all write-operation fully completes, and also stops the kernel from making any changes to the fs in the middle of your fsck-run.


    • > ext2 is a quite stable fs ... there is no reason why an ext2 fs should magically develop inconsistencies after 3-6 weeks of runtime.

      Agreed. My typical login time is 180 days, and I almost always have > 100 processes alive on the system for that whole time, but the only time I've had disk problems in several years of operating that way is when I had hardware failures.

    • >That said I would recommend going with a journaling fs for that extra safety that comes from never getting inconsistent even if the power goes out at the worst moment. ext3 and reiserfs are both good,

      agree I put reiserfs on a 'puter for my wife's use, she was a total computer newbe for safety. she'd some times just punch the power button out of frustration but never a problem with the file system. I couldn't detect any seat-of-the-pants difference in speed.

  • I've user ReiserFS since it first appeared in Mandrakes distro. I have never had any problems what so ever with it. It just keeps running.

  • old hardware (Score:2, Informative)

    by wotevah ( 620758 )
    I had a couple of these systems, they would develop filesystem errors out of nowhere. It always turned out to be faulty old hardware (memory, cables, motherboard etc). The PC components get old really fast, my plan has been to get new hardware at least every three years and get real server hardware (ServerWorks mobo etc).

    For the last series, I have not noticed any unexpected filesystem errors after 200-300 days of uptime (they need to be rebooted from time to time for kernel upgrades).

    To conclude, always suspect your hardware first, especially if it's at least a couple years old.


    • This is correct. Actually, however, suspect that your hardware has developed a bad connection first. Many problems are corrected by pulling every adapter and cable out about 1 millimeter, then pushing it in again. That wipes the contacts clean of oxide.

  • Ext3 vs ReiserFS (Score:5, Informative)

    by DarkDust ( 239124 ) <marc@darkdust.net> on Friday November 15, 2002 @07:13AM (#4675945) Homepage
    I don't know about XFS and other journalling fs's since I've only used ReiserFS and Ext3 so far.

    My experience so far is that Ext3 is more reliable (read: repairable) than ReiserFS simply due to the fact that Ext3 is a kind of "extension" to Ext2, so you can just run the good old well tested and known to work fsck.ext2 on a Ext3 partition should it screw up.

    But I have yet to see a Ext3 partition screwing up, I've set up several PCs and servers with Ext3 and it works fine, no single problem to date.

    Unlinke ReiserFS. I have to admit, my only experiences with ReiserFS were about one and a half years ago or so, but at that time I had set up a home PC with ReiserFS and somehow I f***ed it up beyond all repair. I don't remember what I did then but I just got scared of ReiserFS :-)

    On the other hand I have still another home PC, running SuSE Linux 7.2 updated to 7.3 with ReiserFS which just runs fine, and this is my home server, running 24-7.

    So I guess until you don't do anything stupid like I did both ReiserFS and Ext3 are pretty reliable today, given their widespread use you would probably have heard of any major glitches/problems ;-) The decision whether to use on or another is more performance/religion-wise, IMHO :-)

    • Re:Ext3 vs ReiserFS (Score:5, Informative)

      by GigsVT ( 208848 ) on Friday November 15, 2002 @09:25AM (#4676237) Journal
      Well I can fill in your gaps on XFS. XFS has run flawlessly for me for over a year now, on a 1.9TB RAID volume. EXT2/3 seems to have rough edges in regard to large file systems, for example, by default mkfs will set aside 5% of your disk for root use. That's 97GB wasted! Another problem is that you need to specify -T largefile4 or it will try to create way too many inodes, taking forever to create the filesystem, or fsck if you ever need to.

      XFS is a very mature file system, and file systems that are many TB work fine with its defaults. Performs more consistantly too. EXT2/3 was very sensitive to RAID stripe size, and things like that. Even setting the special stride option, you had to recreate the filesystem many times to make sure things worked right. XFS performed consistantly at any stripe size, with no strange dips in performance if boundaries didn't line up just right.

      In all, if you are building a large RAID, I would go with XFS. For day-to-day use of 200GB or less on a single disk, EXT2/3 is fine. (You probably still don't want to let it waste 5% of the disk, that is such a retarded default, use -m 1 to help reduce it)
      • Good to know, thank you... I've never set up something beyond the 200GB scale, not even RAID servers so I didn't run into any of your mentioned problems yet... except for the 5% root-reservation, but normally I just don't care about that ;-)

      • Re:Ext3 vs ReiserFS (Score:3, Insightful)

        by joshki ( 152061 )
        My personal experience with XFS has been horrible -- I've had several partitions completely trashed.

        Quoting from the Gentoo x86 install guide:

        "Please be careful with XFS; this filesystem has a tendency to fry lots of data if the system crashes or you lose power. Originally, it seemed like a promising filesystem but it now appears that this tendency to lose data is a major achilles' heel."

        I would not recommend XFS until some major work has been done.

        • I second his motion. Twice now I have been burnt by Reiser. Granted the first time can be very easily attributed to the infancy of the FS itself, but the last time (about a year ago) was inexcusable. Ext on the other hand, has been very nice to work woth, once in a blue moon repair, and upgrade. I was able to migrate my Ext2 partition to Ext3 VERY easily. I didnt have to worry about the integrity of the data. Considering that I would literally have to WORK at not finding a tool to assist me in a crisis with Ext -- I feel at this point its a no brainer. Now, I would like to ask this one qwuestion however : RedHat pumped an assload of time and money into ext3. Has anyone done the same for Reiser? I realize that Mandrake, Slackware and SuSE have included it as an FS option for quite some time (whereas RedHat has not).
    • I once did dd if=disk.img of=/dev/md0 instead of /dev/fd0 (only 1 letter difference). It took about 4 days of reading docs before I even could begin any recovery.

      fucking up your filesystem is easy, and you generally have yourself to blame (in my experience).

    • Re:Ext3 vs ReiserFS (Score:3, Interesting)

      by JLester ( 9518 )
      The more recent ReiserFS is much more stable. We run it on a RAID5 system that serves e-mail for about 7000 users. We use Maildir format mailboxes and ReiserFS is supposed to be really fast for small files. We converted from a dual-550 to a dual-1.26 system and went from ext2 to ReiserFS and noticed a huge speed increase in opening large mailboxes.

      Jason

  • High uptime? (Score:2, Insightful)

    by mnordstr ( 472213 )
    "when the uptime gets high, 20-50 days"

    That's not high uptime! Maybe if you're running Windows 95. I've had my system running for a little over 320 days now, and I haven't experienced any problems on any of my ext3 drives. And I've never before experienced any problems, on ext3/2 HDs. If you want reliability, I think the best thing you can do is buy a UPS. That makes it much more reliable than any FS change can do.

    • Re:High uptime? (Score:1, Funny)

      by Anonymous Coward

      When you say "320 days", do you mean "320 days since the last reboot", or "320 days since the last crash, ignoring normal maintenance restarts (e.g. for security patches)"? If the former, what is the host's IP address? I need another zombie for my DDoS network.







      (For the sarcasm impaired, I'm just kidding about that DDoS thing. My zombie network has nothing to do with denial of service attacks. Instead, I will use it to take over the world! Mwhahahahaha! Phear my elite haxor skillz, luddite-mortals!)

    • I used to get 20-50 days uptime regularly with Win98SE, back when I let it run 24/7. That was before I started downloading patches and such for it, though, which seems to have made it much less stable. I wonder if they do that on purpose, so people will be encouraged to upgrade for increased stability?

  • It's anecdotal evidence only, but I swear by Reiser these days. My Dell Inspiron 7500 has developed a problem that causes it to hang on return ing from suspend a couple of times a week and I haven't lost data once. At this point I've probably had upwards of one hundred hangs and crashes because of the flakey hardware and Reiserfs has saved my data every time. It can't do anything to protect data that hasn't hit disk yet, but once it's there it's pretty good about keeping it intact. After living through crashes that left Ext2 filesystems chiseled spam it's a good feeling to be blase about such things.

  • No problems with Reiser (Score:5, Informative)

    by inkfox ( 580440 ) on Friday November 15, 2002 @08:33AM (#4676093) Homepage
    I'm afraid you won't get much more than anecdotal evidence on this question. Here's mine.

    We had a bad network adapter which would fail when other DMA devices were busy. This meant that whenever disk I/O was heavy, using the network adapter was likely to cause a complete system lockup. This took a while to diagnose as the problems took upward of two weeks to reproduce.

    Despite the equivalent of having the power cable yanked randomly a dozen times when the machine was at its busiest, we never had a single problem with Reiser. The file which was being written to existed as the old version, and there wasn't even a lengthy fsck. Integrity was 100%.

    • I'm having a similiar problem. What kind of symptoms did you see? I get a blank screen, and the machine requires a power cycle.

      What kind of NIC was it?

      How did you diagnose? Just start some massive dd's and ftp's?

      I was getting ready to buy a new motherboard, because I thought the IDE chipset was going bad (had a power supply overheat, so it's possible but unlikely.) The NIC is much more likely.

      Any info is appreciated!
      • RealTek 8139.

        The RealTek DMA system is a joke. Tiny failures aren't tolerated, nor do the two popular drivers handle buffer overfills -- a bad network cable is enough to destabilize a system.

        Still, it was a hard lock, no screen blanking. (Or are you sure it hasn't locked up while the console screen blanker was active?)

        Grabbing a DEC Tulip-based card will increase performance and stability both, and you can get one for $5-15 if you have a decent shop near you.

        • Still, it was a hard lock, no screen blanking. (Or are you sure it hasn't locked up while the console screen blanker was active?)

          Most of the lockups are while the blanker is active (the machine sits in a closet.) The one time I saw it lockup, it hung leaving the boot up fsck progess indicator on the screen. No errors were displayed.

          The machine has 2 different 3Com cards. But as I think back, I did see the machine lock up with both cards out. That leaves the Video card, motherboard, RAM or CPU. Guess I swap out the RAM, since that's cheap-n-easy.

  • Do not trust your fs (Score:5, Insightful)

    by jsse ( 254124 ) on Friday November 15, 2002 @08:34AM (#4676098) Homepage Journal
    trust your backup.

    Two cents from an old admin.
  • You say that when you fsck your ext2 filesystems, you get errors. I'm assuming that you're only running fsck on dismounted partitions here.

    This indicates, to me, some hardware flakiness on your end. (Even though you say this happens on a wide variety of hardware.) In every account I've seen, journaling filesystems are more stressful on the hardware because - surprise! - the journal is constantly being written to. I'd stick to ext2 if I were you, and figure out why you get any errors when you fsck a dismounted file system.

    I'm in charge of roughly forty Linux boxes, including many desktops and many servers. I've never seen any problems that I could blame on the filesystem. (Though there have been kernel releases in the past - including one in the 2.4.x series, IIRC - where there was a bad filesystem bug, fixed within a day.)

  • I have been using ext3 since around 2.4.13...and it has not given me one problem yet and has in fact saved my ass serveral times. I have frequent power failures at my house, but ext3 recovers gracefully every single time....

  • ext3 on woody (Score:1, Informative)

    by Anonymous Coward
    I've been running Debian3.0 with ext3 on a few machines here. None of them have extremely high loads, but I haven't had any problems. For me it seems the biggest advantage of ext3 is that if something happens you can always use something without ext3 support and mount it as ext2.

  • I have been paranoid about moving away from ext2 in production environment myself, so I have been trying out ext3 and reiserfs on personal boxes for the last eigth months. No real problem so far (once I still had a fs error on reiserfs after replaying the log, a fsck fixed that). These are no hard hit or database servers, but I tend to copy a lot of files and data (several gigs) simultaneously between reiserfs file systems and mix up my cd-rom eject button with the reset button from time to time, and on the ext3 box I tried hacking some usb driver which made it crash every ten minutes over three weeks...

    Being concerned for reliability, you might look into the ext3 options to have ordered writes etc., or otherwise your meta data might be fine, although your actual file data might get screwed... The kernel mailing list should hold hints on stability (the more absent complaints are, the more stable it is ;), and as said before, in the end you have your backups... Don't you?

    I still wonder about meta journaling on database servers with huge data files, and the next thing I am paranoid about is using LVM on Linux :)


  • I've only tried reiserfs and xfs for a few days each, for the most part I've stuck to ext3 in recent days. I've hard-crashed (pull the plug type of thing) several different machines with ext3 while filesystem write activity was going on and never had a problem. Based on my time with ext3, my limited experience with reiserfs/xfs, and reading lots of lkml, I think ext3 is the safest choice at this point in time, even if it's not neccesarily the best performance.

  • ext3 is simple to install or uninstall. I promise. (Score:3, Informative)

    by Cecil ( 37810 ) on Friday November 15, 2002 @12:20PM (#4677486) Homepage
    Aside from everybody telling you "that shouldn't happen, you're doing something wrong", which is probably true, I just wanted to chime in with my support of ext3. I think you're making a mountain out of a molehill.

    You obviously haven't looked very closely into ext3, because it's an extremely simple layer on top of a standard ext2 filesystem. Essentially, all it is, is an extra file in /, a daemon to do journalling, and a bit or two toggled on the disk itself.

    the FAQ [sapienti-sat.org] has one question that lists the two steps required to install a journal on a stock ext2 filesystem (provided you've got a 2.4.16+ kernel, or have patched your older kernel).

    Not only is it very simple to install, but it's very simple to uninstall too. Blindingly easy, in fact. Mount your filesystem as ext2. Done. No journal. If you want to do it permanently, there's an answer about that in the FAQ too.

    So really, you have nothing to lose by trying ext3. I've had 0 problems with it, and I use it on a laptop that gets a lot of abuse WRT being turned off at random times (I can't view my battery level in Linux, but I can in Windows. Thanks broken ACPI BIOS...)

    The only downside is that the filesystem will sync every 5 seconds or so, which completely destroys any possibility of ever letting the disks spin down for power saving, but that's more of a laptop issue than a server issue.
    • The only downside is that the filesystem will sync every 5 seconds or so, which completely destroys any possibility of ever letting the disks spin down for power saving, but that's more of a laptop issue than a server issue.

      If you mount it with the noatime option, then it won't constantly rewrite the last access time for files; this means that your disks can spin down again. It's worked for me.
  • I've been using reiserfs exclusively since SuSE 7.1, and it's been great. I haven't had a single problem, even during power outages and such (no, I don't have a UPS).

    That's about as annecdotal as it gets!

    Anyway, I'm not going to recomend reiser over the others since I don't have any experience with them, but I will say that I've developed great confidence in reiser's reliability. If I had any old data that I really cared about and wanted to use the same drive, though, I would probably go with Ext3 for the non-destructive (or so I've heard) upgrade.

  • I know this is not common, but I had a bad experience with ext3 a few months back that resulted in the first ever time I have had a complete catastrophic data loss without hardware failure.

    I'll start with the observed events (the stuff I know that happened):

    1. System crash. Probably one of those that happens when my laptop's flaky power management hardware picks a fight with the kernel
    2. First reboot. fsck runs and claims to be fixing some broken stuff. The box doesn't make it through the init scripts before hitting a busted program
    3. Second reboot. fsck runs and claims to be fixing some broken stuff. The box makes it less far through the init scripts before hitting another (newly) busted program
    4. Third reboot. Kernel cannot find init
    5. Boot from floppy and manually run fsck on the disk. (In retrospect, this was probably not the best of choices.)
    6. Fourth reboot. lilo cannot find the kernel. (Or more appropriately, the kernel cannot make it through its own init.)
    7. Rescue disk, take two. No surviving superblocks to be found on the disk.

    The following is my hypothesis as to what was happening at the low level to cause this series of problems. (Note, however, that my knowledge of the ext3 internals is sketchy, so the following is probably somewhere between slightly mistaken and out to pasture in left field.)

    1. System crashes. Data corruption happens. Probably both in the journal file and the superblocks/inodes referring to the journal. Possibly, the magic ext3 bit indicating an active journal on disk is not set properly.
    2. Aparently, fsck.ext2 was running, trying to fix broken file structure, but ended up making changes to the journal itself.
    3. Some brilliant (buggy) piece of software on my poor machine decides to replay the "fixed" journal. Since this is no longer the carefully constructed entry we expect (see #2), this is effectively spitting random information around my hard drive.
    4. I observe still busted machine and think that I should give the restore process another chance. I reboot, effectively returning to step 2.
    5. There is no step 5. The partition has effectively ceased to exist.

    6. Note that I cannot confirm that steps 2 and 3 took place in that order. My guess is that if the journal damage was bad enough, step 2 would not even be necessary (though it was clearly happening). I can only assume that the progressive nature of the failure resulted from data from journal replays between each step #2

    My guess is that some buggy/un-updated version of some piece of software was likely to blame. (I was quite a bit behind on my updates.) However, my hypothesis leads me to believe that storing the journal as a file on the journaled filesystem itself is a bad design decision that probably contributed to the extent of data loss on my system.

  • Hi,

    I've been using farious Linux FS on multiple production servers, ext2/reiserfs, no problems at all. Today, I'm using ext3, quota support is better and you still can use chattr with ext3;)

    Anyway, you should run RAID1 or 5, if possible with a hw RAID controller, hotswapable disks and perform regular backups, if you are serious about your data.

    If anyone has some info, why ext3 is in most benchmarks faster then ext2, I would like to hear.

Slashdot Top Deals

E = MC ** 2 +- 3db

Close