Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Operating Systems Software Linux

Logging Unexpected Shutdowns/Crashes w/ Linux? 86

Posted by Cliff from the finding-evidence-of-the-problem dept.
sweede asks: "I have a dedicated server that seems to reboot more often than it should. In Windows 2000/XP (maybe NT4.0?), if your computer or server crashes it will leave an event message in the Event Viewer for you to review on what went wrong. Is it possible to do something similar in Linux? Where a power outage or an unexpected kernel panic will leave a message in /var/log/event (or whatever) Searching Google for 'kernel trapping' doesn't give me a whole lot of info on the subject."
This discussion has been archived. No new comments can be posted.

Logging Unexpected Shutdowns/Crashes w/ Linux? More

Logging Unexpected Shutdowns/Crashes w/ Linux?

Comments Filter:

  • I'm pretty sure.. (Score:4, Informative)

    by Creepy Crawler ( 680178 ) on Monday September 15, 2003 @12:11AM (#6961211)
    That the reason Linux doenst write anything to the HD after Panic si so that it doesnt mangle/destroy the FS.

    And if I'm correct, if you turn on serial console, you'll get a Panic output on serial. Add a serial2IP box and you're set.

    • Re:I'm pretty sure.. (Score:4, Interesting)

      by Chester K ( 145560 ) on Monday September 15, 2003 @12:40AM (#6961364) Homepage
      That the reason Linux doenst write anything to the HD after Panic si so that it doesnt mangle/destroy the FS.

      Why not reserve a set place on the hard drive and write out error trap information there? There's no reason the filesystem needs to be involved at all. I'm going to guess that's what Windows does.

      • Re:I'm pretty sure.. (Score:4, Interesting)

        by Creepy Crawler ( 680178 ) on Monday September 15, 2003 @12:48AM (#6961395)
        OK. Then how do you guarantee the state of the kernel? If you use bios calls, it screws up the memmap even more. Thats assuming you can even pass something like that.

        100$ question: How do you break out of code inserted that might have had a bug? How do you determine what code had that bug?

        Answer those, and then I'll trust Write_after_system_crash api
        • Why don't you just write to the swapfile? If your on the way to rebooting, nothing gets hurt by munging up the swapfile some. Of course, if your harddrive drivers crash, it's no good, but for all other cases, it sounds like a good idea.
          • Why do you trust a kernel that has got its knickers in a twist to be able to know where the swap partition is?

            I'd be happier with it writing to a floppy, serial, or other isolated subsystem. The difference between your swap partition and your root directory structure might be just 0x10000 in one of the register values, and that's considered too close to be worth risking.

            YAW.
        • OK, I've got an idea. When it panics, it reruns the bootloader (use BIOS calls to read the first HD sector and go from there) and passes it some special flags which basically say "I did a bad thing, clean up after me." The bootloader will unpack another set of routines (checksummed for quality) in the same way it loads the Linux kernel off of the HD, and place them into an area of RAM that's hopefully not used by anything kernel related (app space). It will then read in the pagetables and other info still r

          • Re:I'm pretty sure.. (Score:2, Interesting)

            by Tux2000 ( 523259 )

            When it panics, it reruns the bootloader (use BIOS calls to read the first HD sector and go from there) [...]

            When Linux panics, it usually has a good reason to do so. Something like a damaged descriptor table, overwritten kernel code, hardware that works wrong, and various other catastrophes. Panic means a real panic: You can not reliably use any hardware. So you can not rerun the bootloader, and you can not access the BIOS. You can only hope that a hardware watchdog card notices that the kernel has pani

            • Yeah, it would have to drop back into real mode. The assumption is, though, that there will be enough of the machine (and kernel) left to reboot and write a panic log. If not? Oh well, the machine crashes again. Not like it'll hurt anything (checksumming the crash dump code once when its loaded and again before it dumps to disk will ensure that no bits have been flipped, and sanity checking both on the inputs and outputs should take care of any weirdness not caught by that). Most of the kernel panics I've e
        • Obviously it can be done, since pretty much every Unix and clone except Linux already does it. And nobody says you have to trust being able to make crash dumps. Why wouldn't it be a tunable setting? If you don't like it, turn it off. Fact is, some people would find this sort of feature really useful, so why deprive them of it when adding it won't affect you at all?

          I suppose I shouldn't be surprised. Linux doesn't implement a feature, ergo that feature is not worth implementing. I'm just glad the peo

          • Good point. I've been a Solaris admin since God was a corporal, and the crashdump in the swap slice has save our corporate asses about a zillion times. (Damned UltraSparc II level-0 cache memory bit error...)

            I guess I didn't realize that the Linux kernel doesn't do a comparable function. I wonder why not? Solaris seems comfortable locating the swap slice and writing the kernel core dump. Is partition management so much less reliable on x86 boxes? And if so, why is that a limitation for non-x86 Linux kerne

      • Re:I'm pretty sure.. (Score:2, Informative)

        by Anonymous Coward
        Why not reserve a set place on the hard drive and write out error trap information there?

        The IDE/SCSI driver could still corrupt your data. How would you know where to write the info anyway? The kernel could easily calculate and store a block number, but could it trust the stored number after panic() is called? Maybe somebody overwrote that variable, or maybe some bad RAM caused it to spontaneously change.

        Kernel panics are generally used as a last resort, when something goes really wrong and there's no s
      • I've done some of the larger systems like VMS and they would crash leaving at least a minimal dump of the exec event message queues, to a full dump of the system state. The area used for system dumps is pre-allocated and set aside during startup so the exec can locate the file directly by block number without using the file system (or even the regular drivers).

        Part of the system startup would scan the dump for the logging buffers, extract the messages and append them to the log file. The file system would

      • Why not reserve a set place on the hard drive and write out error trap information there?


        Funny suggestion. That's exactly what solaris does. It writes its kernel core dumps to the swap partition.
      • That would be too elegant and convenient. Linux/Unix doesn't do either of those because they're for morons who like to point and click at stuff like the malformed children they are.
    • Or a dot-matrix printer. Seriously, I did this for a while; you can turn on console-on-LPT support in your kernel config, and pass a parameter with your bootloader. It takes a while for the stupid thing to display all the kernel messages at boot, but the sound is priceless. Sadly, the 20-some-year-old printer decided to kick the bucket (still lasted longer than my HP DeskJet, thank you very much), so I switched to a 286 laptop running Minix 1.5 and term, which might be a cheap way to implement the serial
      • Too true. I knew about those options too, but if he wants logging, he can make a cheap logserver that archives these problems across the whole network.
        If it's only 1 computer, I'd probably use a real terminal or a cheapie like your minix box or printer.
    • What's the reason that just about every other unix does write to the HD after panic?
    • This is why things I like to call "more thought-out" operating systems (Solaris, AIX, IRIX, etc) allow for a separate raw partition simply to write out system dumps for later analysis, safe from the dangers of filesystem corruption. You don't know how useful these tools are until you've had to use them.

      - A.P.

    • This doesn't seem to be a difficult problem. I may have read this somewhere but no idea where.

      Mark the filesystem as 'not cleanly shutdown' when the OS boots. If it shuts down properly, mark it 'clean shutdown'. The only remaining option, an improper shutdown, will leave the flag in the original state - 'not cleanly shutdown'.

      Only thing to watch out for is a proper shutdown not updating the flag correctly, which should be rare - if you get to the 50th item on a shutdown list ("update flag"), you're pre

  • Easy... (Score:4, Informative)

    by icemax ( 565022 ) <matthew_d_stone.hotmail@com> on Monday September 15, 2003 @12:15AM (#6961236) Homepage
    'last reboot' should show you all the recent boots

  • Logging Unexpected Shutdowns/Crashes w/ Linux? (Score:3, Informative)

    by krishnaD ( 514548 ) <kdagli at infofin dot com> on Monday September 15, 2003 @12:16AM (#6961239) Homepage
    /var/log/messages, /var/log/syslog should give you enough info about kernel . Also there are lots of tools to enable various kind of accouting check sa.

  • Flag it. (Score:3, Interesting)

    by slittle ( 4150 ) on Monday September 15, 2003 @12:31AM (#6961316) Homepage
    Same way they know to fsck/chkdsk the drives: if a 'dirty bit' (or file, in your case) exists during boot, shutdown was unclean - log it. Otherwise create it. Only clear it as the last step of a clean shutdown.

    • Re:Flag it. (Score:4, Interesting)

      by Creepy Crawler ( 680178 ) on Monday September 15, 2003 @12:42AM (#6961373)
      You fail to understand what happens to create the "Dirty Bit".

      1: System starts up (say clean).
      2: It marks a bit on the partition that system has been started up.
      3: Usage Usage Usage
      4: Send shutdown
      5: System umounts cleanly. Undoes "dirty bit"
      6: Power == 0

      On a dirty FS, stage #5 is never hit so when system comes back on, it checks the bit and detects unclean shutdown. The bit is never wrote during the unclean shutdown.

      In the similar problem, I see problems when NTkern crashes. How exactly does it manage to:

      1: Read the partitiom
      2: Read the program on the partition
      3: Run the insert log program to add log entry
      4: Still have the "blue screen"

      I smell nasty data corruption waiting to happen. After all, if you cant guarantee the state of the kernel, does it really justify reading, writing, and executing on a crashed kernel????

  • Kernel Panic on Linux? Sounds like hardware prob. (Score:3, Informative)

    by Radical Rad ( 138892 ) on Monday September 15, 2003 @12:39AM (#6961357) Homepage
    After 10 years without ever needing to apply the knowledge I forgot how.Would the magic sysrq key [tldp.org] help? I bet it is a hardware problem though. And what about logging power outages? That is easy to do. APC probably has Linux software already to do this. For other logging there is ample facilities on Linux. Start a syslog server. Point everything to the loopback address.

  • Other OSes (Score:5, Informative)

    by menscher ( 597856 ) <menscher+slashdot@u i u c . e du> on Monday September 15, 2003 @12:49AM (#6961398) Homepage Journal
    This will probably be modded down as flame bait, but I can't resist pointing out what some other OSes have done when crashing:

    IRIX will core dump to the swap partition. On the next boot it analyzes this core file, which includes various system logs, etc, and saves useful output in /var/adm/crash. You know you've done a good job when the kernel panic causes a panic, called a double panic. I used to be able to trigger those at will. Hrmm, I should test that on the current release.

    AIX summarizes the likely causes of failure (power failure, someone pressed the power switch, or power supply died, etc). I've seen (but do not personally use) a similar thing with IRIX that actually assigns a percentage confidence level to its guess.

    Of course, usually you know there was a power failure because your UPS told you so.... I did have one case where we had a very brief outage (or maybe just a brownout). Every machine in the building had rebooted.... except one. That RS/6000 had an eerie log message like "power failure detected". And no, it was not on a UPS. I was rather impressed.

    Sadly, I don't know how to get any useful information out of linux. And don't give me crap about it never crashing. I can prove otherwise. Too bad I can't figure out why.... Maybe a kernel developer will read this and copy some ideas from the commercial Unix vendors.

    • Re:Other OSes (Score:4, Interesting)

      by FueledByRamen ( 581784 ) * <sabretooth@gmail.com> on Monday September 15, 2003 @01:33AM (#6961587)
      f course, usually you know there was a power failure because your UPS told you so.... I did have one case where we had a very brief outage (or maybe just a brownout). Every machine in the building had rebooted.... except one. That RS/6000 had an eerie log message like "power failure detected". And no, it was not on a UPS. I was rather impressed.
      I had a similar interesting experience with an SGI Indy (Irix 6.5.13, or thereabouts). I was booting it up after it'd been sitting for a while, just to see what I had running on there. While it was going, and I was fumbling around for an ethernet cable for it (it takes several minutes at boot to wait for a cable instead of noting its absence and moving on), I kicked the power strip that it was on and the plug wiggled around in the wall socket. I heard a spark jump in the socket, and the monitor it was on (Dell/Sony Trinitron 19") went to half-height mode for a few seconds, spitting and clicking, turning the screen on and off and varying the vertical height randomly.

      I expected the Indy to kernel panic or turn off. Instead, below the complaints about the missing ethernet cable ("en0: link carrier not detected" or similar), there was a lone status message: "Power failure detected."

      No UPS, no power saving devices of any kind, only the filter caps in the power supply between the logic board and the unreliable, crufty power system of a 70 year old house at the mercy of a power strip first used on my (brand new at the time) Atari 800. The other computer on the power strip (350 P2 running RH 7.1) rebooted hard, right in the middle of heavy FS activity. I had to hit the reset button before it would come back up again, too - the brownout hung the POST.
      • I guess that they have some sort of monitor on the "power good" wire that comes from your power supply. Maybe it latches and is then reset? If you get a brief "power bad" and still have enough juice in the capacitors to run things, the kernel notes the latched line and resets it and adds a "power failure detected" log.

        Well, that's how I'd do it if *I* was a Quality Computer Manufacturer, anyway ;-)

    • Re:Other OSes (Score:3, Informative)

      by Ster ( 556540 )
      Mac OS X writes a crash dump to the non-volitile RAM in the event of a panic. Then, after the next successful boot, it reads out the dump and adds it to /Library/Logs/panic.log. If, for some reason, the machine won't come back up, you can probably read the dump from OpenFirmware.

      -Ster

      • Re:Other OSes (Score:4, Interesting)

        by kinema ( 630983 ) on Monday September 15, 2003 @02:24AM (#6961750)
        I wonder if /dev/nvram (the small amount of NVRAM availible on the RTC) is large enough to store such a dump.

        • Re:Other OSes (Score:2, Informative)

          by Tux2000 ( 523259 )
          Nope. RTC memory is something between 128 Bytes (IBM AT) and 2 KBytes (IBM PS/2 series). And each bit of it is used for the BIOS and some hardware stuff (Microchannel requires a lot of memory). Perhaps, some machines have a few unused bits. But you can't stuff all your memory into them. You can't compress several megabytes or gigabytes into 10 to 20 Bits (at least not lossless). With a lot of luck and deep knowledge of the used machine and BIOS, you may be able to store a dirty-or-clean shutdown flag. But a

    • Re:Other OSes (Score:5, Informative)

      by anthony_dipierro ( 543308 ) on Monday September 15, 2003 @02:37AM (#6961793) Journal

      IRIX will core dump to the swap partition.

      FreeBSD does this. HP/UX does this. I always assumed Linux did it too, it just wasn't turned on by default. I guess I was wrong.

      As a side note, my first job out of college was to analyze core dumps from HP/UX. There's an awful lot you can learn from these things. Not just stack traces, the entire memory of the system is contained in the dump. It's time consuming, but a large portion of the time you can find out *exactly* what went wrong.

    • FWIW, Solaris does the same as IRIX, saving the output to /var/crash/`hostname`

      From that, you can run some tests on the core files to get some info about what went wrong as well as things like stack traces & process lists. Even without analyzing that, you'll generally get some info in /var/adm/messages.

      Linux should have some method of capturing what errors were generated during crashes; count it as one of those "enterprise level" features....

    • IRIX will core dump to the swap partition. On the next boot it analyzes this core file, which includes various system logs, etc, and saves useful output in /var/adm/crash.

      Solaris does the same thing. Actually, I think several commercial Unixes do this. Some even provide some basic analysis tools so that you can pore over the /var/wherever/crash dumps yourself; see which processes were running, which ones were on the CPUs when it crashed, which instruction was executing, etc.

      I've always been disappo

      • Solaris does the same thing. Actually, I think several commercial Unixes do this.

        I recall that even 4.1 and 4.2 BSD had crash dumps and rudimentary crash dump analysis tools. I think it was 4.2 BSD that introduced dumping to the swap partition and the utility that snarfed the dump on reboot. (I could be wrong though: it was 20 years ago).

        Warning: gratuitous punctuation detected!

    • Re:Other OSes (Score:2, Interesting)

      by cookd ( 72933 )
      Windows does something like this too.

      At Blue Screen, it will make a dump in the swap partition if so configured. The dump can be a 64k error summary (MiniDump), kernel memory dump, or a full physical memory dump (if swap > physical memory). While there is a slim possibility that doing this might make things worse (if the code to write the dump is corrupted, or the disk driver is corrupted), it is MUCH more likely that the information written will be useful. Also, the swap partition driver is pretty s

    • Re:Other OSes (Score:4, Funny)

      by isorox ( 205688 ) on Monday September 15, 2003 @04:59AM (#6962163) Homepage Journal
      Nah, if kernel developers read slashdot, nothing would get done!

    • At work, we used to have a SUN E250 [1]. One day, the power went away. (Turned out to be a problem with the airco.) After the power came back, I checked the logs, and I saw that the machine had been writing messages like "Power lost, running on backup power", and when the power came back "Switching to AC". (The monitor wasn't protected, so there was no way to check the machine during the blackout.)

      The PC in the same room didn't have backup power and shut down. It came back with no ill effects.

      The RS/600

  • Try the Linux Kernel Crash Dump (LKCD) patches (Score:5, Informative)

    by bigsteve@dstc ( 140392 ) on Monday September 15, 2003 @01:02AM (#6961458)
    If you are adventurous, you could try applying the LKCD patches to your kernel. Start looking here

    • Re:Try the Linux Kernel Crash Dump (LKCD) patches (Score:2, Informative)

      by Anonymous Coward
      Its located here [sourceforge.net] for the unitiated.

      I took a look at it a while back and it looked interesting. Just checked back at the site and there is a reasonable howto provided by IBM in the doc section which should give you some idea of what/how it works.

      Its worth a look, but honestly, it sounds a LOT more like a hardware issue than software. Is the server on an UPS? If not, get it on one. Reseat all your cards and ram etc then see if it crashes as regularly.

  • rc.local (Score:3, Informative)

    by bobbozzo ( 622815 ) on Monday September 15, 2003 @03:18AM (#6961919)
    As others have mentioned, there are various ways to see when the system rebooted.

    If you want to be emailed if the system reboots, put something at the end of /etc/rc.d/rc.local, if you're using something like RedHat (SYSV init, IIRC).

    Logwatch will probably let you know if the system rebooted also.

    If you want a log of the kernel panic, or something else, that's a lot more complicated, as others ahve mentioned

  • "Linux crashes" are probably contact failure. (Score:4, Interesting)

    by Futurepower(R) ( 558542 ) on Monday September 15, 2003 @03:37AM (#6961979) Homepage

    As others have said, the "Linux crash" is probably hardware failure.

    The most common cause of serious failure, if the software has been installed correctly and tested, is bad contacts. To fix the problem, just loosen the screws that hold the adapter cards, pull the cards out about 1 millimeter or 1/32 of an inch, push the cards back in fully, and re-tighten the screws. Also, pull all connectors off a similar amount, and push them back on. Do the same with the memory modules. That's all.

    The scraping caused by moving the contact points a tiny amount is actually very violent on a micro scale. The scraping removes oxide that causes a contact to lose electrical conduction.

    This is reliable information. I've been selling and occasionally repairing PCs since before IBM sold PCs, back in the days when personal computers cost $2300, had two diskette drives and no hard drive, and ran the CP/M operating system.

    My guess is that, if you had a penny for every real crash of a stable distribution of Linux, after a few years you might still have to borrow money from your little brother to buy a piece of bubble gum.
    • Do you recall the parable of the Good Samaritan? Of course you do. Now bear with me here:

      Well the context around it is he is telling it to a Pharisee in answer to the question "Who is your neighbour?"

      Well, Jesus tells the story, and when he asks the Pharisee which of the three men was his neighbour, the Pharisee answers: "The one who helped him." Y'see, the thing is that the Pharisees despise the Samaritans so much that he couldn't bring himself to even say their name.

      I was just wondering if you saw any


      • Nonsense.

        I have a lot of experience fixing hardware failures. Before I started doing computer work exclusively, I was an electronics design engineer. So, I'm able to understand hardware issues, and have something to contribute in that area.

        Perhaps Slashdot people don't have much experience with hardware problems, and are skeptical of anyone who does, because answers to hardware problems are not usually modded up, and are often attacked.

        Linux has millions of technically knowledgeable users. Tho
        • Remember that the drivers are part of the linux kernal. Linux does contain drivers of high quality, but it also contains drivers which sometimes crash and finding out which driver causes problems is quite a problem. So he might have a piece of hardware with a bad driver.

          Martin

        • I built the machine I am writing this on around 18 months ago. After a few months it became totally unstable (after a software upgrade to SuSE 8.0, I think). Now it runs SuSE 8.2 with absolutely no hardware changes and has not died on me for months.
          Other people had no problems with that level.
          The driver for the Realtek 8139 that came with the early 2.4 kernels used to kill the machine I first ran it on. Kill it stone dead, I had to hit reset to restart. The machine is dual-boot and worked fine under Win
  • When will M$ learn that their crappy WinDoze software is just a bug ridden mess of......excuse me, what? Linux....crash??!?! I have dreamed a dream and now that dream is gone....

  • It'll never catch the things you want... (Score:3, Interesting)

    by Bazman ( 4849 ) on Monday September 15, 2003 @04:51AM (#6962143) Journal
    Murphy's law will apply and the thing that causes your system to crash wont be trapped by whatever magic you try to log it with! We recently had a machine that would just power-down without warning. I eventually discovered it happened after intensive CPU load for about 20 mins, figured maybe it was some heating problem, kicked up the sensors package and spotted the CPU temp heading into egg-frying temperatures. It seems the BIOS would just protect its motherboard by shutting down. The kernel had no chance to report anything.

    • Check out the lm_sensors project at http://secure.netroedge.com/~lm78/ [netroedge.com]

      There are tools to monitor CPU temperature under Linux. Most PC motherboards made in the last few years have included temperature monitoring. The biggest problem is that most motherboard makers include different "fudge factors" in their setup, so different mobos have different settings and finding the actual temperature can be tricky.

      What I do:

      • Don't worry about the accuracy of the temperature that's being reported. It has no bea

  • Depends on what it's doing (Score:4, Informative)

    by Sits ( 117492 ) on Monday September 15, 2003 @05:18AM (#6962210) Homepage Journal
    To the best of my knowledge linux doesn't automatically reboot after a kernel crash unless you have told it to. If the crash was that severe this means you can walk up to the crashed machine and read the oops off the screen. If the machine isn't oopsing before the reboot this suggests some sort of hardware fault (e.g. your CPU is overheating). If it is hardware resetting the machine it is very unlikely that Linux can tell you what the fault is by itself (e.g. if it was the CPU overheating you will have to find someway to log the temperature to a file and observe the graph up to crash yourself).

    Oh and here's a useful way of working out whether there was a crash or not:
    last -x | grep "shutdown\|reboot"
    Every reboot that doesn't have a matching shutdown was probably a crash (other than the last line).

  • Here's how: (Score:5, Informative)

    by samjam ( 256347 ) on Monday September 15, 2003 @05:19AM (#6962213) Homepage Journal
    1) First disable console blanking, that way when you get to the crashed box and plug the monitor in you can see the kernel panic message. /usr/sbin/setterm -blank 0 -powersave off -powerdown 0

    We had some early kernel 2.4 redhat boxes crashing like the dickens for a while, it was a kernel problem and only when it happened on a local machine under our eyes did we get to realise what had happened.

    2) Network syslog;
    If you syslog to a central machine not only does it make error spotting centralised and easier but it means you have the last gasps of the crashed machine logged on a machine that is still up.

    Sam

  • serial console (Score:4, Informative)

    by treat ( 84622 ) on Monday September 15, 2003 @05:47AM (#6962279)
    A serial console (make sure you enable the magic sysrq key! for some reason RedHat disables it by default) is an essential tool for any Linux server you care about. If you don't have the money for a console server, just plug servers into each other.

    If your machine crashes without a panic message, however, you're out of luck. Wait until crash dumps are available - I'm surprised this isn't a 2.6 feature. Until we get crash dumps that work 99% of the time (like on Sparc-Solaris), Linux will continue to suck. At least it sucks less than the alternatives.
    • Actually, the "magic sysrq key" is disabled by default for a damn good reason.

      The "magic SysRq key" is a key sequence that allows some basic commands to be passed directly to the kernel. Kernel software developers use this interface to debug their software. Under most circumstances it can also be used to uncleanly reboot the computer, something that is otherwise difficult or expensive to do remotely.

      Anyone can dial into a modem and send a break, so if the serial console is attached to a modem we need
      • So. the SysRq key is disabled because it can be used (remotely) to do bad things, like an unclean shutdown,

        So you're saying that this is a massive security hole on every one of Sun's Sparc machines that has gone unnoticed all these years, as it has the same problem.

  • Although not really capable of providing an audit of reboots (for a variety of reasons, already outlined above), Snare for Linux (google for 'snare') is roughly analagous to the Windows Event log.

    Snare is capable of monitoring events such as file opens, execve's, setuid/setgid and so on, which may assist in tracking down the problem.

    Red.

  • The cleaning team (Score:3, Funny)

    by Tux2000 ( 523259 ) <alexander.slashdot@foken@de> on Monday September 15, 2003 @08:10AM (#6962664) Homepage Journal
    "Now, where was the power outlet for the vaccuum cleaner? Hell, I'll tear out that red cable and plug the vaccuum cleaner there."
    • Even better:

      "Hmm, that box in the corner is beeping. That doesn't sound good - I think I'll turn it off."

      The box was a UPS.
      • Something similar happened with our film processor (processes film which comes out of the imagesetter). It beeps when it wants chemicals --- cleaning crew turned it off. Know what happens to unheated, unstirred chemical in a fim processor? It solidifies into a crystalline mess.

  • Yes! (Score:5, Funny)

    by twistedcubic ( 577194 ) on Monday September 15, 2003 @11:14AM (#6964133)

    Is it possible to do something similar in Linux?

    Yeah, but we have to wait until our SCO insider funnels us the code.

  • Linux Trace Toolkit (Score:2, Interesting)

    by bendl ( 551642 )
    I'm working on a project called Linux Trace Toolkit (LTT) [opersys.com] that is suitable for an automatic logging.

    LTT log every system call at a ns precision in a RAM buffer and then on disk. The events include, for instance, read/write/open operations, system call, interuptions, process state, disk and internet interface operations and so on. You can add specific event by modifying your application and recompile with the LTT library.

    LTT is not yet included in the kernel and was not choosen after the "Halloween Freez

  • I don't know if this applies to you situation but it can't hurt to check the fan. My work Linux machine got in the habit of crashing for a bit. Turned out the CPU fan wasn't working. I haven't have a crash in months now that I've fixed it.

    Anyway Linux machines rarely crash in my experience and my top suspect is ussually hardware when it does.

  • Some ideas (Score:5, Informative)

    by Gudlyf ( 544445 ) <<moc.ketsilaer> <ta> <fyldug>> on Monday September 15, 2003 @03:19PM (#6966730) Homepage Journal
    Mission Critical Linux [missioncriticallinux.com] does this.

    There's also the LKCD [sourceforge.net] (Linux Kernel Crash Dumps) package:

    KCD contains kernel and user level code designed to:

    • Save the kernel memory image when the system dies due to a software failure;
    • Recover the kernel memory image when the system is rebooted;
    • Analyze the memory image to determine what happened when the failure occurred.

  • A few hints (Score:3, Interesting)

    by kasperd ( 592156 ) on Monday September 15, 2003 @03:54PM (#6967074) Homepage Journal
    You can use a serial console or try out some version of the netconsole patch to get the messages on another computer. (Notice that netconsole over the internet is probably possible, but it is sent in clear and can be snooped or modified). I also recall reading about some patch to keep a new kernel ready in memory that could be booted with arguments telling it where to find the log from the old kernel, I even think it included a checksum to prevent booting the new kernel if it had been corrupted.
  • Unexpected shutdowns? Crashes? You must be mistaken. Linux does not crash. Ever.

    Now, what was your name and address again?
  • You can use a central NetDump server to collect oops message and a dump of physical memory of every Linux box on your network...

    chekout
    http://www.redhat.com/support/wpapers/r edhat/netdu mp/

    another link with the lkcd patches
    https://projects.clusterfs.com/lustre/Net Dump
  • Type last
    You wont be able to find out a why without taking a stroll down /var/log/messages and guessing what lead up the a problem, but then again how often does that happen anyways?

    Quick tip, try checking your irq's..

    DRACO-

Slashdot Top Deals

Get hold of portable property. -- Charles Dickens, "Great Expectations"

Close