Anyone Besides Zune Owners With New Year's Crashes? 480
aputerguy writes "My Fedora 8 Linux server crashed sometime between 18:59:40 EST (GMT -5:00) and 19:00:00 EST (GMT -5:00) on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same hardware non-stop for more than six years and other than the occasional reboot for kernel (or distro) upgrades, it has not crashed more than 1 or 2 times in 2237 days of cumulative uptime. Nothing other than background processes were running at the time of the crash. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues with their computers or electronic devices?"
Well (Score:4, Funny)
Well, you know what they say, this wouldn't have happened with Red Hat.
Re:Well (Score:5, Funny)
Nobody got fired for going with Redhat?
Re: (Score:3, Funny)
Re: (Score:2)
Shazbot!
SKY TV set top box (Score:5, Interesting)
Here in the UK, our skytv settop box crashed (lost all tv channels but not the menus precisley at 00:00 1/1/2009 needed a cold boot to get the channels back.
Re:SKY TV set top box (Score:4, Interesting)
Re:SKY TV set top box (Score:5, Informative)
My mythtv box (running mythbuntu) crashed within about a second of midnight as I was trying to watch the fireworks, and stopped responding to ping, ssh, everything.
My excuse for staying in and watching the celebrations on TV is that... my dog ate... my shoes.
Yes, that'll do...
Re:SKY TV set top box (Score:5, Funny)
A similar thing, though probably unrelated to the leap second - my parents VHS clock has been flashing 12:00 since 1986.
It would probably bring bad luck for the new year to set it correctly for 2009, so I think ill leave it.
Re:SKY TV set top box (Score:5, Informative)
That's because typically Cable (or Sat) channels are contracted to carriers over a calendar year. So, at midnight on Jan 1st, some channels are added and some dropped. You probably will notice new channels and a few missing ones if you look close.
Well, after drinking a couple of beers (Score:4, Funny)
Re:Well, after drinking a couple of beers (Score:5, Funny)
Being more refined, I spilt a Gin & Tonic all over my keyboard. The keyboard doesn't work now - maybe it was the drink, but as it was an MS jobby I'm willing to bet it was a Zune like crash. Crappy Microsoft products.
Re:Well, after drinking a couple of beers (Score:5, Funny)
Re: (Score:3, Funny)
Yes, I spilled Gin & Tonic on my laptop a few months ago. I had to replace the Keyboard. Frickin' Vista...
Re:Well, after drinking a couple of beers (Score:5, Funny)
nope... (Score:5, Informative)
debian etch, RHEL, centos, all 300 odd servers stayed up. so did irix and solaris boxen from ancient times of the roman empire..
driver (Score:5, Insightful)
The Zune crash was due to a specific hardware driver. Perhaps you also have an unusual hardware driver on your setup that was affected?
Re:nope... (Score:4, Interesting)
why doesn't he just set the time back and let the new year happen all over again?
Re: (Score:3, Interesting)
That wouldn't be a conclusive test. Servers are connected to networks and have persistent storage. Unless the network behaves like it's the last seconds of 2008 GMT and the disk is in the same state as before the crash, a smooth transition is no indication that the problem wasn't date related.
Re:nope... (Score:4, Insightful)
Re:boxen! (Score:5, Funny)
On Debian, RHEL, Centos & Boxen! On Irix, Solaris, Ibex & Vixen!
Re: (Score:3, Interesting)
My Debian lenny laptop froze showing 00:59 (CET). Wouldn't respond to mouse, keyboard or ssh.
Re:nope... (Score:5, Informative)
My Debian lenny laptop froze showing 00:59 (CET). Wouldn't respond to mouse, keyboard or ssh.
Thats right when the leap second hit. Time changes can cause arts to freakout which can be nasty if it's running with realtime priority. Maybe other software does the same?
No. (Score:5, Funny)
No.
You are alone. Very, very alone.
Well this is obvious... (Score:5, Funny)
Re:Well this is obvious... (Score:5, Funny)
Re: (Score:2)
Re:Well this is obvious... (Score:5, Insightful)
What's with all the 4chan idiocy on Slashdot recently?
4chan is funny when you're a teenage boy, but for those of us that aren't...
Re: (Score:3, Funny)
This thread is relevant to my interests...
Re:Well this is obvious... (Score:5, Funny)
Ever heard of OLPC project? Heeere's the result!
Re: (Score:3, Funny)
Hey now, 4chan /b/ is funny sometimes.
Hang on, I'll get you an example.
nevermind.
No problems (Score:5, Funny)
Nope. Everything's fine here in New Ampst
Re: (Score:3, Funny)
You guys still on Dialup? Want an AOL coaster?
Re: (Score:3, Funny)
I guess the real lesson here is that it's spelled New Amst
Adding some data (Score:2)
Not really enough data to tell yet, but my Fedora 8 system had no issues at all. I'm also in GMT-5. Dual Xeon 32 bit system. E7520 chipset (Intel).
What sort of crash, just froze? Kernel OOPS, spontaneous reset?
Re:Adding some data (Score:5, Informative)
Re:Adding some data (Score:4, Interesting)
A bottle rocket hitting a transformer isn't out of the question. If it hits near the high voltage terminals it may briefly arc.
Errrrrrr (Score:5, Insightful)
Re:Errrrrrr (Score:5, Funny)
Because thinking rationally is hard.
Re: (Score:3, Insightful)
Re:Errrrrrr (Score:5, Informative)
Re:Errrrrrr (Score:5, Funny)
Its more fun to be paranoid. Join the club. No wait, we don't trust you to join you might be one of 'them'.
Re: (Score:3, Funny)
Yeah, he needs a faster hard drive.
My phone did it (Score:3, Interesting)
My phone froze right after midnight and i had to remove the battery to make it work again.
It's a SE w810i.
Probably coincidence. (Score:5, Insightful)
> Could this be a coincidence
Yes. People are wired to see causality everywhere, even where there is none. Had your server crashed a week ago you wouldn't think anything of it (maybe 5% of all servers mysteriously crashed exactly one week ago, but because it was an 'ordinary' day nobody noticed). Anyway, since you noticed your server crashed at new year and reported it on /., and with 6 billion people on this planet we will soon hear stories about other computers that mysteriously crashed around midnight. Not because there has to be anything special, but because computers are crashing all the time and new year (and your post) made it appear special.
I doubt it has anything to do with leap seconds, if your computer ran for 6 years it survived the leap second of 2005.
Re:Probably coincidence. (Score:5, Funny)
So you see a pattern in people's behavior? ;)
Re:Probably coincidence. (Score:5, Informative)
People are wired to see causality everywhere, even where there is none.
Very true. There is an interesting book by Leonard Mlodinow called "The Drunkard's Walk: How Randomness Rules Our Lives" which is all about the way humans misinterpret random events to see patterns that are not there.
http://www.randomhouse.com/pantheon/catalog/display.pperl?isbn=9780375424045 [randomhouse.com]
http://www.amazon.com/Drunkards-Walk-Randomness-Rules-Lives/dp/0375424040 [amazon.com]
Re: (Score:3, Funny)
Once, at the end of the staff meeting I was holding, I casually mentioned that I noticed that 40% of all sick leave were taken on Fridays and Mondays.
Immediately, 2 people jumped out and said it was an outrage, and that we should do something about it. That something like this shouldn't be happening.
*sigh*
Re: (Score:3)
You are very stupid, aren't you? I can't say this in my staff meeting because someone else said that previously? My god, quick, someone, anyone, go ban all books on quotes.
Re:Probably coincidence. (Score:5, Informative)
Let's use that number. The odds of a server failing during the 20 seconds before midnight on 31 december are 1 in 5 million. Suppose there are 50 millions servers. Simple math says the chance of your server crashing is extremely small (1 in 5 million), but there will be about 10 people who have a crashed server. That is normal (using your number there will be 10 servers crashing every 20 seconds every day of the year) but those 10 people will think it 'an awfully unlikely coincidence', while the other 15379200 server crashes during a year are ignored.
Don't get me wrong, I'm not saying the new year can't have anything to do with the crash, I just think it's way more likely that your server crashed randomly and you see causality where none exists.
Re:Probably coincidence. (Score:4, Insightful)
Re:Probably coincidence. (Score:4, Insightful)
I think your logic is incorrect. The original poster did not say "my server went down around midnight, could this be a coincidence?" rather he said "my server, which has a particularly excellent track record of not going down, did so near midnight with very high precision. Could this not be a coincidence?" Given that this happening at any specific time is very unlikely compared to the relative abundance of rollover errors, this is a very legitimate hypothesis. Furthermore your argument is essentially saying that anything with a non-zero probability of occurring randomly is probably not a coincidence. Otherwise, instead of comparing to some 50 million servers you ought to be comparing to a much smaller number of servers meeting the description of the original poster's. I don't think you pose any legitimate argument that this is coincidental, and it strikes me as very probable that it is not.
If you want to show that this is anything but a coincidence, you either need to show that this happened to more than one server, or you need to demonstrate the mechsnism. At this point we have exactly one server and we can't point to a specific bug. Until that changes, "coincidence" is the best answer.
For instance, this could be an entirely local problem. The motherboard or some other hardware component is beginning to fail, and the server will start crashing more frequently until that component dies completely. Or it could have been caused by a power surge, or a problem resulting from some bad wiring. Or the guy who manages the server above it came in to swap out some hardware and accidentally unplugged the server, and won't admit to it. (I have a former boss who did exactly that, after he went to work for a customer.)
Sure, it could still be related to the time. Without any additional evidence, though, it's just speculation.
Re:Probably coincidence. (Score:5, Informative)
Hence, you, and the 7-9 other people who shared your experience... and nobody else.
Re: (Score:3, Insightful)
You didn't pick a random server, you picked one already known to have crashed. And you didn't pick a random 20 second interval either. The odds of that server crashing in that 20 second interval was 100%, because it was already known to have happened. This is a classic mis-application of statistics.
Admittedly, the interval right at New Year's is a bit suspicious
Re: (Score:3, Insightful)
No, it's not a misapplication. It's a textbook-standard application of statistics, which looks at the probability of an event (which did occur) happening under a "null hypothesis", in this case including 1) no extraordinary event associated with year roll-over (no time dependence); 2) all servers are stochastically identical (i.e. they each have the same failure rate).
The hypotheses are a bit strong, but it's not a mis-application.
Statistics often answers the question "How likely was that to have happened w
Google searching (Score:2)
As of this morning, Google thinks I have a spyware problem and every time I've tried to do a search I get the sorry page with a captcha to complete the search.
It sure scared me - how does a Linux box and a Macbook get infected while they're both asleep? I've done all the checking I can and I'm pretty sure my network is clean.
Re: (Score:3, Interesting)
Re: (Score:2)
Re: (Score:3, Interesting)
I think you were given a very badly infected "zombie" machines IP by your IP provider overnight.
If you give your IP to sites like http://http//www.completewhois.com [http] or http://www.senderbase.org/ [senderbase.org] they may give very good clue since zombies mostly end up on such lists. Of course if you aren't victim of some sort of monopoly, it is best to find a better managed ISP with zero tolerance to both spammers and zombies on their subnet. Not every IP address/block is equal on web since the early dial up times. At one t
Proximity card system (Score:2)
Given an infinite number of server monkeys... (Score:5, Insightful)
I Second That (Score:4, Informative)
My parents are using a MythTV box on Fedora 8 (Athlon XP1700+) and it also froze up last night at the same time (right in the middle of a recording :-( ). That was my first thought, too, because that would have been midnight UTC. However, after restarting it today, is has frozen again.
I can't see anything in the logs, but the recording ended at 19:59 AST. It should have kept going for another hour.
I have a second MythTV/Fedora 8 box (P3, 1GHz) that I use and never had any trouble with it last night.
Re: (Score:3, Informative)
Re:I Second That (Score:4, Informative)
I did some comparisons between these 2 boxes and found the following. We were both recording the same program.
messages - both had entries from the channel change script at about 19:29:50 AST.
The next message on the good box was "Dec 31 19:59:59 localhost kernel: Clock: inserting leap second 23:59:60 UTC." This message was not on the box that froze.
When I stat'ed the recording, it was last modified at 19:59:59.431 -0400.
Re:I Second That (Score:4, Insightful)
My Mythbuntu-based HTPC also froze up last night.
This is what my /var/log/messages file looks like: ...) /boot/System.map-2.6.27-9-generic
Dec 31 16:03:45 puppet -- MARK --
Dec 31 16:23:45 puppet -- MARK --
Dec 31 16:43:45 puppet -- MARK --
Dec 31 17:03:45 puppet -- MARK --
Dec 31 17:23:45 puppet -- MARK --
Dec 31 17:43:45 puppet -- MARK --
(... below is when I noticed the box was hung and restarted it
Jan 1 14:02:31 puppet syslogd 1.5.0#2ubuntu6: restart.
Jan 1 14:02:31 puppet kernel: Inspecting
Every 20 minutes, I get those "-- MARK --" messages and the last one is at 5:43PM local time which would be 11:43PM UTC (also my system clock is set to UTC, not local time). The next "-- MARK --" should have been at 12:03AM UTC, so there's a good chance the leap second messed something up.
test (Score:5, Insightful)
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
Re:test (Score:5, Insightful)
Re:test (Score:5, Funny)
Re: (Score:2, Interesting)
Amen brother.
5 of about 70 of our production servers died at exactly midnight GMT. No point in speculation until some testing is done.
Ubuntu 8.10 - MythTV Crash (Score:3, Informative)
I was watching the new years London celebrations on my Ubuntu 8.10 MythTV box. With 10 seconds to go to midnight, it crashed. Missed the start of the fireworks.
I think it may have happened around midnight before, so not necessarily an New Year problem.
RiteAid pharmacy y2k09 bug (Score:5, Interesting)
On 12/30/08, I submitted a request with my pharmacy to refill a prescription to pick up on 12/31/08, and received the following email, verbatim:
Your Rite Aid prescription confirmation
Greetings from the riteaidonlinestore.com pharmacy,
Thank you for choosing to refill your Rite Aid prescription(s) online at the riteaidonlinestore.com pharmacy.
The following refills have been sent to the Rite Aid store that you selected, along with your preferred pick-up date and time:
Patient Name: ********
Rx ******** ********
Rx ******** ********
Rite Aid Store Location:
********
********, ********
********
********
Pick-up Date and Time:
Thursday December 31, 2009 at 3:00 pm
If you have any questions regarding your prescription, please contact your local Rite Aid directly at ********. Please note that you will need to pay for this prescription when you pick it up. If you have selected to self-pay for this medication, you will pay Rite Aid's price.
Thank you for visiting the riteaidonlinestore.com pharmacy. We invite you to visit us for your other prescription needs and great deals on nonprescription items. We look forward to assisting you!
Some things to note: I've got to wait until next christmas to pick up my drugs, and they were so concerned about patient privacy, they obscured all my contact information, prescription numbers and the pharmacy's phone numbers with asterisks. (I didn't do that myself!)
So, I wonder if their log files are full of java.lang.Exception logs today...
--ob
Mysteriously coincidental with this event.. (Score:5, Funny)
Re: (Score:3, Funny)
my cat hid under the bed at almost 25 seconds into the New Year. Right after he heard the first of the fireworks. However he did restart normally about 22 minutes later after a soothing saucer of milk. I wonder if ...
FYI, servers don't restart if you give them a saucer of milk, no matter how soothing it is.
Re:Mysteriously coincidental with this event.. (Score:4, Funny)
Is that a cat-5 or cat-6? Is re-installing M.I.L.K. a normal procedure? I understand some models have patches available after a fix, is your model fixed?
It would be fair... (Score:2)
...to at least tell us why it crashed. Hard drive failure? Your fan gave out? Some sort of kernel panic or MCE? Was there a black cat in the server room? Maybe there's more Zune in it than you think.
billing software (Score:2)
A billing program I maintain failed to run today, I won't figure out why until tomorrow when I go back to work. Its no big deal but probably not a coinky-dink that it failed on new years day.
UK ISP crashed at 2356 on 31 Dec! (Score:2)
anyone know if it was Z2K-like?
Sandisk Sansa e260 4GB Media Player (Score:3, Interesting)
I had purchased a Sandisk Sansa e260 4GB Media Player for my father-in-law for x-mas and it worked on thru the 30th, but it wouldn't turn-on on the 31st all day. We finally managed to turn it on today. Interesting...
Crashy box crashes, you say? (Score:5, Funny)
Apparently, you have pre-existing stability problems with this box. The fact that it crashed yet again yesterday should come as no great surprise.
Linux 2.6.21 hangs on leap seconds (Score:5, Informative)
You didn't specify your kernel version, but if it was 2.6.21, you may have hit this:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux2.6.gita=commitdiffh=746976a301ac9c9aa10d7d42454f8d6cdad8ff2b
Thankfully this was a short-lived bug which only affected 2.6.21.
Fedora 8 locked up here (Score:5, Interesting)
Re:Fedora 8 locked up here (Score:4, Funny)
I was my DHCP server, so the network went down.
I was my DHCP server too (I entered my DHCP responses on a hex keypad) but then I got dnsmasq, and now the computer hands out addresses for me.
Anyone With New Year's Toast Accidents (Score:4, Funny)
Not here, just a coincidence (Score:3)
My Fedora 8 and Fedora 10 machines did not experience any problems. Maybe you had a power glitch, if there's nothing in the logs.
Leap Year (Score:2)
The only special thing about 2008 I know of is that it is a leap year. That was the cause of the Zune problem. Could be the cause of other problems also.
Speaking of coincidences, many people said of the Z2K9 problem that it was a merely a coincidence that it happened on New Years and on a leap year--but they were wrong. One of an item (Zune, server, etc.) crashing at New Years is probably coincidence. Several probably is systemic.
Nothing crashed on me -- madplayer hicked however (Score:4, Insightful)
Madplayer hicked three times at about 0100 CET. I thought it might have been my RAID system I had just repaired. (There was a bad sas/sata controller.) This happened over about 20 seconds. I only use Unix/Unix-like systems and to the best of my knowledge there are no embedded MS devices in this house.
Unix/Linux, etc. handles things like this well. All time sync services like NTP, DCF-77, MSF, WWVB, GPS and the rest give fair warning. I personally are in favour of ditching 'leap seconds'. Time corrections would best be made day to day, the length of today being based on yesterday. That's better, but surely someone can think up the real solution?
BillSF
PS: Frequent updates to Java caused by US daylight saving time are pathetic.
Well, I didn't get called (Score:2)
Didn't yet get a call from a panic stricken rep wondering why they can't access their webapp. So I'm assuming that either the voip system is also dead or everything's going grea [...Connection Reset by Peer]
Apparent leap-second bugs (Score:2)
I had one RHEL 4 server (out of a half dozen identically configured systems) crash at exactly the moment the leap second should have been inserted. The logs run up to 18:59:59 CST (UTC-06:00) and the system froze, when all my other Linux systems logged they were inserting a leap second. I have read reports of some Debian systems having a similar problem. The leap second code is probably one of the least tested areas of the Linux kernel (there have only been 8 leap seconds since Linux was started); there
same thing here (Score:2)
Amazing (Score:3, Interesting)
I thought there was no way that Y2K9 would affect me, then the girlfriend asked me to check on a flight for her--and I found that United Airlines' website returns 2008 flight data if you search for flight information for Jan. 1 or Jan. 2 2009! How amateur is that?
~Ben
Another anecdote (Score:4, Interesting)
I switched from Windows 95 to RedHat 6.2 many years ago, and except for reboots to upgrade the hardware (started with 200Mhz Pentium I w/ 384M and now have Dual Core w/ 2G) or OS (now on CentOS 5.2), it has crashed only twice - due to a defective USB2.0 card which I replaced.
We run LTSP so that the single server runs the entire family, using old '90s hardware for thin clients. We simply could not afford to run Windows (or Mac).
Re: (Score:2)
Maybe I've missed something, but pointing out that some Linux-based servers crashed at midnight on New Years hardly seems like an advertisement to run Linux to me.
Re: (Score:3, Informative)
Fedora 8's end of life doesn't occur until January 7th, so it would still get timezone updates.
Just because its old (Score:2)
I am running Redhat 7.2 on one of my servers w/o a problem. On one of my desktops, I am running IBM OS/2 (Ecs 2.0RC5) and there was no problem overnight.
Just because something is old does not mean it won't work. I have old pencils that have not crashed. My PC Lint 7.0 t-shirt from 1992 still works without a crash.
Re: (Score:2)
awww nothing eventful happened to my old PIII mini-Optiplex 110 running CentOS that I use as a home router/file server.
[jsd@www ~]$ uptime
15:36:53 up 64 days, 9:16, 1 user, load average: 2.47, 1.86, 1.57
[jsd@www ~]$
The only weird thing that happens is that Apache stops for no good reason every once in a while.
I'm jealous that I don't get to participate in the conspiracy theory. If my system did go down, I'd use the time offline to put Slackware back on the server. CentOS keeps working so I let
Re:Time Mathematics and Microsoft (Score:5, Insightful)
Try once yourself to code conversion from "seconds since 1/1/1970 00:00:00" to any other user digestible presentation.
It's not as easy as it might seem.
Re: (Score:3, Funny)
Try once yourself to code conversion from "seconds since 1/1/1970 00:00:00" to any other user digestible presentation.
It's not as easy as it might seem.
Done:
$ perl -MHTTP::Date -e 'print time2str(1230796800)';
Thu, 01 Jan 2009 08:00:00 GMT
Re:Time Mathematics and Microsoft (Score:5, Informative)
Wikipedia [wikipedia.org]
NTP and the leap second (Score:4, Interesting)
A surprising number of NTP servers didn't add the leap second correctly. On the mailing list for pool.ntp.org contributors, it was reported that at just after midnight UTC that about 158 servers in the pool (about about 2000) were reporting times that were around 1000ms off. A few hours later it was only 13 that were doing that.
My own (stratum 3) NTP server got confused and declared that it couldn't determine the correct time. Some of its sources were 1000ms off from others. Given enough time, NTP will sort itself out, but I intervened manually by ditching the upstream servers that hadn't gotten it right.
If enough NTP servers were temporarily in the state that mine was in (was so unsure of itself that it wouldn't serve time to clients) then I could imagine some process that tries to sync the time and fails because ntpdate doesn't return anything useful.
Re:NTP and the leap second (Score:4, Interesting)
Failing to update surely shouldn't be a matter of crashing, however. Network services do go down; it's something you plan to cope with.
Re: (Score:3)
Yes, you're totally right of course; but the operative word is "should". No computer "should" ever crash, but they do precisely because the programmer didn't "plan to cope with" something. As a programmer, I take full personal blame.