Big Red Button Disasters? 508
FredDC asks: "The Daily WTF has a story about a Big Red Button disaster. What Big Red Button disasters have you experienced? Which ones have you caused? Are there any that you've heard about, or do you know of any that can happen any day now?"
Well... (Score:5, Funny)
Re:What kind of idiot... (Score:5, Funny)
A literal "Big Red Button" disaster (Score:5, Funny)
A technician from U.S. Worst had finished his work in the server room and on his way out he hit the Big Red Button thinking that would open the doors, like at a hospital.
Hilarity ensued.
Later that day I printed out several mock "Big Red Buttons" on sheets of paper to use as decoys next time the tech had to visit.
History Erase Button (Score:2, Funny)
Command-line FTP (Score:5, Funny)
Connected to www.workplace.domain.
220 Microsoft FTP Service
Name: shag
331 Password required for shag.
Password:
230 User shag logged in.
Remote system type is Windows_NT.
ftp> cd
550
ftp> put index.html
local: index.html remote: index.html
227 Entering Passive Mode.
125 Data connection already open; Transfer starting.
226 Transfer complete.
ftp>
The realization that one has just overwritten a public-facing, high-traffic
It's worse when
After I did this two or three times, I decided to stop being such a hardcore geek and got an FTP application with a GUI.
The different PDUs (Score:5, Funny)
Anyway, we did a big datacenter migration at my last company. I'm not going to name names, but it's a Fortune 100 company based in Austin, TX. The move was happening because we built our own building with our own datacenter.
As part of the technical staff (network engineering/security), I was given a tour of the new datacenter before it opened. My boss and assorted other folks were on the tour. My boss, by the way, was a huge...jerk.
The electrician showed us the Big Red Buttons by each of the exit doors. He also said that each of the Power Distribution Units (of which there were three) had a Big Red Button that would cut power to just the areas powered by that unit.
My boss said, not jokingly, "If you need to cut power in an emergency, see if you can figure out which PDU is involved and just cut that one, so we don't lose the whole datacenter."
I piped up: "If I'm getting 220 across my nipples, cut the whole damn room. I really don't care enough about the company to die. I can see my epitaph now: 'Here lies Dimwit. He died saving two-thirds of the datacenter.'"
Man, if looks could kill.
Re:Command-line FTP (Score:5, Funny)
I used to work for a site in Argentina that was quite famous in it's time, Datafull.com
Once, I mistakenly overwritten index.php with a version I had on my home server.
That version had black background and in white letters at the center, it said: "FUCK OFF".
The file was live for about 10 minutes, in wich 20.000 people saw it.
I always make it a point... (Score:5, Funny)
Story seems to show its age (Score:5, Funny)
I say we press it.... (Score:4, Funny)
The landlord was abroad in Tokyo, so it was just ourselves and the agent. Nice house, but whilst looking around we saw a big red button in the main bedroom. For those to whom it's obvious what the purpose was, at that time it was my first encounter with such a device - first encounter for all of us in fact. And so, with the agent waiting downstairs, the conversation went...
Friend 1: "What's that for?"
Me: "I say we press it. That's what big red buttons are -for-."*
Friend 2: "ok" (presses button)
The next scene - pandemonium as the alarms all round the house go off. It's a panic button of course - we'd never come across one at that point, so we pressed it anyway. Up runs the estate agent to find out what we'd done. We tell him - yep, love the house. We'll take it. Oh, the alarm thing? That's fine, it's because we pressed this big red button. Ah - the owner's in Tokyo and you don't know the code? And it's -what- time in Tokyo? Hmm. Err...
And out the house we went, as fast as possible. And away we drove, again as fast as possible. We'd left the agent in charge of a screaming house, which every neighbour for a mile must have heard, and with absolutely no way to shut the alarm off for several hours. It was, as the saying goes, time to be somewhere else.
Still took the house though - lived there for a few years, enjoyed it actually.
Cheers,
Ian
*I was actually quoting a friend of mine, who in turn says he was quoting some film or comic. If you happen to know the source of the quote, I'd be interested to hear it.
Re:A QA Intern Story... (Score:4, Funny)
Schwab
Choice of button (Score:5, Funny)
So anyway, there is this big fire alarm panel with tons of buttons that we never really thought about, until one night when it started beeping constantly. The ops guy found a key to it, and then we both stood there looking at the probably 60 buttons and flashing lights, etc. Personally, I would have chosen one of the black buttons marked "mute", but the ops guy went straight for the biggest red one on the board.
The result was more beeping, lots of red lights and about 5 fire-engines.
Vaxen, my children... (Score:5, Funny)
And as long as we're talking halon, who can forget the classic Vaxen, My Children, Just Don't Belong In Some Places [hactrn.net].
The Big Red Button, play in three acts. (Score:5, Funny)
Big test floor, where several large (multi-million dollar) computer systems are being configured and tested before shipment to the customer.
Tall skinny hyperactive developer (no, not me, I was just an observor) leaning against the wall of the test floor, actually *fiddling with* the Big Red Button.
Someone suggests that he ought not do that. He promises to be careful.
Act Two
Five minutes later. All the power has just gone out. It's amazing how quiet it is all of a sudden. Everyone is looking over at the tall skinny developer with his hand on the Big Red Button.
No words are spoken.
Act Three
Half an hour later. Electrician is leading the tall skinny developer around as he turns on each part of the power system in the right order. CEO and various unmollified developers watching. Back by the door, guy from facilities is bolting a flap over the Big Red Button.
The Magic Switch (Score:5, Funny)
***
Magic Switch Story
Some years ago, I was snooping around in the cabinets that housed the MIT AI Lab's PDP-10, and noticed a little switch glued to the frame of one cabinet. It was obviously a homebrew job, added by one of the lab's hardware hackers (no-one knows who).
You don't touch an unknown switch on a computer without knowing what it does, because you might crash the computer. The switch was labelled in a most unhelpful way. It had two positions, and scrawled in pencil on the metal switch body were the words "magic" and "more magic". The switch was in the "more magic" position.
I called another hacker over to look at it. He had never seen the switch before either. Closer examination revealed that the switch had only one wire running to it! The other end of the wire did disappear into the maze of wires inside the computer, but it's a basic fact of electricity that a switch can't do anything unless there are two wires connected to it. This switch had a wire connected on one side and no wire on its other side.
It was clear that this switch was someone's idea of a silly joke. Convinced by our reasoning that the switch was inoperative, we flipped it. The computer instantly crashed.
Imagine our utter astonishment. We wrote it off as coincidence, but nevertheless restored the switch to the "more magic" position before reviving the computer.
A year later, I told this story to yet another hacker, David Moon as I recall. He clearly doubted my sanity, or suspected me of a supernatural belief in the power of this switch, or perhaps thought I was fooling him with a bogus saga. To prove it to him, I showed him the very switch, still glued to the cabinet frame with only one wire connected to it, still in the "more magic" position. We scrutinized the switch and its lone connection, and found that the other end of the wire, though connected to the computer wiring, was connected to a ground pin. That clearly made the switch doubly useless: not only was it electrically nonoperative, but it was connected to a place that couldn't affect anything anyway. So we flipped the switch.
The computer promptly crashed.
This time we ran for Richard Greenblatt, a long-time MIT hacker, who was close at hand. He had never noticed the switch before, either. He inspected it, concluded it was useless, got some diagonal cutters and diked it out. We then revived the computer and it has run fine ever since.
We still don't know how the switch crashed the machine. There is a theory that some circuit near the ground pin was marginal, and flipping the switch changed the electrical capacitance enough to upset the circuit as millionth-of-a-second pulses went through it. But we'll never know for sure; all we can really say is that the switch was magic.
I still have that switch in my basement. Maybe I'm silly, but I usually keep it set on "more magic".
GLS
(1995-02-22)
Re:Well... (Score:4, Funny)
On Pushing Buttons... (Score:3, Funny)
Ford: "What happened?"
Arthur: "A sign lit up saying 'Please do not press this button again.'"
(Douglas Adams)
Then again, this would make really want to push the button...
Re:Command-line FTP (Score:5, Funny)
Yello Button DIsaster (Score:3, Funny)
E P O -we have- E P O (failure IS an option) (Score:3, Funny)
I have.
Big orange cord (Score:3, Funny)
I have two solaris oopsies. (Score:5, Funny)
Two solaris oopsies.
One: Somehow, I don't know how, I accidentally deleted
Two: Not wanting to accidentally halt the machine without really meaning it, I moved the halt command to halt.ireallymeanit. I then replaced halt with a small shell script that echoed "You don't want to halt this machine" (sleep a few seconds) "If you do, type halt.ireallymeanit" (sleep a few seconds) exit.
Then, to test it, I type halt. Without (duh) first typing which halt to make sure there wasn't a halt command before the
Needless to say, it's not Solaris' fault, but somehow I always managed to screw up that OS without meaning to, so I have developed a healthy fear and loathing for it. I'd like to think I've grown up a bit since then - this has been like 3 or 4 years now, and I've learned a helluvalot since then.
~Wx
Re:Big orange cord (Score:2, Funny)
The kitchen door had 'Do not enter' plastered on it, which we assumed would be enough. During one of the competitions, all of the admin PCs and servers (and some of the network gear) just died. Lanners started whining, so we had to look for the source of the power outage. Just before we went into the kitchen, one of the lanners walked out with his recently reheated KFC. He'd gone into the kitchen to use a microwave, saw that it was unplugged so he unplugged the extension cord which was in the way.
We considered making "Microwave Boy" t-shirts made for the next LAN..
Re:Well... there is this red handle.... (Score:5, Funny)
It is on a chain that goes way up to the roof...
I don't know.
I wonder, I wonder.
Other people wonder.
Maybe it has been pulled many times? Maybe someone will pull it and sprinkle all the PCs? Maybe someone pulls it and we all get flushed down the intertubes. (Funny, my kids have never seen a toilet with a chain)
Life is full of little puzzlements.
(It all goes wrong tomorrow, IT WASN'T ME! I HAVE RESISTED TEMPTATION FOR YEARS NOW!)
Re:I have two solaris oopsies. (Score:1, Funny)
My favorite Big Red Button story (Score:4, Funny)
Big Red Button in the New Datacenter [ouch] (Score:4, Funny)
But, in their wisdom, they offered a Big Yellow Button to hold off on opening up the sprinklers. As long as the Big Yellow Button is held down, the sprinklers won't actually spray water. So you can see that this could be very useful in a false alarm situation.
Well, one day, an employee decided to play around with the Big Yellow Button. Yes. Do you know what is coming? No, you probably don't. And neither did the employee. Well, earlier in the story, I mentioned that this was a new datacenter. And it looks like they didn't do a great deal of testing of its emergency features before they put it into use. You see, the people who wired up the Big Yellow Button had swapped the wiring at the other end with the wires to the Big Red Button. But nobody tested the buttons out to make sure they worked. Since you are a Slashdot reader, I'm sure you understand the result of this unfortunate wiring mistake, and lack of testing.
The employee, however, was not fired or significantly disciplined for the significant outage (and disk damage from a sudden power loss) that resulted.
Re:A literal "Big Red Button" disaster (Score:3, Funny)
Yes, but not until after someone has accidentally pushed it.
We had an EPO button near the door, as required by code. But, it had no guard at all on it -- not even a shroud that required you to press the button with a thumb (instead of the heel of your hand).
We usually stacked boxes of continuous feed paper (for a line printer) against the wall, on the other side of the EPO button from the door. One day, someone delivered more paper and was stacking it, and you guessed it: a box slipped and smashed into the EPO button.
The next day, maintenance installed a shroud around the button.
General Motors Distribution Center Big Red Button (Score:1, Funny)
Yep, the IBM tech was setting up our server which was placed directly in front of one of the 3 Big Red Buttons (all of which kill the entire plant), about 27 inches away from it. After connecting some cabling, he stood up and his back pressed the button. Well the building was out for 27 hours as a certified electrician (the union electricians are not allowed to touch live wires or fuses/breakers as they are dangerous) had to be called out, the main power into the building restored and EVERY fuse and breaker in the building reset or replaced.
However, the next day we did receive 3 plastic covers with key locks for the buttons.
Re:The Magic Switch (Score:3, Funny)
My attempt at scripting a plausible origin:
"Hey Bob - it looks like that 100 MHz crap on the line goes away if we let the front chassis panel float."
"Great. Problem solved. Just isolate the panel and let's forget about it."
"Okay, but let's add a switch just in case we ever want to change back. Say, what luck - this old toggle switch shorts one pin to the chassis internally."
"Great. Be sure to label that switch so we know what it does."
"But we don't know why it works in the first place. I know, let's call it 'magic.' Here, hand me the label maker and grab another beer while you're up."
- skip five years -
"I figured out why the new translation board keeps dropping frames - some joker added a switch that isolates the front chassis on this old machine from ground. When this thing is in the 'magic' position, all the hardware in rack 7 is floating!"
"You mean we spent eight days troubleshooting something that turned out to be a goddamn switch?"
"Yup. We're just lucky we didn't fry anything expensive."
"Tear out the switch."
"But someone must have put it there for a reason."
"Okay, then just make sure it's labeled so we don't end up doing this again in five years."
- skip five years -
etc.
Re:Monday, January 22, 2007 (Score:5, Funny)
oldskool ibm professional write (Score:2, Funny)
anyone remember ol' professional write? (http://en.wikipedia.org/wiki/Pfs:Write [wikipedia.org])
Some people seem surprised that i remember this vividly. I was 3 years old, my mother was just starting to work on her law degree, and was writing up something important via professional write. The "microcomputer" we had i believe was built by my uncle, and the power button on that thing was big and visible. From my vague memory it was a red circular button, with at least an inch in radius. Just like a frustrated horny bull running towards a red cape, my nubby little index finger launched towards that big red button. Result: (insert the dirtiest swear words here), my father comes running in, my mother starts crying (my guess is it was important, and back then autosave was only a dream), and the rest of the memory is too fuzzy to remember. Perhaps for the reason that it was something that shouldn't be remembered. Now that i look at my watch, it's almost time for my psychotherapy appointment! Cheers!
And then it all went black (Score:4, Funny)
The other thing she did was she worked out that she could get into the machine room with her pass if she went via the emergency exit. We kept finding puddles of water under the raised floor that we couldn't explain until one weekend I noticed her carting a bucket and mop into the machine room to give it a good scrub with a liberal amount of water.
She doesn't work here anymore.
Well I didn't press the button.... (Score:3, Funny)
We had several antiquated BBC micros in one of the classroom blocks public areas - in theory for getting work done during break but since no other classes used those machines they usually ended up having games on them for those that knew how to find (or write) them.
Bored one lunch time I typed in the same 20 or so lines of basic on each machine and with the help of a friend hit enter at the same time on each.
The screen now flashed from black to red and black, beeped every second and read "This computer will self destruct in: 5:00" counting down every second. After a bit of a giggle (ok we were 12) at how this looked we walked away and wondered who would find it.
It turned out to be one of the dinner ladies (for those that didn't have this concept in school - they are non teaching staff that wander the open building and grounds during breaks keeping an eye on things). Being about 60 or so she obviously beleived what the computers said and hit the fire bell!
One evacuation later and a short investigation of the computer screens (which if I had got the code right should have one letter on each screen - B O O M - I never did see the final result so I don't know) everyone returned to classes.
Didn't really hear much until my next IT class, at the end the teacher took me to one side - his comments were basically:
"Very nice trick, but please, don't do it again, ok?"
Was fairly obvious it was me, only the lowest years of the school were allowed in that area, I was the only one in that categry that could have done it. I do remember the teacher was trying to stop himself from grinning.
Re:First Job Ever (Score:3, Funny)
Also is not precisely the same as the intended delete from tblcustomer where customerid = 1783.
Figured that out the hard way.
Still not quite as funny as the co-worker of mine that managed to justify /etc/passwd (as in line-wrapping it at 80 cols).
Don't even ask what he was doing in there with a text-editor in the first place...
Re:Well... (Score:3, Funny)
1-year old daugher hits UPS switch (Score:5, Funny)
I even got a "decoy" keyboard for my 11-month-old daughter to play with.
Of course, she found the UPS switch in seconds. It had a nice glowy LED above it, and was sitting on top of the Linux server just at her shoulder height.
All three PCs, the whole study, powered down, and not in a nice graceful apcupsd way, just a sudden BOINK, follwed by darkness and silence, penetrated only by a happy gurgle.
Thank heavens for Linux software RAID mirroring.
(A couple of months earlier, she managed to cause Windows to prompt "Add new hardware - Searching for drivers" [blur-o-matic cameraphone photo [aoakley.com]] by sucking the end of my iPod USB cable. Unfortunately I didn't have any Win2K drivers for a 9-month old baby. I bet Ubuntu installs them by default, even though the GNU crowd complain they're not truly free.)
Annabel is one on Sunday. Wish her happy birthday.
This takes some 'splainin'.... (Score:3, Funny)
The US Navy's P-3 Orion (in many variants) is a 4-engine maritime patrol airplane. The engines are Pratt and Whitney T-56 turboprops, a powerplant shared with the C-130, the E-2 and the C-2.
In the flight station, in the top center of the instrument panel are four big yellow handles that you pull when you need to shut the engines off in an emergency. Because they are used for emergencies, the are cleverly called "e-handles". Underneath each e-handle is a red button. This is the the button that releases the contents of the high-rate-of discharge (HRD) fire extinguisher in the corresponding engine compartment. You can see a picture of this configuration here:
http://www.namsa.nato.int/gallery/systems/p3orion
I was in the navy flying with a P-3 crew in the mid 1980s. We were at Kadena Air Base on Okinawa, trying to take off and get to the same place in the ocean where some foreign naval unit was exercising its right to free navigation in international waters during the Cold War. Even though there are other P-3s on the ramp that day, *our* P-3 was special, since it had some sensors that that other kids didn't have yet..which is why we got to hang around the airplane during this maintenance delay...
During our engine starts, there was a problem with the number two engine (inboard on the port side). It was fixable in an our two, but the mechs would have to pull the plane into the hangar to do the work.
It's late spring, a mild sunny day, and I curl up by the port overwing exit in the tube; this part of the crew cabin it has enough space to stretch out and get a decent. The overwing hatch is open, cool breezes are flowing off the East China Sea. Others are lounging in their seats, on the bunks in the back, in the flight station, listening to the radio on the ADF receiver. We're just chillin', waiting for the mechanic on the ladder under the number 2 engine compartment to work his magic so we can go flying.
I can hear the sound of his tools banging around in the engine compartment, and just as I'm about to go asleep, I hear him call to the flight station (whose side window was also open): "Hey, somebody pull the number 2 e-handle"...
The e-handle does a number of things, including severing some mechanical connections between the propeller and engine turbine, cutting fuel flow, and generally making sure that the the motor you shut down during an inflight emergency won't be further damaged.
That's the 'splaining. Here's what happened next....
The guy in the flight station who responded to this request was neither an aviation mechanic nor an aviation electrician, nor an aviation hydraulics technician. Regrettably, he was an aviation electronics technician, and this particular one was not, shall we say, the sharpeset tool in our shed that day.
Here's what he didn't know:
He didn't know that the red button under the e-handle was *not* the push-to-release-button for the e-handle. So, before he pulled the e-handle, he pressed the red button underneath it, believing it *was* the push-to-release button.
When he pushed the button, the contents of the HRD fire extinguisher emptied --immediately-- into the number 2 engine compartment...where our helpless mechanic was still working.
The good new was that nobody got hurt (including the poor bastard who pushed the button, who was spared physical harm by the mechanic). The mechanic was none too pleased, because now cleaning the engine compartment just got added to his list of things to do...we didn't get to go fly that day because it takes many more hours to clean up the engine compartment after the fire extinguisher is emptied out in there.
Big red buttons ang big yellow handles...equal sources of entertainment.
Re:THNTD (Score:2, Funny)
Re:Blue dye (Score:3, Funny)
You could spread some dye on the outside pull station itself - somewhere out of sight in the handle - but that would require you know where someone would pull the alarm or dye a massive number of stations just to be sure. Even then you run the risk of shorting out the damn thing which would result in a horribly ironic outcome.
UPS's don't always work... (Score:1, Funny)
He consequently displeased lots of little old ladies by accidentally taking the station off-air as the server powered off the second the serial cable was plugged in.
He later had a regular on-air slot guesting as the IT Expert.
Nuts (Score:1, Funny)
2 seconds interesting
Half hour unpleasant and educational.
It's a lesson I've held on to since.
Comment removed (Score:5, Funny)
Wish her Happy Birthday Annabel! (Score:5, Funny)
Re:The different PDUs (Score:3, Funny)
Obviously, you were in the wrong line of work. You should be working for the State Department.
Re:Well... there is this red handle.... (Score:4, Funny)
Hang a note on it that says "Pull me."
Re:Well... (Score:4, Funny)
ah...the days before
and people turn on the 9 to dial an outside line , then stick the 1 in when dialing long distance and then stick the 1 in for the actual phone number. so it ends up dialing 9,11-555-555-5555. i use to do internet tech support, it happens A LOT
I shut down the IRS (Score:3, Funny)
This was nearly 3 decades ago. I was a new clerk and had never even had a computer. I had done a good job with my paper-shuffling, so much so that my boss lent me out to another function (big perk, goes on your record as a wider set of experience) where I was to work in the "NEC room." This was a tiny room with a big NEC computer hooked up to an even bigger line printer inside an acoustic shell that held pin-feed, 8-part forms. These forms were Revenue Agent Reports, the final results of all audits, the paperwork you sign to agree to a change in your taxes.
I *really* impressed those folks. Seriously. First off, they couldn't get the forms aligned in the printer; they were always printing everything a fraction of an inch too high or too low on the form. It seemed a simple job to me. I noted the position of the pin-feed holes relative to some random part of the printer, printed one report, changed the position of the forms relative to that printer part to a degree roughly designed to compensate for the error on the first form, then printed another sample form. It was, as expected, properly aligned. I had aligned a new box of report forms with just one test print. The long-time workers in that little room thought I was a god. Literally, mouths dropped opened. They were accustomed to spending hours and half a box of forms getting set up. They loved me.
Next, they had a bunch of garbage records in the database that kept printing out. It was pretty simple to figure out that if I deleted each record, they wouldn't start each day with 20 garbage prints. By this time, they loved me so much the manager stopped by to meet me, sent an official memo of praise to my boss (something normally never done until a detail is complete) and started making noises about creating a position for me in his group. I was flyin' high.
A couple of days later, I asked the question I'd been curious about since I got there but there was never anyone around who could answer. "What's the button for?" "What button?" "This big red one next to the door" I said as I pointed at it. I SWEAR that I didn't intend to touch it; the tip of my index finger just barely kissed the dome of the button.
KLUNK!
Every light in THE ENTIRE BUILDING went out. This was the emergency shut off for EVERYTHING, pre-dating the installation of the computer equipment and intended to be tripped only in case of fire. It took building maintenance about 6 hours to go floor by floor and get every circuit up and running again.
My temporary boss called my permanent boss who called me at home that night and informed me that not only was I no longer on the detail, I was not to set foot in that building until further notice. There were apparently about a hundred Revenue Agents who lost their cases (Remember, this was back in the days of dual-floppy computers without hard drives and saving your work meant deliberately pulling out a disk and inserting another) that morning and had to rebuild their files. Each and every one had apparently vowed to strangle me on sight.
Re:Using jed or emacs (Score:2, Funny)
Re:What kind of idiot... (Score:3, Funny)
Re:First Job Ever (Score:3, Funny)
How about completely wiping (near as makes no difference anyway) not one, but 70 linux-machines with a single silly command ?
BTDT. Went like this:
You guessed it. At the next round hour, spontaneously, every single one of the 70 linux-machines in 3 student-labs got the brigth idea of, essentially, deinstalling themselves. Hilarity ensued. Curses flew. It ended up a -LONG- day at work fixing everything again...
So there you go, one simple command, 70 bricked computers.
Any better ?