What is the Worst Tech Mistake You Ever Made?

CraigEagle asks: "Mistakes are made every day. The more technical knowledge you have the bigger your mistakes can be. What is the biggest technology related mistake you have ever made?"

"In the interest of full disclosure, this is mine:

I was working at a Fortune 50 bank as a consultant. I was due to go on vacation for a week and the company did not have webmail. I decided that I would try forwarding emails to my corporate account. (I know this was a bad idea, and probably against several corporate policies.) I set it up so that any email that came in would forward to my consulting company's account. My mistake was I also left Delivery Receipt on. This was not Microsoft, it was Lotus Notes. The system began forwarding the incoming mail to my account. But then it would get a Delivery Receipt, which in turn would be forwarded to my account, which would generate another delivery receipt, ad infinitum. When I got back from vacation they claimed I had brought down the email system for 4 hours. This incident caused the bank to stop allowing consultants to set up email rules. What's your story?"

One time...

I launched SkyNet, right before my daughter and future husband rushed in to warn me. Boy was my face red!

easy...

staking mine and my family's needs in a technical career!

Re:easy...

Anybody remember those commercials on TV: "There are 50 thousand ice machines in the US and somebody needs to fix them." And then they'd offer to send you to ice-machine-fixing school. There are others, fixing/ driving the "Big Rigs", learning how to be an X-Ray Technician in 6 months, etc. etc. etc.

Well recently I've seen at least one each for "IT Manager", "Computer Technician", and "Internet Technologist"... "learn in 6 months!"

They might as well be playing "Taps" [west-point.org] as far as I'm concerned. Well not quite but those ads are ALWAYS for commodity positions, "anyone can learn", etc etc.

And if there are any ice-machine repair-folks reading this, there's nothing wrong with that, it's just that most of us have spent our entire lives deeply involved with technology, and we are used to our compensation reflecting that. Those ads tell us that people think they can learn in 6 months, and schools are filling up with people doing just that.

No they're not going to directly take our jobs if we have 10 years of experience, but all those folks sure will lower the paycheck bar for the entire spectrum of IT workers. It's called Flooding the Market.

Add that to overseas outsourcing... it's so depressing.

I used to console myself with the adage "You get what you pay for.", but way too often the people doing the hiring don't understand what they need or what they're getting anyway, they don't understand the benefits of paying for something better, so they go cheaper, time and time again.

There are 50 million computers in the US and somebody needs to fix them.

[buries face in hands]

File errors

When I worked for a library I noticed a log of files with red ! as their icons. I determined that they must be erorrors or duplicates. So I removed them. Turns out that in windows 95 a red ! means that it is a critical system file.
And the library did not have the system source media anymore so we spend the next day looking for any machines with a similar version of the deleted file and moveing them back by hand.

DB + Audit Trails

Back in the mid 80s I was a jnr op on an old mainframe. Not much disk space so we used to save old audit trails to tape and remove them. Another pertinent fact is the DB starts UDX* and the audit trails start UDXA*

I wonder what might have happened if a certain jnr op had not being paying attention and thought he knew it all.

Yep, there goes the audit trails and the database :-( God knows why they kept me around.

Posting on Slashdot

"What is the biggest technology related mistake you have ever made?"

Statement by Slashdotters after the supoenas start rolling in: "Posting an admission of wrongdoing on a semi-anonymous public forum, whose owners will most likely cooperate with law enforcement when asked about an admission of wrong doing in a semi-anonymous public forum."

Logging on to /.

My biggest mistake was finding this website. I've wasted more time here that could have been spent doing my job and getting actual work done.

Failing to face reality...

...and force others to do the same thing.

Never, never, never, never commit to a schedule that is not realistic. If you know it isn't realistic before you get started, imagine what happens when you discover the unknown problems.

No matter how much that guy in marketing wants to meet his roadmap, he will not help you design, code, or test your product. If you are lucky, he will complete the requirements before you are supposed to ship the product.

A small one

After working on a bunch of NT servers all day I needed to make a quick change to our company's primary file server which ran Linux. After I logged out I hit control-alt-del and the machine started shutting down. About 30 seconds later the helpdesk switchboard lit up like a christmas tree. After working on the NT machines all day I didn't even think before I hit the keys. I guess I thought I was going to lock the screen or something. 15 minutes later I had the linux boxes configured to write to a log file when ctrl-alt-del was hit rather than rebooting. Lesson learned.

Thankfully, that's the worst I've done so far.

Damning evidence

Recently, I went to a work-from-home arrangement. Soon after I left, I got a call from several people at work asking me if I had emptied several SQL tables that were key to our QA environment. I swore up and down that I had done no such thing, and it was dropped.

The next day someone powered up the monitor to my old desktop (still at the office) and what did he see?

SQL Query Analyzer maximized with:

DELETE FROM IMPORTANT_TABLE

Query successful: 18,452 rows processed

(I still don't remember doing it.)

Re:Damning evidence

This wasn't my mistake, but I guy I work with left a where clause condition of an update SQL statement and released the work to production. In development he was working on a system with a reduced data volumes and it updated a few stray records and he didn't notice. In test, the test cases didn't cause the SQL to run. In production it wiped out about 20000 records and had to full scan a very large table to boot. We actually found it because of the performance problem first. It took us a few weeks to reconstruct all the erased data.

My biggest mistake was in my first programming job years ago. I intentionally wrote an infinite loop into a program that was running on a very powerful (for the time) reasearch unix box used at the Naval research lab where I had an internship. It was a sonar imaging optimization routine and I would let it run for short periods (10-30 seconds typically) and then CTRL-C it to force it to stop and inspect the log file to find the results. I was new to unix and so I would use "ps" as opposed to "ps -aux" to see what processes I had running. I had multiple sessions up and managed to leave one of my programs running, switched sessions, ran ps which showed no processes running and went to lunch. The sysadmin was also a meeting and then lunch. When I returned I had a bunch of nastygrams telling me to kill my job immediately, not to run processes that hog the CPU because other projects couldn't use the system and to get approval before running long running jobs because the CPU time was billed (this was around 1985). I actually sat down, ran ps again, saw no job, and wrote back saying I didn't know what they were talking about. The sysadmin (who had returned from lunch) came over to visit me and educated me on a whole bunch of things.

Re:Damning evidence

One of my former coworkers always typed "# rm" instead of "rm" at the beginning of a long delete command, and only when he was finished typing would he go back to the beginning of the line, delete the leading "#", and hit enter. Were he to accidentally hit enter before he was ready, then he did nothing more harmful than send a comment to his shell.

I've adapted that idea to a lot of other situations; my SQL queries always start out as "-- delete ..." until I'm sure about what I'm typing.

Re:Damning evidence

Similar to the if (0 == c) trick in C, I've been trying to train my fingers to type DELETE WHERE whenever I mean to type "DELETE". Then, I fill out the WHERE clause and only then go back to say what table to delete from.

This also gives you time to ponder the wisdom of first running a SELECT statement with the same WHERE clause and comtemplate whether you want to do this.

rm -rf *

...five minutes later after coming back from getting coffee: D'oh!

I actually did this once... while logged in as root... at the top level in /home... on a production server. Thank baphomet for nightly backups!

Hopefully none of my clients are reading this. :-)

Re:rm -rf *

Hopefully none of my clients are reading this. :-)

You're known as FunkyRat to them!? :)

One that I saw...

None that I've done come to mind - I tend to make lots of little stupid mistakes rather than occasional huge cock-ups. But I had a client that had a CIO who was actively hostile to the idea of any kind of computer security what-so-ever. Waste of time and money for a made up threat he said.

They were running 13 servers at remote locations (and I mean remote, as in out in the boonies 4 hours from nowhere on back roads) and these servers were unpatched, had out of date or innactive anti-virus and were connected to the net via a combination of satellite and dedicated (always on) dialup. Their communications were secured with nothing more than Windows 2000's built in VPN.

Needless to say, my audit report told them that they had big beefy powerful angels on their side since they hadn't yet had a noticable intrusion. (They had no way of detecting one, but at least the servers weren't hosting porn sites.) I warned them that a virus or worm would come along though and knock the whole thing out. The CIO scoffed at my report, called me an alarmist and said that my opinions were right up there with the Y2K doomsayers.

When Slammer hit, I had described the vulnerabilities and outcome so accurately that this guy actually accused me of writing it myself. Took the whole corporate network down and they couldn't bring it back up until their techs visited each site. It took two teams seven days to get to all the sites. The company lost 6 business days, three customers and a months worth of transaction records.

Needless to say the CIO was demoted (they didn't fire him, which I consider itself a major tech mistake) and had me re-issue my audit report which they then followed to the letter taking every precaution I suggested.

didnt really hurt anything, but ...

When I was first learning linux/unix I installed RH5.something on my computer (cyrix 6x86 133+ iirc), anyway I was having weird issues with several programs so I decided i needed a fresh start, those darn dot files must be currupted.
So I typed:
rm -rf .*
This disk started churning ...
about 30 seconds later
the disk is still churning ... damn I must have a LOT of those dot files.
about a minute later .... *DOH*
CTRL-C
Where did all the files go? DAMNIT! I recursively deleted .. ( I was running as root, It was my personal box what could be the harm)
I learned my lesson very well:
CREATE AND USE USER ACCOUNTS!! DONT RUN AS ROOT IF YOU CAN AVOID IT!

blowing up my computer

I was young (around 8 at the time, can't remember) and I was bored one afternoon. I started fiddling around with the back of the computer, the PSU, to be exact. The red button looked fun to play with.

It was on 220v. I turned the computer on. It worked. Then I tried putting it on 110v and turning it on. Nothing. Then I switched it back to 220v, turned it on, and switched it to 110v while it was on.

Boom.

Moral of the story is, trial and error isn't the best way to learn hardware, and don't throw water on the smoking PSU while it's still live.

more fire and smoke

Last year a friend gave me a pentium 200 mmx that he coundn't get working. Since my parents were in need of a firewall I figured I would drop a couple nics into this box and build one for them.

The first thing I did was plug in a keyboard, monitor, and turn the box's power on to see if it would reach the POST.

Smoke started coming from the box, and soon open flame. For a brief moment I just stood there looking at it thinking, "That's interesting. First time I've seen a computer catch fire." Then I pulled the plug from the wall and the flames soon stopped.

I looked into the case to see what went wrong. It seems that the power supply connector for a floppy drive is roughly the same size as a speaker connector on the sound card. My friend had plugged the power supply into the sound card which seems to have caused the fire when the power was turned on. I suppose I should have checked for something like this instead of just plugging in the machine.

Really? I believe the opposite is true

The more experience I and my coworkers have, the fewer mistakes we make, and they are less severe than before. This is because we learn from previous errors, and apply those lessons to entire classes of problems, not simply the error itself.

I think the articles implication of "the more we learn, the less we think" is wrong.

not so bad -- but still stupid

So this wasn't a production machine I screwed up or anything, but I'm still a moron.

I had a Linux workstation that was ultimately adopted by the development group I worked with in the late 90's. Anyway, for some reason I needed to make a boot disk from an image. For some other reason, while typing in my command line, I was thinking fd0 but managed to type hda. So my line was dd if=/wherever/whatever.img of=/dev/hda.

Anyway, before looking at what I had typed, I hit enter. About 2ms later, I glanced up at what was on the screen and exclaimed something along the lines of "holy fscking shit!" and simultaniously hit a ctrl+c. Interestingly enough, the drive still kind of worked. I tried copying the contents of the disk over to another device, but I found that with each command - nay, each disk access, the filesystem would disintegrate further. I was able to save /home -- but I otherwise had to reOS the system.

I guess I've done much more stupid things with production machines -- but these were better machines, with storage on a NetApp NAS, which all had snapshots, so recovery was nearly instantanous.

These are not things that I include on my resume. (So -- anyone want to hire a disaster waiting to happen?) ;)

Playing soundfile on remote UNIX box

Oh man. So I was a grad student, right? I was always trying to portray myself as a very serious, dedicated student to my thesis advisor. And he had the fastest computer in the department (a Sparc10!) and he gave me permission to use it for batch runs. So I pretty much kept one of my xterms as a remote terminal to his machine.

Anyhow, one day I found this funny .au (sound) file and wanted to play it for my office mates. So I did a 'cat naked.au > /dev/audio'. Nothing happened. So I turned up the volume and tried it again. Still nothing. Then I screached in horror! I was typing this command in on the xterm I use for my advisor's machine! Sure enough, two seconds later an email comes trickling in from my advisor stating 'Please note that you are logged into my machine so your sound file is coming through my speakers.'

So what was this sound file that I had inadvertently played for my advisor?

Butthead: "Whoa! Naked chicks!"
Beavis (excitedly): "Yeah! Naked chicks! Naked chicks!"

GMD

Re:Playing soundfile on remote UNIX box

Geez, kids these days.

sally.au

You old-schoolers know what I'm talkin' about.

Oh, come on, old schooler, get with the program. This is the web, for crying out loud! That should read as:

sally.au [ibiblio.org]

Much better.

Warning, not safe for work ;-)

(A class 2 sarcasm warning has been issued for this post.)

FoxPro-based MRP & Bad Networking

I have two, both at the same company. I was hired on as The Tech Guy(TM) at a local OEM Manufacturer. They were willing to take me on w/limited experience but my pay reflected it. My first job was to replace their existing DOS-Cobal Manufacturing Resource Planning (MRP) software before the Y2K problems hit. The original software company wasn't supporting the old version any longer and the Mfg was getting too large for the program. They looked at a number of demos that were slow and painful, but they decided to try a "favorable" application (See also: cheap).

My mistake was to give the techie "thumbs up" under pressure. I folded to the "We needed this yesterday" argument despite my misgivings about the software. I paid for that mistake for the next year in slavish tech support. We became the software company's test bed as we found bug after bug. The software "worked", but operator efficiency dropped, and uptime was sub-optimal. "Customization" caused problems, etc., etc.

The second mistake I made was to attempt to use VPN over Broadband with Citrix MetaFrame. Although MetaFrame was a pretty secure and slim protocol for remote desktops, the Internet provider on the remote site had horrible latency problems and was run by a group of amatures. I should have stuck with the original Sprint frame relay proposal.

Morals of the story: don't let PHB push you into a solution you don't trust, and when network reliability is important, pay for assured quality of frame relay.

My Commander told me to kill the network

I was a young pup in the Army, during a training exercise. My Commander told me to kill the network, to "simulate" it's loss. We were operating a frequency hopping radio network, which of course is based on time. As the master node, I controlled the time. I pumped my transmitter to full power, and slowly pulled the stations that could recieve my signal out of time. Lowered power, pulled a smaller number of stations even farther out of time. Wash, rinse, repeat.

Commander thought I was brilliant, and so did I. I had fractured our network into at least 10 different domains. No one could talk to anyone, effectively "simulating" an enemy jamming attempt. It would take hours to restore the network, with many mad commo guys having to drive about with Pluggers, early GPS devices, to restore each radio to propper time.

Then a tank flipped. Someone died. No one could call for help. I am so damn smart.

No moon black, At 2 in the morning, in an upside down tank, the gunner figured out how to put his radio in plain text to call for help. It took him almost half an hour.

Re:My Commander told me to kill the network

It sounds like you did exactly what you were supposed to do. Isn't part of the point of a training exercise that the danger is real and freak accidents like that can happen? Maybe the real mistake was not having an emergency backup communication system (and the same would apply for a real conflict situation).

Re:My Commander told me to kill the network

You know, while that had bad consequences, it also isn't something that I'd call your mistake. Your CO asked you to kill the network, you did so. The issue that came up wasn't a technical one specific to the issue, so you weren't in a position where I'd expect you to be able to advise your CO using specialty information. This wasn't something unethical, where you might have taken another course of action, or illegal, where you should have refused to follow an order. You did exactly what your orders stipulated that you do.

Frankly, from the post, I doubt that the person dying could have been saved if the radio had been active.

Furthermore, unless the Army has rules requiring that personnel on training exercises have a comm system always up, I'm not sure that even your CO made a poor decision. There is such a thing as tragedy -- where everyone really did do the right thing, and someone still gets hurt as a result.

It's kind of like deciding to drive down Main Street in a town instead of Lambert Street and hitting and killing a kid that ran out into the street. Yes, had you taken the other road, the kid would have been alive. However, you can't be expected to or be *able* to always make the decision that produces the best outcome, or you'd be the best gambler in the world. The only thing that you can do is what seems the most sensible thing given the information that you have at the time. You did that, and nobody could ask for more.

Re:My Commander told me to kill the network

This sucks, but you did what you were told. And it is not your fault the tank flipped. Better simulate a loss in comms and have your men learn that they can send plain text over their radio, than lose 10 men in wartime.

Still, I'd feel horrible. But I wouldn't hold you responsible.

I flooded the building.

Not about computers, but: While working for a Physics research lab, I made a laser water jacket without a bulge at the end of the inlet pipe. The water pressure rose at night, the tubing slipped off, and the 2nd floor and part of the first was flooded, including expensive test gear like oscilloscopes.

Lotus Notes... er..

Working as a consultant I turned up at a new customer (moderate sized pharmacy) to see what they needed. Walked in, all confident, the local tech guy met me, and I asked to look at their server room (I always liked seeing the hardware).

Anyway, as we are standing there, I think, well lets see how many users they have, so I ask if I could look at the Name & Address book. Opened up the people view, hit Control-A to see the count at the bottom of the screen of the number of records. Unfortunalty it was a very small compaq keyboard, hit delete as I turned to the local tech..

Oooops.

I was doing phone support for a national bank in Canada. One of the problems we routinely had was a connection would freeze-up on a teller's terminal in one of the 1000s of bank branchs across the country.. We'd have to go into a program running on our AS-400 and reset the connection. On the odd occasion it wasn't just one terminal but serveral at the branch. We'd have to get all the tellers to exit out of their terminals for a second, then, in the program, we'd esentially hit the 'back' button, be up one level so we saw all the connections by bank branch instead of by terminal, hit 'backspace' to send the command to reset the connection and then 'y - enter' to confirm.

I got one of these calls, and I went one level up the tree, got distracted by something, and without thinking hit up-backspace-y-enter, going up two levels in the tree instead of one. This reset all the connections for the whole network, to all the banks, all across Canada.

Every phone in the call center started ringing. Every LED that could flash red did so. Everyone in the call centre looked around frantically. I looked at my terminal and almost died on the spot.

Not only had I reset all the terminal connection, but trying to bring them back online flooded the network so as soon as they tried to come back up they all went offline again. It took several hours to get things stabalized and the banks could start serving customers again.

Fortunatly my boss was a decent guy. He saw it as an accident and something that no one should be able to accidently do. The command to reset the entire network was modified so you had to type in your password to confirm, instead of just 'y-enter'

Worst...Tech...Mistake...Ever

(comic book guy voice)

By far, my worst tech mistake was dropping out of college to take a full time job as an outsourced computer admin. Not having my degree has kept me from being competitive for better jobs with larger companies.

I love job now, but I don't have much room to grow, being as I'm the top IT guy in a 70-person company that's family owned (and I'm not in the family). I'm working on finishing my degree now so that when the time comes to move on, I'll be able to find jobs that have room for growth.

Ahh, stories.

My best singke mistake was after a long night of re-installing an updated version of solaris on a SparcServer 2, I needed to clear out the /tmp dir sor some stupid reason. So, I did the old: "mkdir newdir ; mv * newdir"

I wasn't in /tmp. I was in /.

My next command was 'ls'. It returned: unable to find /usr/lib/libc.so.0

AAAAARGH!

I now know how to solve that under solaris. Under /usr/sbin/static there are 5 statically compiled binaries: cp, ln, mv, rcp, and tar. /newdir/usr/sbin/static/mv /newdir/* / would have fixed it.

Ever since then, my prompt has had my current directory in it. That experience certainly made me more careful.

Better (or worse) was when a stupid service rep came in to replace a bad CPU on a sun e10000. The idiot shut down the sub-system, and powered off the board correctly. He then managed to pull out the wrong board, despite the blinken lights. Of course it was the peoplesoft domain. Running year end reporting.

AAAAARGH!

I ruined a phone book

I work for a telephone directory publisher. A few years back, we were pushing a deadline and the man was not happy with the completeness of zip (postal) code info in the book. I purchased a new zip coding utility, ran it against the listings, and told the production dept to proceed with pagination, thinking that the army of proofreaders we have would notice any errors introduced by the new software.

I mean, what, I'm supposed to proofread the entire phone book by myself?

Anyway, the software used some kind of crazy soundex routine to "fix" addresses that it wasn't able to resolve, and thousands of people ended up with completely incorrect address information. The book went to press, was distributed, and a day later the phones were ringing off the hook. We had to pick up the old books, fix the data, schedule more press time (no easy feat), re-print, and re-distribute.

Total cost to correct was around $1M, got my ass chewed royally, but managed to keep my job anyway.

Must be doing *something* right!

Trying out Kmail

Trying out Kmail was my biggest mistake, because it had a different interpretation of the file OUTBOX than did my previous mailer. My previous mailer stored every email (6 years worth) in OUTBOX. And kmail took OUTBOX to be the file where messages written offline were temporarily stored until next coming online. The first time I fired up Kmail, a indeterminate-time progress bar came up, and it kinda hung. I went to get a coke, giving it time to snap out of its funk. Unbeknownst to me, during that time it re-sent every email I'd ever sent. When I got back and checked my INBOX, I screeched in horror.

Funny thing is, people from my previous job were getting work related emails from me again, and they didn't seem to mind that (1) they were on outdated topics and (2) the company was defunct, they played right along and replied stuff like "yeah what ever happened to that issue?".

Click here!

Well, this is more of a graphical error than a text error, but it's still amusing. My company developed a technology where you can watch video from mulitple angles. (note: this is going back a few years.) So we were pumping out demos like mad. At one point, we got some stock footage of a horse show or something. It had a horse jumping over a fence, filmed from different angles. I had to insert the words "click here" at the bottom of the video because I was going to make that clickable. If you click there, then you get s'more info about our software.

Back then, we didn't do letterboxing like Media Player does. If the window you play the video in isn't the same as the aspect ratio of the video, then cropping occurs. I did not consider this little fact about our player, rather I got it up on the site as fast as I possibly could. Then, I went to lunch.

When I got back from lunch, I noticed the CEO was looking at the demo. So I poked my head in to say hi. He says "Why is this video telling me to lick it?" Wha? I go up to the screen, look at what he's watching, and... eep. The c in click here was perfectly cropped out of the shot. I mean perfectly. I mean you didn't know it was missing. So here's a horse, reared up on its hind legs, with the words "LICK HERE" just below its.. uh.. tail.

I am so glad that we had the one CEO in our industry that understood what took place.

Very Enterprising

As a joke, I once set the transporter to low resolution. The Captain was not amused.

I was coding the Linux kernel...

I was coding part of the Linux kernel, and thought to myself "Am I reinventing the wheel here?". So I looked around on the internet, and found some SCO code that did what I wanted to do, so I just copied it and submitted that to Linus. He accepted it and put it into the kernel.

Boy, do I feel stupid now.

Oh geez.

1. I was working on the development database but my boss needed a quick count of a number of checks so I opened a new window (Query Analyzer for SQL Server) to the production database and gave him his count. I then proceeded to finish what I was doing on development... without switching windows back to the development server.

TRUNCATE TABLE Checks

TRUNCATE isn't a logged option but thankfully Log Explorer Pro from Lumigent can retrieve truncated data if you move fast enough. As well we had a backup that wasn't so very old handy. Out of 1.3 million checks we only lost 34000, but I was so stressed out.

2. Way, way, way back when we had just gotten a new Dell server. I was showing an interviewee the server who I had found out I had known when I was younger. So, joking around I said, "Want to see a hot swap of a drive?" He was like, heh, that'd be cool. So I pulled the drive out of the RAID 5 array. Alarm klaxons started going off from inside the machine, I swear. I stuffed the drive back in but even though the drive officially -was- hot swap we hadn't purchased the high end Dell with an array controller that could dynamically rebuild the data. We'd gotten the cheap version. 8 hours later - with the machine beeping constantly at us - the rebuild was done.

3. This one's not mine but a guy I work with. I had asked him to migrate some databases to a backup server so he set up a DTS job to do the migration. Unfortunately he did two things wrong: the destination was the same server as the source, our primary production machine, and he set the DTS process to execute nightly instead of once. We ended up filling 300Gb of drive space and not having a clue as to what happened to cause it. When we found it we were giggling (it is funny :)) but beating ourselves in the head.

4. Another one that's not mine. New network administrator was installing Windows NT 4.0 (this was ~6 years back? Roughly?). He was complaining about it taking forever to install and I asked him what he was doing. "Well, shit, NT has like 35 disks man." I asked him why he wasn't installing off the CD and he just hung up on me. He didn't know the NT CD would allow you to do that. :)

5. On a similar vein my original boss when I started here was I thought a technical God. It's fun to see how that belief fades over time. In my case he was showing me how to install Netware 3.12 and configure it the way he wanted it to be configured. He sent me off on my own the next week to install a new office. The week at home I had burned all the Netware 3.12 files to a CD so I wouldn't have to cart around all those floppies. Apparently the load time off CD blew my boss out of the water because he didn't believe I'd installed the server already when he called to see how things were progressing.

6. I'm walking my COO through hooking up a new modem in our Kansas City office. He's getting mad at me and asking me if I know what I'm doing because we can't get a response from the modem. (I'm working blind over the phone.) I had asked him earlier if he had hooked up all the cables like they were to the old one and he had indicated that he did. Finally I said, "Look, don't take this the wrong way but let's check the cabling. You should have a phone cable to the wall, a power cable to the power, and an interface cable to the computer. These should all be coming from the modem." He had forgotten to hook up the RS-232 cable. To this day I razz him about modems telepathically communicating with machines.

7. My CEO is one of the brightest people I've ever met in my life and has my eternal respect for his intelligence and moral integrity. He called me and indicated he couldn't print. I told him to not get insulted but I was going to start with the basics. "Is the printer plugged in?" "Yes." "Is the power on?" "Thanks Brian, I'll call you if I have any more problems."

8. I had just come off the road from setting up our Texas operations - a 4 mont

Live deletes

I was in a telnet session (and forgot that fact)... in E:\WINNT...

What the heck are these files doing on E: on this machine? Fsck! Ok... let's delete them...

Sudden realization that it wasn't local after all.. it was our main server for the ISP we ran, 25 miles away!

Hopped in car, had to reinstall, got it back up and running about 2 sweat filled hours later.

Moral: Always be mindful of WHERE the command is running.

--Mike--

The very first Fiber run in Phoenix

This was told to me by the former supervisor for the phone systems in the AZ/NM area.

The very first fiber run in Phoneix went from one federal building to another. I'm not sure which, but they must have been important.

If you've ever seen an phone cable room underground, you know that the cables are straight, so straight that you can easily follow them across the room and usually clearly labeled. Well some dumbass manager went down into this one cable room underground in Phoenix, and saw this great big looping yellow piece of shit cable run and wanted it fixed pronto!! So he gets some new hire (been on the job less than a month) to go down there and I quote "Fix that Fu**ing thing! I want it to look just like the rest of the cable down there, and I'm gonna get the guy who installed it fired!!" (yes, he does come off as a jackass doesn't he?)

So this poor newbie goes down into the manhole and starts hammering, and tying down, this 'cable' run. He's using pliers, 3 pound mauls (why won't this stuff stay flat?) and whatever else he could do and wouldn't you know it, after 4 hours or so of this, it looks beautiful, just like the rest of the runs and even re-labeled!

Well, when this guy pokes his head out of the manhole, there are like 20 officers from the FBI, State DPS, County sheriff, ATF, and whoever else waiting for him with guns drawn!!!! Poor guy is fired on the spot and questioned for over 2 days, telling them he's not a sabateur and that his boss told him to do this. The boss doesn't fess up until the 3rd day of questioning, at which point HE is fired and the pleeb gets his job back.

The second first fiber run in Phoenix was back up shortly, and the other workers educated about it's "don't take a hammer to this shit" properties.

--
This sig writes better than I do.

My embarrasing moment...

Since I am an impatient guy, I wanted to make my external USR Sportster 33.6k modem to dial faster with initialization string parameters.

Well, it was the middle of the night (3 am?) and I was a teenager. I made the modem dial fast, but one of the BBS phone number started out with 914... Well, the modem accidently dialed 911. I didn't have the modem speaker loud enough so I didn't hear 911 operator. Then, a cop came by after a few minutes. My folks weren't happy that day.

DOH! :(

NET SEND

While working at HP I did a NET SEND command to get whoever was logged into one of the servers I was using to log out of PCAnywhere. Unfortunately, I missed one of the parameters and sent the message to everyone in the login domain (ie. a few thousand users).

After hitting ENTER, I hear a hundred Windows 'dings', and everyone in cubicle-land starts prairiedogging. I got a few nasty replies asking who I was, and a very nice one saying "Don't worry: once I sent 'I know you don't have any pants on' to most of HP Belgium".

Worst thing was, the guy clogging up the server was my cubicle-mate who'd gone out to get coffee.

Linus's big mistake

This tidbit from Lars Wirzenius [liw.iki.fi] is a part of Linux Lore [liw.iki.fi]:

"Linus also got some other stuff via mail. For example, a pair of 40 megabyte hard disks. That was really nice, since it meant that Linus was finally able to keep some backups. Not that he did, of course. One of his well-known quotes is: "Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it." He said that even after dialling his hard disk.

"At one point, Linus had implemented device files in /dev, and wanted to dial up the university computer and debug his terminal emulation code again. So he starts his terminal emulator program and tells it to use /dev/hda. That should have been /dev/ttyS1. Oops. Now his master boot record started with "ATDT" and the university modem pool phone number. I think he implemented permission checking the following day."

I once did something similar -- I was going to back up my MBR to a floppy. Using the 'dd' utility. I got the command line options backwards, and overwrote the first 1.44MB of the hard drive with the contents of a blank floppy disk. Required a low-level format of the hard drive to reuse the sucker. Thankfully, it had no critical or irreplaceable data on it.

Brought down the house, so to speak...

I worked for a fortune-100 company as a UNIX admin/general systems geek.

We noticed that one of the filesystems that held the log files for an Oracle Application Server (two machines, shared storage) was filling up.

At this company, the security wannabees gave no one root access, but gave sudo privs to all UNIX admins. No big deal, huh? Well, they gave permission to everything in /usr/bin and /usr/sbin, save su to the admins. You can do things like 'sudo chown' and 'sudo rm'. Psssht.

Anyhow, my boss asked me to clear out the rotated logs in an attempt to free up some space.

I logged on to one of the two boxes and went to the directory in question. I typed "rm *.*"... Permission denied. Bummer. I guess I'll have to use sudo.

I typed in 'sudo chown [myid] .' and then the previous command. After some time, I got my prompt back. I did a quick 'df -k .' to check my work and noticed that the filesystem was WELL within acceptable limits. I was so pleased with myself (and shocked by the tens of gigs of rotated logs) that I went to tell my boss that it was taken care of and to state my amazement at the amount of space that was being taken up.

I got my 'attaboy' and continued working.

After about an hour, we went to lunch (boss went to lunch with me almost daily.) He gets a call on his cell from the PHB (although, to be fair, 'balding head boss' would be more appropriate.) He said that the OAS cluster for the largest app we supported was down.

After about 30 minutes of investigation and head-scratching on the part of my teammates still at the office, my boss got another call. One of my teammates asked him "who is [my id here]?"

My boss asked me if I knew, and my heart nearly exploded. I told him it was me.

I didn't even think to mention the change I made as a possible cause because so much crap happened every day that I forgot about one project about 5 minutes after completing it. I always fess up immediately when I make a mistake, so my boss knew I wasn't trying to hide anything...

Apparently, the server crashed when it had to rotate the log file (too large) and couldn't write to the directory. It wouldn't come back up again (with a completely non-descript error message, of course) after the crash for the same reason.

I'd left the directory permissions set to my user id. D'oh!

What makes this funny (in that sick kinda way) is that this app server crashed constantly, and the higher-ups tried to make themselves look good by being concerned (even though no business loss was actually incurred.) They always wanted a root cause analysis for every crash, and they were all the same - "unknown. vendor support not available because software is past end of life."

The higher-up jumped on this opportunity to make a freaking "oh my God, this guy is so dangerous" case out of it because it gave him something concrete to go to his higher-ups with, after so much "idunno" action.

I was given a written warning (my boss was forced to do so.) He smiled and laughed with me over the stupidity of it.

Business degree

I got an MIS degree instead of a Computer Science degree. No idea why.

When I started my current job I was assigned to testing one of our apps. I was given 2 databases to log into and was told to "exhaustively test" the entire system (adding/deleting records.) So I started adding all sorts of odd stuff (I was going to delete them anyway.) Now I'm not dumb enough to name them dirty things (afterall this was my 2nd week) but I was still typing silly things.

Anyway about an hour into it I got an email that was addressed to the whole department that said

"To whomever is modifying the market database while I'm trying to demo it to one of our most important customers: not funny."

Re:rm

Atleast you did it in the wrong dir. I was working on adding another domain to /etc/apache/vhosts/(each domain gets its own file), and all of a sudden I notice the dir is empty. It wasn't empty when I started.. scroll up 4 lines and I see rm * in my buffer. To this day I still have no clue what I was intending to do, but from now on no more working on a server with customers without proper amounts of caffiene.

Re:rm

since then ... aliased rm to 'rm -i'

Bad plan. Now, the next time you log into a new machine you'll think that rm will be safe and will wipe out an entire directory tree again.

If you want to have a safe alias, use a different name! For example del would be appropriate. If you're not good enough to use rm correctly, then an old DOS command seems appropriate...

Re:rm ... I did this once

I don't see how you can call that a mistake. Thats more like quitting cold turkey.

Suggestion to avoid accidental "rm"s

Aliasing rm to rm -i will do nothing if you use the -f flag, as you did. -f overrides -i.

However, accidentally separating a wildcard from text is an infrequent mistake that can cause much pain. For example, typing rm -rf * .jpg

Zsh, by default, will complain at you and ask you if you *really* mean it if you use a bare wildcard with an rm command. Invaluable, and has saved my ass a few times.

try it from a cronjob

rm -rf from the command line? Lucky bastards!

I once added the following to a cronjob
rm -rf $foo/*
My intention was to wipe contents of a directory that I was reusing. Unfortunately "foo" was unset. The cronjob ran overnight with rm -rf traversing every NFS mounted drive in the company. I remember coming in at 10 the next morning and thinking "christ what kind of idiot deleted all of my files?", and then "shit! that idiot deleted everyone's files" and then "shit that idiot is me!".

Ever since then I usually do something like
rm -rf ${foo:?}
mkdir $foo

Later as I recovered my composure I started thinking "Now why can't those idiots set their umask correctly?".

The only positive aspect of what happened was that it revealed a weakness in the backup procedures being following by the IS department.

Personally I count my self lucky to have had the benefit of such a humbling experience w/out loosing my job.

Re:rm

Along the same lines. On a customer's precious and mission-critical machine. Intending to copy filesystems to new volumes I'd created, I started with:

mv /usr /newusr

I managed to get it stopped, but not before /usr/bin and /usr/sbin had moved. Finding any useful commands still functioning to let me assess the damage and fix it was interesting. I needed to do something like this once to learn, deep down, that everything is different when working as root.

~~~~~~~