What Bizarre IT Setups Have You Seen? 874
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
Bizarre IT setup seen around the country... (Score:1, Funny)
Nothing THAT bad... (Score:3, Funny)
Router at the end of a pier (Score:4, Funny)
...it really is the answer (Score:5, Funny)
Seal it up (Score:5, Funny)
Re:Dungeon radio (Score:5, Funny)
Analog e-mail. (Score:5, Funny)
long ping next door (Score:5, Funny)
At one point, they had changed their routing so that they were using the new link but we hadn't, so we decided to see how a ping went.
A packet between the two machines would go through our router, over the ethernet that the two companies shared, out the (old) external router, and down the coast through Seattle, to California, then back up the coast to Vancouver, and then finally over the same shared ethernet cable that the packet had originally gone out before finally connecting to their router.
A cross-border round trip of a few thousand miles for a net distance of about 60 feet.
Oh, and did I mention that our server room was a converted bank vault?
some totally wicked "weird" security (Score:0, Funny)
Phone cables from modems (Score:5, Funny)
My direct experience... (Score:5, Funny)
About 1 month into the gig, I was in the front office which was connected to the computer room by fiber optic cable (probably the smartest thing this company did.) However, once the fiber terminated at the switch in the office, the horizontal wiring to the workstations was, God help me, silver satin cable. Telephone wire. The shit was everywhere. There were about 100 workstations salted through the plant (which ran high voltage AC and heaters and whatnot) and everyone complained about the server performance. I wasn't even allowed (!) to put a network analyzer on the wire and was too naive/stupid at the time to realize what the problem was. The guy who had the spend authority, the "chief engineer," told me the problem was lack of RAM in the server and was always harping on me to upgrade the memory.
Another time I opened a closet to find a splice of this satin cable (they must have bought it surplus, they had hundreds of reels of the stuff) and the splice was made with, I kid you not, wire nuts.
I lasted 18 months there. I heard they brought an ex-Accenture conslutant in soon after to fix the "computer problems" and she ran the company into the ground.
Re:long ping next door (Score:3, Funny)
collision detection? (Score:4, Funny)
Redneck Network Alarms (Score:5, Funny)
Sprinklers from hell (Score:5, Funny)
We had moved into larger building with a server room in the basemnent (cue ominous music).
We rapidly began to run out of space so decided to place the chief sysape in the basement near the servers, which made sense. We cleaned up some items in the basement, moved them into storage, carpeted, dry walled etc. Since it was in the basement it needed an egress window with a steel casing and ladder. This actually turned the office into a nice garden level. You could look out the window and watch the sprinklers, see trees and grass etc.
On day, the chief sysape comes in and notices water on the floor. He looks over at the egress window and there is about 2 feet of water collected in the base of the exit well.
Well, they shut down the water to the entire building. Luckily the server room actually had about an 18 inch raised floor, so no damage.
To make a long story short, upon investigation it turned out that when the sprinkler system was installed, instead of capping off the ends of the plastic piping, they folded it over and crimped it. They relied on the mass of the dirt to keep the ends crimped, and for years it worked. Until the egress well was installed and the dirt was disturbed. Once it was disturbed, the crimps began to fail under water pressure. Leading to a near IT disaster.
Newb Haxx (Score:3, Funny)
The most messed up LAN, evar! (Score:3, Funny)
The backbone was a five port AUI concentrator... it was too primitive to be called a hub. (AUI was Sun's insane proprietary ethernet connector.) Hanging off that was a Sun server that was shiny and new when the Soviet Union was still in the news, which was the router to the DMZ, and a media adapter for thicknet. That thumb-thick yellow cable snaked over to the engineering cubes and hardware labs, with "vampire taps" hanging off it everywhere - vampire taps have a screw that drills into the cable, which is how you hook stuff up to thicknet. No lie. These were connected to 10Base-5 thin-net adapters, which hooked up to co-ax concentrators, which hooked up to AUI media adapters which hooked up to the various Sun workstations. I had never seen before, nor have I seen since, a BNC co-ax hub used just to hook up workstations in a star topology... for whatever reason, they decided that ring topology wasn't good enough to string five lightly used workstations together. I have no idea why any of this worked. It usually didn't, and needed various pieces of arcane equipment power-cycled and jiggled and cursed at to get any data to make it from the file servers to the workstations.
It gets worse. Another port on the AUI concentrator went to the Cabletron TPT-2 setup, which took care of accounting, sales, support and executive row. This was like 10Base-T ethernet, with a patch-panel that was wired to RJ-45 jacks in the offices and the cubes, except it was completely incompatible with 10Base-T equipment. Media adapters for all! And when one of the adapters goes down, the whole TPT-2 system locks up, a hundred or so systems. Let's play the hunt-the-locked-adapter! So much fun when the CIO is screaming at you.
I went on vacation, and the engineers were left to figure out how to bring the network back on when one of these adapters froze. You'd think they would unplug the patch cords one at a time in the computer room until the network came back up, but no. They just remembered that I told them power cycling an adapter would usually bring it back online. So they powered down the building. Serious. They needed to reboot the building... by this time all the critical systems were on UPS, so nothing was fuxxored, but still.
I eventually got the penny-pinchers in charge of the business to invest in nice 100B-T and 10B-T switches and AUI adapters and a few nice new Sun servers. Worked much better thereafter.
reboot monkey (Score:2, Funny)
the L-Bend (Score:2, Funny)
A good example of lateral thinking [wikipedia.org]. Good thing they're cloning you.
Re:300 wires with a conduit sawed off (Score:5, Funny)
(Of course the reasons for all this are probably lost in the mists of time going back to fabric-insulated wires hung on insulators nailed to the studs. You'd think with modern wiring with obvious differences between 12 ga high voltage cable and cat-5e wires it wouldn't matter
Peace and quiet and a good cup of coffee... (Score:4, Funny)
Re:Sodomized service (Score:1, Funny)
Re:http://bash.org/?5273 (Score:5, Funny)
MS Exchange in place of a mail server (Score:5, Funny)
I've seen people try to use MS Exchange in place of a mail server.
Hey, you did ask.
Re:Phone cables from modems (Score:5, Funny)
Booting a PDP11 with no boot ROMs (Score:5, Funny)
Well now, remember I said it had no BIOS? What it *does* have is an octal debugger, similar to DEBUG in MS-DOS, called ODT. This is actually built into the microcode of the CPU; the CPU requires a console serial port to be present to even POST. If it's not there, a little LED lights on the edge of the CPU board and the machine will never come out of halt. So, at worst, all you need to do is hit <BREAK> type in the boot loader code on the terminal, and the machine will boot. Right?
Right. But that's a pain in the gluteus maximus, because it means typing in a load of stuff like and so on for a few dozen lines. There must be an easier way. What, like burn them into an EPROM? Well yes, but I don't have an EPROM burner. What I *do* have, though, is a VT-510 terminal, which allows you to program key sequences into the function keys. So, what I do now is power up the terminal and the PDP11, press HALT and then RESET on the front panel, hit a key sequence on the terminal, drop back into RUN once the disk seeks (controller is ready) and it's booted.
Yes, I'm buying an EPROM blower off eBay...
Incredible but true (Score:1, Funny)
I know it's hard to believe it but i saw it with my own eyes!
Re:Sodomized service (Score:5, Funny)
Re:collision detection? (Score:5, Funny)
Re:My direct experience... (Score:2, Funny)
Was this technology on display in the museum, or was it being used in day-to-day museum business?
Re:...it really is the answer (Score:5, Funny)
My tech knowledge is a bit rusty but if I recall we had a fairly bad firmware on our dial in modems / boxes which caused the winmodems to disconnect a lot (I know they sucked 7 -> 10 years ago but most ISP's seemed ok with winmodems 5 years back)
Anyhow I got tired of dealing with angry customers trying to get a reliable connection with their winmodems so I gave them a string which forced the modems to connect at 33.6 baud instead of 56k, I then set the string to report the PORT speed and not the modem handshake speed and bobs your uncle! Customers loved me "He got me a 57600 connection!" all the time.
Be damned if I recall the string but I think it started with AT....
Re:Sodomized service (Score:1, Funny)
But what surprised me most was when the same thing happened due to people *stealing* fibers along the road: they would spot a maintenance trap along the highway, cut the fiber here, drive some miles ahead to the next trap and fetch miles of fibers to go !
Re:Bedlam DL3 (Score:5, Funny)
Re:http://bash.org/?5273 (Score:4, Funny)
Re:qmail (Score:1, Funny)
The Curious Case of the Magic SCSI Clock (Score:5, Funny)
I went to his desk once trying to access a file on his external drives, and I kept getting disk errors. I called him over, and he said "Oh! That disk won't work unless you open up the system clock and resize it to this kind of size, and put it on the screen here". He opened the old analogue-face clock program that came with the amiga, resized it to about 200 pixels square, and stuck it in the top right of his screen.
I stood there smiling. He was, after all, a designer.
The file opened fine though after he did that.
I did some messing around on his machine afterwards. I was convinced there was some kind of obscure problem that we were missing - incorrect termination or bad cables maybe. I put the clock incident down to coincidence.
I could find nothing else wrong - but I still couldn't access the disk. So, I opened the clock application. I tried it on one side of the screen. File would not open. Moved it to the top right corner. The file opened. I did this about ten times as I couldn't believe the results myself. Every time I had the clock in the top right corner, the external SCSI disk behaved itself. I tried different applications, none of them worked in the same way - it had to be the clock.
I was completely spooked by the whole thing, and decided this was something sent by the Gods of SCSI to taunt me. The logical side of my mind believes that it is probably some obscure DMA issue, the rest of my mind believes the machine was possessed.
The thing I was never able to figure out was how Mark discovered the SCSI-healing properties of the Magic Amiga Clock and why he felt it was perfectly normal behaviour for his machine!
Jolyon
Where do I start... (Score:5, Funny)
2) The Netware 3.x file server which was a Toshiba T3200 plasma screen laptop locked inside a filing cabinet (a very secure solution on a military base). While I was working on it, a telephone began to ring in the next drawer up. I mentioned this to someone as nobody seemed to have heard it and the reply was "Oh, we don't answer that one"
3) The Olivetti M24 (AT&T 6300) that lived in a milking shed in the middle of a dusty field that eventually died and had to have a 2-3 inch layer of 'field' vacuumed out.
4) The computer room built with the existing radiators walled in but not turned off - took ages for the aircon guys to figure out why the room never cooled to the calculated temp.
5) The installation test of a new halon system (with a cylinder of CO2) where the engineers had not properly screwed the nozzle onto the 'j' pipe in the centre of the room. When the system was fired, the nozzle shot through the false ceiling, the gas followed it and the pressure blew down all the ceiling ties - the computer room looked like a scene from Die Hard.
6) The school network that comprised 5+ 'backbones' of 10Base2, each with around 20-30 D-Link *hubs* wired directly to cat5 outlets. Netware servers strategically placed round the building acted as repeaters with 2-3 NICs in each. We also found some Cat4 cable buried directly into the walls (no trunking).
7) 140m of Ethernet coax buried below a school field to link two buildings.
8) The over-length Token Ring network that included specially designed and developed repeaters that had to be 'tuned' using a screwdriver to adjust variable resistors to get the timing 'just right' so that the whole thing worked.
I have to add that I was *always* the support person brought in to sort things out - not the one creating the mess.
Distributed file storage (Score:3, Funny)
Once upon a time (+-1989), we had a set of some 50 Apollo workstations linked up via a Token Ring network. Not only did that ring have a habit of being physically broken every so often, the worst part was that there was no file server. Everybody stored his own files on his own machine. Project accounts were housed on the machine of the project owner. Nice and orderly, huh?
Well... except for the fact that there were people who didn't have a personal machine. Their data was initially housed on the machine of someone they cooperated most with. When disks filled up, new people without a machine would end up on whatever disk happened to have spare capacity. Then we (or rather "they", as I was there but not part of the IT gang) found that the amount of data people store over time outgrows the size of their disks, especially if you have shared project accounts. So ever so often, accounts had to be moved around. And sure enough, before you knew it the owner of the machine to which some high profile project had just been moved would complain that his box was overloaded doing other people's I/O. And just when that had been sorted out, there typically would be a reorganisation involving people switching offices or desks. Sometimes the machine followed its owner (not all were equally fast and some of them had black&white displays that nobody wanted to inherit), but most often IT would object to moving the boxes. By now the physical link between the data and its owner is totally gone. In the end, most people didn't have a clue what machines their files were stored on.
And now the fun really starts. We relied a lot on students and interns. In those days, if a student had seen a computer before, either it was a Commodore 68 or an early standalone PC. They didn't have a clue what the network was used for, so whenever they were done and went home, they'd physically switch off the machine they had been using. To make matters worse, even just keeping employee data storage away from the student machines was not an option, because there were not nearly enough student machines around. Typically, students would use the machine of an employee who happened to be out of the office on that day or during that night. Oh, what sweet memories... Not!
Not to mention the backup problems... Nor the fact that we also had a parallel experimental ethernet network with non-Apollo machines, of couse also without a proper file server. After a while, some data was being stored there instead. Now where the ... did I save that f... file last week???
Re:the U-Bend AKA trap (Score:5, Funny)
Re:Router at the end of a pier (Score:4, Funny)
It's DNS in France (Score:5, Funny)
One day, DNS went down. This had happened in the UK a lot, so we barked up the wrong tree for hours thinking it was a Keyring issue over the Transatlantic connection. Nope. Hours later, we found the DNS for France was on a different subnet. This led to discovering that their DNS service was on a set of IPs that pointed to one MAC. Finally, the people in charge of the data center said, "That's not our subnet. I don't know where you are getting DNS from.
We traced back and back through routers, entering territory that got scarier and scarier. It went to an older building that were were in the process of closing down and selling. It also had a data center, but that room had been dark for months, and DNS had been working up until now. Back and back we went.
Finally we found that the trace went through a disused subnet through a former office LAN in that building. This traced it back to an office, which traced it back to...
Turns out that when the French network architecture was being set up, they had to transfer DNS somewhere temporarily as part of a testbed, so some guy had an old laptop in his office he just hooked up. Then he was laid off before we went live. Nobody ever switched it back, and since the office space was being abandoned, no one every went into the office to turn anything off, figuring it was somebody else's problem.
A week later, French DNS was running on a production server.
I am impressed it lasted that long on such a platform.
We also used to run the flight schedules for Lufthansa. It was a Windows NT 3.5.1 system that was running on a 486, and was running some proprietary terminal service and scheduler. It crashed once every 31 days (there was some bug where it would crash after xxxx hours which was between 30-31 days). The only way to fix it was to hard reboot the box, and the directions were scary: "Go down to the older server room, and find an unlabeled shelf next to the first door near the panic switch. On the bottom of that shelf is a box which is behind a stack of old 10base hubs. Hold down the power button until the green light goes off. You may have to lie on the floor on your stomach to reach the button. Count to ten, power back on. Make sure the amber light labeled 'turbo' is lit on bootup. If not, repeat, but wait 60 seconds before powering back up."
I sure hope they got that fixed, it was last like that in 2000.
Re:Unsynchronized air conditioners (Score:3, Funny)
Re:300 wires with a conduit sawed off (Score:2, Funny)
Re:I hack, therefore I am (Score:2, Funny)
Re:the U-Bend (Score:5, Funny)
This is starting to sound like the introduction for the most boring Mythbusters ever.
"And then we waited for several weeks, comparing the rate of evaporation to our control toilet...."
Re:the U-Bend (Score:5, Funny)
Network Failure (Score:2, Funny)
So I ordered replacement parts and took the network switch out of the 19in rack into my apartment to replace all the components (thanks HP for their modular design).
On the way I encountered a panicking student, who knew me as the guy responsible for the internet, and asked me _WHILE_ I was carrying an insanely large box with a giant bunch of network connectors, "Do you know that Internet stopped working, like, 5 minutes ago?"
Re:the U-Bend (Score:3, Funny)
Re:Using the heat (Score:3, Funny)
remote monitoring (Score:4, Funny)
Re:MS Exchange in place of a mail server (Score:3, Funny)
I loved extended AT strings (Score:3, Funny)
ATZ0%F0&UC&K2O1F&F which by the way is (IIRC) a perfectly acceptable init string to reset the modem to chipset default and then turn off some reporting that nobody uses anyway. It might also disable one of the 56K compression algorythms.
Re:VHS backups (Score:3, Funny)
Re:Server room heating & worker Safety (Score:1, Funny)
The Sun freezes because it gets too hot?
Re:Sodomized service (Score:2, Funny)
If you ever want to find some buried fiber in Oregon just put a backhoe in a field. When you check it in the morning it will be pointed to the neared fiber.
Re:Where do I start... (Score:5, Funny)
So, I spent about $200,000 having actual ethernet installed and replacing all the computers in the (relatively small) company since everything they had was so ancient it couldn't even be connected to a contemporary network, set up a nice reliable server and backups, and after several months of intense work had everything running.
Then just as it was all stable, the boss called me into his office and explained calmly that our lease on the space we were in would be running out and he'd decided that we were in fact going to move, so I should plan the move of our network and equipment, bring in my wiring contractor to handle the new space, and ensure that we'd be back up and running in the new space in minimal time. Okay, no problem boss, when will we be moving? "In about half an hour." That's right folks, he didn't bother telling anybody that we'd be moving until half an hour before we did it, and I had just spent large amounts of money wiring a space we were about to move out of. And then for the new space of course you can't get a good wiring contractor on half an hour's notice, so all I could do was get a pile of long 10-base-T cables delivered and distribute hubs throughout the space and tape wires to the floor. I wanted to cry.
A few weeks later a psychotic middle manager who hated me because she couldn't understand what I did managed to push me out of the company and replace me with some kid who didn't even know what half the stuff I'd installed was, but he was willing to kowtow to her. I was terminated for "insubordination", for the unforgiveable offense of telling the kid that he couldn't plug the high volume laser printer into the UPS for the main server because it would overload the UPS and result in a shutdown. While the middle manager was gleefully screaming at me about what a nasty horrible person I am and that I was fired, the UPS was screaming from overload. I hear the UPS took the server down about 5 minutes after I walked out the door, and I knew offhand that that particular UPS, once it overloaded, would refuse to come back up until it'd had a (timed) 4 hour cooldown period. So, after the server I'd installed had been stable for a year, it died 5 minutes after I walked out the door and the new guy just couldn't make it go.
They'd also forgotten to ask me to tell them anything, like the admin passwords for any of the workstations, the BIOS passwords for anything, etc, which of course as a professional I would have been happy to tell them right up until they escorted me out the door. A week later they realized that they had hundreds of thousands of dollars of equipment that they couldn't reconfigure. They wheedled someone at the company I'd been friendly with to ask me for the passwords. I asked her "Did they offer to give you anything, like maybe a bonus, if you get the passwords out of me?" She said no. I told her that come to think of it I'd forgotten all the passwords since I didn't need them any more.
Re:MS Exchange in place of a mail server (Score:3, Funny)
Heck, I've seen some people actually try using WINDOWS as an OS in the server room?!?!?
Re:Bizarre IT setup seen around the country... (Score:2, Funny)
Re:My direct experience... (Score:3, Funny)
Possibly one of the best typos ever.
Re:Using the heat (Score:2, Funny)
Re:Coat Hangers (Score:3, Funny)
That's at least better than using coat hangers as power lines in JPEG. No compression artifacts.
Re:the U-Bend (Score:4, Funny)
Re:Where do I start... (Score:3, Funny)
Awesome. That could be straight out of a Douglas Adams novel.
Suspended floor (Score:2, Funny)
ADOT inherited a building in the easment next to the I-10/I-17 intersection near the AZ Fairgrounds that formerly contained some heavy cranes. These cranes were meant to lift heavy equipment onto trucks and were suspended on rails som 30 feet from the warehouse floor.
When the state inherited the building they decided to lease the downstairs to the Arizona Magazine for printing and assembling that fine pictoral magazine. The area in the rafters where the aforementioned crane resided was useless to them... so along comes the genius.
They used the heavy beams meant to support the crane as the basis for hanging a plywood floor. On this hanging / suspended plywood floor they would put in "Office Space" and lease that to ADOT for their IT development group.
It gets better.
Yes, all the electrisity and data wiring came into the building and was drapped across the gap between the wall and the suspended floor.
Yes the floor moved noticably, although it was too big for a single person to shift it on their own. After all there were 10 cubicles, 2 offices and a conference room on the floor.
Yes they put up pseudo walls separating areas of the room.
Yes, it was a warehouse so the ceiling and walls were just corregated steel... Yes it was in Phoenix Arizona... a dessert.
They did provide air conditioning so it was warm, but not unreasonably.
Another nice feature of the buiding was that it was partially beneath the I-17 N to I-10 E ramp that was about 60 feet off the ground. Every once in a while you would hear the clunk of someone Super Big Gulp hitting the metal roof, or the lite tap of a cigarette butt. At one point an ADOT truck in the parking lot was crushed by a truck tire that came off and went over the railing. Another truck was damaged by a water tank that came off another vehicle.
The best part about the IT solution was that they also put the servers up there. We had a separate room where all the servers were. It had extra air conditioning blowing on it, but it was not a contained room, it was firly open with walls that went part way to the cieling and a gap between the floor and the print shop below.
So the server room was in a metal building with no insulation in the dessert on a plywood floor suspended above a print shop under a highway were large things occassionally rain down. I shudder to think what would happen during a power outage.
Security incident (Score:4, Funny)
At an old workplace, there was a server (ok, a Sun Ultra 1, but it was running Oracle) which no-one seemed to know where it was, but it was on the network, running OK. I resolved to track it down...
First plan was to have it write something on the screen asking whoever saw it to call me. No joy; guess no-one went there.
Then I figured that it had a sound card & speaker - I also knew it would play .au files natively so went a searching and found a line from Monty Python's Holy Grail: specifically, "Help, Help! I'm being repressed". I then set up a cron job to cat this file to /dev/audio every 15 minutes. Unfortunately, all someone could hear was "help, help" from outside the comms room it was in and assumed someone was trapped inside. Security guard looks around and eventually finds the server with my name on the monitor.
At least we found out where the damn thing was, which was useful when some numpty builder cut the ethernet cable while working in the room.
Re:VHS backups (Score:3, Funny)
And then there's the old story about the pawn shop which used a VHS backup system. Why? Because they had gotten it from someone pawning it, and a bird in the hand and all that.
The payoff was when the pawn shop got robbed. They got a wonderful shot of the perp on their surveillance tape of him pulling the backup tape out of its recorder, then waving it defiantly at the security camera.
Re:the U-Bend (Score:2, Funny)
the pre 10BaseT days - what's ethernet?). All users had serial lines running to their 'workstations' with
dumb terminals. The several VAXs chugged away in the room, and the fire control system consisted of, er,
water.
things did actually catch on fire in those days too!
Long Runs (Score:4, Funny)
Another friend has the dilbert boss, decided that the router _needs_ to be at one end of the building, far away from the office. A distance of 350 yards, add even more when you take into account the fact that the wiring goes from the ground floor at one end, up above the second floor, spans the building, and comes back down to the ground. Then, he decides that 10 base T is too slow, and he demands gigE. My friend tries explaining to him why all of this is nuts of course, but the man never listens, and says, "just buy a bunch of repeaters." So, this gigE run of nearly 400 yards, is daisy chained together, every 100 yards or so, by a 4 port gigE switch. Most of them live up in the drop ceiling above the second floor, and have some ludicrous power lines run to them. Store and forward nightmare. Boss is currently pissed because it is slow, and because the gigE switches didn't make his 3 mbit cable modem go any faster.
Re:Windows on a ATM (Score:3, Funny)
10 minutes later as we were on our way to lunch, my colleague had to withdraw money from one of the brand spanking new wintendo ATM's that had just been installed. The ATM bluescreened while he was making a withdrawal (after it said - wait for your funds). So basically, he got his card back but the money never came. He checked the account and noticed that the money had left his bank account (without any money actually being paid out to him). He spent his entire lunch break arguing with bank staff about getting his money back while I went to eat. I never heard him say anything pro MS after that.
I find it incredible that - in addition to running Windows on the ATM, it had no concept of transactions.