What Bizarre IT Setups Have You Seen? 874
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
300 wires with a conduit sawed off (Score:5, Interesting)
The fire marshall came in and said "you can't have those low-voltage wires run through that conduit, that conduit is designed for high voltage wiring." So the electricians came in and sawed off their beautiful conduits, leaving the wires draped between the four-foot-spaced supports. They tie-wrapped the bundles every foot or two, but it still looked like a dead python hanging between branches.
To this day I still can't fathom what the hell that inspector was thinking.
Dungeon radio (Score:5, Interesting)
Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.
Coat Hangers (Score:5, Interesting)
Re:Server room heating & worker Safety (Score:5, Interesting)
After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).
This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).
Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.
I hack, therefore I am (Score:4, Interesting)
There was the time back in the late 1980s when the multiplexed underground cable that my college was using to provide terminal connectivity for their new-fangled computerized class registration stopped working, and they ran dozens of hastily-created RS-232 cables from the data center to the hall where the action was, half a block away... secured to the sidewalk by duct tape, of course. Which at least didn't remain in place as long as the 10base2 cable that connected two dorms, strung between their 2nd floors (until it was taken out by a lightning strike).
More recent ugly hacks that I can claim personal credit/blame for are mostly of the sort that involve pulling a rabbit out of my ass because a solution needs to be found By Tomorrow Morning... like for deploying 200 installations of Windows 95 in a week (in the days before Ghost, or even backup software that preserved Long FileNames) using DOS boot diskettes, Netware, a utility called lfnbk, and ncopy... or building an e-mail server out of RedHat Linux 6 and spare parts (no, I didn't even have a complete working computer at my disposal) when the company's glorified BBS mail software found itself unable to exchange mail with the standards-compliant system used by a major new business partner.
Re:the U-Bend (Score:5, Interesting)
Yeah, and it's a huge hassle in vacant commercial buildings. Somebody needs to run every water tap and flush every toilet about once a month, or the whole place will stink up. Then the smell gets into the carpeting, which makes it hard to rent the building.
For special situations, there are calibrated drip valves. These are often found as part of fire sprinkler systems, which usually have a drain valve for when you need to drain the system for maintenance. The water from the drain valve has to go somewhere, which usually means a sewer connection. But you can't hook a water line to a sewer line; there are situations when you'd suck sewerage into the water system. So there has to be a vacuum break open to air. After the vacuum break, there's a U-trap with water to keep sewer gas inside. But since such drains are seldom used, the water will evaporate. So a tiny bit of water has to be dripped into the drain to keep up with evaporation. There are special "drip valves" for this.
One of the things you need to know about if you run large data centers.
Stupid IT maneuvers (Score:5, Interesting)
1. A "server room" that was essentially the most worthless room in the entire building, a long skinny room with four windows (perfect for keeping an uneven temperature!). Rather than buy 19" racks or even wire racks, they found a bunch of tables and put one server on each all the way around the edge of the room.
1.a. All of the servers were in fact desktop systems; an Ultra 1 was the mail server, a SPARCstation 5 the print server, a Gateway Pentium Pro 200 desktop the web server, etc.
2. A lab had to be moved one room over, because its current location was deemed too valuable. The original room was designed for a lab, it had 20+ fiber optic networking ports, twist-lock power connections in the ceiling, that sort of thing. The new room had two electrical outlets, no dropped ceiling, and one fiber optic networking port. It had previously been used as a copy room/storage closet. The cost to move the fiber optic wiring (just one room over mind you!) was over $25,000.
So instead, I had the great idea to cut a hole in the common wall (above the drop ceiling line), purchase additional ceiling tiles and cut up 2x4's into wooden supports. The original ceiling boxes containing the networking were put on top of the blocks above the new tiles, and extension cables run through the wall into the new room. In the original room, which was turned into a lounge, you couldn't tell that there was anything funny going on.
The best part is that the lab manager, who insisted they needed every single network port, never used a single one of them in the new room. All of those cables now reside in a box marked "Giant waste of money".
3. The main Windows file server was purchased in 2002 and has an internal RAID (bad idea in my opinion). What was huge then is worthless now; 5 disks that total 135GB. To get more space, the administration begged for a single external 250GB USB drive to host all user data. Nevermind that there is no redundancy, that an external drive is more suspectible to theft or failure, and that USB is unnecessarily slowing things down.
4. A system administrator got it into his head that rackmounting was the way to go (I agree). So he begged for a 19" rack to be ordered, and placed all of his servers into it. Except he doesn't have a single rack mountable server, and he didn't get the rails for any of the cases either. So now he has one $500 rack, and 8 $100 shelves to go in it. Same guy also switched the KVM monitor to a 15" LCD that doesn't support the resolutions of 9 out of 10 systems connected to it.
5. A consultant was brought in to tell us what needed to be done with the computing infrastructure (what DOESN'T need to be done is more the question). His main suggestion was to set up a central backup service just for this college, so as to avoid paying the central university IT group fees to use their central service. OK, thats an idea I guess... except that he wanted us to buy this: http://www.sun.com/storagetek/tape_storage/tape_l
Basically every day is a new adventure in ridiculous IT methodology.
Re:the U-Bend (Score:5, Interesting)
Some amount of cogitation was needed to realize that each of those small pipes was headed to the traps of the floor drains installed throughout the store...
Now that's a (plumbing) hack in the true meaning of the term!
hacks? (Score:3, Interesting)
Hmmm
How about a Netware 3.x server stuck in a closet between two 10base2 (coax) runs, connecting one segment with another (glorified IPX router).
How about a 395 foot run of BNC
How about a 450 foot run of BNC with old, frayed screw on ends. That ohmed out at about 76 ohms. And they wondered why the network was slow.. (I re-crimped all the ends and cut the wire approximately in half, and used a second NIC in the seerver). In that same place, one of the workers figured out that if they took one of the BNC connectors off the T in the back of the PC, the network would go down and they could just sit there and do nothing...
The company that had a 1000 foot run, so instead of buying ARCNet wire
The BNC wire I saw that someone had repaired with a paper clip and electrical tape
And we won't even talk about how many networks I ran into that looked like
Server ----- hub ----- hub ----- hub ----- hub
and they wondered why the people on the 4th hub would lose server connections randomly..
2 SUNs == remote control (Score:5, Interesting)
Once the graphics cards were removed, the machines defaulted to booting with Serial consoles. This meant that if anything went seriously wrong, just about anything other than hardware maintenance could be done by SSHing to machine X and using a terminal program to connect to the console port of machine Y (or vice versa).
This included the ability to do a complete wipe and install, needing only to instruct the CoLo staff to insert the install CD (which were left on top of the machines) into the appropriate box.
One of the monitors ended up on my desk. I can't remember who got the other one.
TV in disguise (Score:5, Interesting)
As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants
quick, not so dirty... but quick (Score:1, Interesting)
There was an identical PC chassis collecting dust... I took that, installed the impressive quantity of 72 mb of RAM in it (maxed!), a new drive; two identical Ethernet cards; installed NetMax "firewall in a box" on it; transferred e-mail and web service to a vendor we were working with (the other unsung heroes of the story) and in about 72 hours after propagation, all was well. Total expenditure on this "rescue" was about $250.
This day-and-night effort to put the network back on its feet earned me the undying skepticism of the CEO, who postulated that I had somehow caused the old setup to fail in order to justify new purchases. So it sometimes goes. YMMV...
Re:the U-Bend (Score:3, Interesting)
bad ethernet habits die hard (Score:2, Interesting)
So I got a data cable guy in to quote a clean installation-- home runs back to a single switch, cable in conduit, all clean and reliable. It was a few thousand dollars.
The owner decided it was too much $$, so he got an electrician to do it and now he has 3 more switches and even fewer home runs than he had before. But it's now Cat6, so he somehow thinks it's better.
I've already been back to troubleshoot... bad cabling.
Crap "servers" overheating? Rig some crap cooling (Score:3, Interesting)
The machine with the user accounts on it had a few more hot, high-speed disks than the case was really designed to keep cool. It got hot and beeped. My boss wouldn't consider replacing it or even getting a new case. So I was forced to improvise: I cut a hole in the front panel and fitted a spare case fan into it. Then I realized that the motherboard didn't have another power connector for the case fan ... but I had a spare 5V wall-wart. A little wire-cutting and electrical-taping later, I had an externally cooled disk bay.
That "machine room" sucked. It was in the corner of the basement of a college office building. In the winter, the (crappy, household-type) AC unit iced over and the servers overheated. One summer, the facilities staff decided to power-wash the wood siding of the building. High-pressure water ran up through the wall and rained down right onto the server shelf. The only thing that blew up was the fancy new monitor that had come with the expensive and utterly overpowered RS/6000 just purchased by the library.
A couple of years ago when I visited the campus, they were still using that wall-wart-powered fan to cool the disks ....
Network closet A/C suction (Score:4, Interesting)
-b.
Re:Phone cables from modems (Score:5, Interesting)
Only it doesn't. It round-robined the requests, so 1/3rd of our traffic was immediately and swiftly rerouted to our emergency site, which some enterprising webmaster had setup to email the webmaster box if anyone hit it (to make sure, I guess, that no one was going to it).
We noticed it because 5 emails came in at once, and then 10 more, and then it didn't stop until Groupwise crashed. We lost all the email in the box, and emails were coming in at some insane rate. We figured it out maybe 3 minutes in, but by the time we logged in and made the change, it was way too late.
Re:Nothing THAT bad... (Score:3, Interesting)
When I started working at a local high school administering their network I found an amalgamation of two networks. One running primarily thinnet (10Base2) through the classrooms uplinked to 10BaseT hubs (three 16 port stackables) which were each connected to a 10BaseT switch ("the core"). The server ran Novell, the PCs ran a combination of DOS and Windows 3.11 which ... worked.
Now the new network installed right beside it consisted of a mighty IBM NetFinity 5500 server with a RAID 5 array of about 50GB and a plethora of IBM switching equipment. The core switch wasn't high-tech or anything, but atleast it had 100MBit fibre running to the IDFs which ran switched 10BaseTX to the workstations. Now, this was all fine and dandy and wired real pretty-like by people who had no comprehension that the labs running PII-400s would most likely place more demand on the network than the labs running 486DX4s, so we had to re-wire the thing to balance the loads somewhat. :)
Shortly thereafter we installed a Linux server to handle DNS and HTTP cacheing for the 128KBit ISDN connection to the Internet (real practical for a network of some 400+ workstations eh?), revamped the configs on the workstations, re-configured the network from the core on outward, re-wrote the network wiring diagrams manually (they were, apparently, somewhat classified "need to know" information and we as system administrators did not "need to know") and generally made the place hospitable.
The network where I'm currently working (along with the phone system) was apparently installed by a monkey. All 19" rack-mountable equipment with its rack mount hardware installed in such a way as to be able to bolt them flat alongside three inner walls of a closet. There's a nice Panduit patch panel there - with about 4' of patch cables tempting their connections by gravity; they just sortof hang there in a loop before connecting to the Cisco switch installed above it. Not so much as a zip tie in sight!
There's a 3Com 16-port switch in there that was powered and creating plenty of heat and noise; but the strange thing is it's not connected to anything but the AC outlet. (Yes, it's now unplugged, but still hanging there all useless-like). I also find myself at a loss to explain why, with a single ADSL connection to the building, we require three (yes, three (3!)) DSL modems. Or why, when there were 2 spare electrical outlets even before I unplugged the 3COm someone felt the need to connect one of the devices to an extension cord running out the closet and 5' along the wall.
The network drops consist of a motley combination of mis-labelled jacks and broken wall mounts compensated for by the random installation of cheap hubs and duct-taped CAT5 cables running helter skelter around the place.
The network is so shaky it's not possible to install a centralized high capacity network printer as of now because, well, too many print jobs and something could catch fire in that closet. I can't WAIT to write up a cost benefeit analysis for my boss to justify the disposal of the dozen or so laser printers installed on various desktops around the place. :)
Oh, and these aren't mine [snerk.org], but they make me feel better about my own situation whenever I look at them.
Space Station Video Hack (Score:5, Interesting)
Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.
The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.
Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.
So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.
Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=1879
Re:Dungeon radio (Score:5, Interesting)
Speaking of radio:
A few months back during the summer, I was enjoying the lovely AC of a rather large server room at my university. On top of a rack in the midst of the farm sat a much abused radio, likely discarded even by the janitors, splattered with paint, and employing a rather frightening extension to its antenna. Its tuner was taped into position, and its headphone jack was connected to one of the machines.
This, of course, supplies the world with a live stream of the campus radio station.
LAN parties (Score:2, Interesting)
The cable remained there for many months - rain, hail and shine until this guy moved to a house directly across the street. There was talk of running the CAT 5 across the street alongside the power lines, however it never happened.
On the work front, once I visited a client complaining of slow network performance. Following the cables, I popped my head up into the false ceiling to find a roll of 100m of 10B2 cabling.
Continuous wiring (Score:3, Interesting)
I recall installing some additional power outlets in one building in the tropics and worrying about violating code by doing it myself. The building was wired with 8-gauge multiple strand copper wire (for 110VAC 60Hz) with one hot, one neutral and (ostensibly) one ground. Once I got a look under the faceplate, however, I realized that code wouldn't be a problem.
The wiring method was bizarre: at each terminal (screw) the wire was stripped of insulation by tearing for about 2 inches (~5 cm), wrapped around the screw and then continued on its way to the next. It went like that for all the outlets we examined.
Re:Server room heating & worker Safety (Score:3, Interesting)
Re:Coat Hangers (Score:5, Interesting)
The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless.
Far East, right? For some reason, they have the same cultural aversion to proper wiring as Middle Easterners have to proper plumbing. [shaking head in belief but oh-my-God-what-can-you-do. At Home Depot, they refuse to sell stuff which the "associate" thinks will be used improperly where customer safety might be compromised. "No sir, 18-gauge lamp cord is not suitable to feed your dryer... Yes, it will generate heat, but I can guarantee it won't be in your dryer... 8/3 by code in Ontario.... Yes, I know it's more expensive, copper is a precious metal.... No, I will not cut you 18/2.... Sir, the building codes in Taiwan are highly suspect already, we've all seen in the news how many "modern" buildings collapsed last time you had a 6.1 quake; this is what ONTARIO requires.... well sir, you're more than welcome to take it up with the store manager, here's my phone, I'll dial 831 for you right now.... Oh, he met with you, laughed at you, and told you we wouldn't sell it to you? Yes, sir, that's why I like working at Home Depot, I know I tried to protect the little children in that house from burning up because you're an uncircumcised philistine. Have an adequate day."]
Moderators: If you don't believe me, Home Depot is hiring. After a week there, you WILL believe. Two years of home-built bidets using kitchen sink side-sprayers (note 1: kitchen sink side sprayers are controlled by the faucet, and it's assumed you're always there when they're on. note 2: kitchen sink side sprayers are rated only for the 5-10 PSI or so they see from aerator/diverter back pressure, not the full 50-75PSI of municipal water pressure) attached to municipal water pressure ("Why it burst? You sold me defective sprayer! What you mean I connect direct to city water? I cannot eh-do that, is connect straight to toilet. I come back from tree week in Yemen and find flood and notice taped to door. Now out $142,000 in flood damage to condo units beneath me! Insurance said I not use right part, they not pay! You pay! You pay!"). Two years of stoves burning through lamp cords ("why do I have to change my stove cord every time my wife uses more than one burner at once?"). Two years of actually having to work to convince people who *tell me they're plumbers* (ie. a guy who scraped together enough beer money to buy a pickup truck and a hammer and who now thinks he's a plumber - a trade which requires at least 5 years of schooling, people! Only 3 more to become a doctor!) that they can't use vinyl tubing to connect natural gas on a water heater for a little old lady who is dumb enough to let him into her house!
Find "Holmes on Homes". Shareaza, Torrents, etc. Download and watch a few episodes.
Jesus fucking wept.
Coathangers? I haven't seen them, but I'm a fervent believer.
IP addresses from the coffee machine (Score:3, Interesting)
We had an honour system for payment - an old desktop PC with a card reader. You swiped your ID badge through a card reader. All this did was extract the card ID string and send it through a shell script to a mysql database which then deducted from your balance the cost of a coffee - hand cash to the secretaries to top up the balance (I'll admit on average most people were in negative balance though every now and then the worst offenders had their balance details mailed to the whole department to shame them into paying up)
The actual purpose of the card reader PC? It was the DHCP server for the (still in use at the time - 2002) token ring network.
Re:the U-Bend (Score:4, Interesting)
Re:300 wires with a conduit sawed off (Score:5, Interesting)
A transformer is basically two lengths of wire running alongside each other for a long distance. If you run a low voltage circuit next to mains cable, you can potentially (in the case of a loop) create dangerous inductive heating in the low voltage cable - Not to mention the problems of the induced voltage in whatever equipment is connected to it.
Re:AC Guys Using A Garden Hose... (Score:2, Interesting)
Re:...it really is the answer (Score:3, Interesting)
There were two 8-port serial boards in it, and about 14 serial terminals connected for users. Those could operate at 19200 baud without trouble.
Then, there was the MODEM to use for UUCP connection. It was an all-new-tech 2400 baud modem, but it had to be locked down to 1200 because the entire machine would choke when the data was coming in at a whopping 2400 baud.
It turned out to be like this: the 8-port serial boards were "intelligent". They had their own micro that serviced the ports and communicated with the main processor using some protocol that apparently could transfer largish blocks of data quite efficiently. So the terminals, that were mostly-output, worked fine on it. Blocks of characters were transferred to the serial boards and sent at 19200.
Receiving was a different story. Apparently the boards cooked up transfer blocks of only one character in size, not knowing how much they could buffer and combine without upsetting the particular protocol running on the line (they were not using information from the serial port ioctls to know that). The overhead of transferring such a buffer was so high that the system could not do it quickly enough, resulting in many overruns and general slowness of the system. At 1200 baud it could cope.
Ironically, it would have worked just fine when that entire coprocessor crap had been left out and 16 uarts were directly interrupting the main 68000. With efficiently coded interrupt handlers it could have well handled the load.
(not the first and not the last time I have observed "cpu offloading" solutions that in practice were more of a drag than an improvement)
Reporting Pyramid Of Death (Score:3, Interesting)
From their the department needed to create a reliable and accurate reporting infrastructure to deliver key business data throughout the company.
What they had was a "primary layer" using Crystal Enterprise to create a number of huge spreadsheets containing the days data in large Excel files. The "secondary layer" were a Excel spreadsheets filled with VBA code to read the files created by Crystal process them a bit and output them as new files elsewhere. The "tertiary layer" was another lot of Excel spreadsheets filled with more VBA code to read the secondary layer and add drop down boxes and graphs etc to format the data. In some cases there was a "quatenary layer" which read the tertiary layer and did some more processing.
Needless to say all the VBA in the Excel files was mainly recorded macros with no attempt to check whether any of the various files it relied on were present or showing the correct data. All the files were scattered willy nilly around the network and all of them had to be ran manually in the correct order every day.
Frighteningly they had a huge backlog of further Excel files to write to do yet more processing and they were rebooting Crystal on a daily basis. I left after a couple of weeks because I didn't want to be around when the whole thing collapsed in a giant mess around their ears.
Bedlam DL3 (Score:2, Interesting)
Mobile phone under desk as production system (Score:2, Interesting)
I later discovered its purpose; basically it all came to do with "legacy" MMS messages, which are MMS to phones which don't have MMS capabilities (SMS only). The network instead sent the message to the customer as an email (SMTP, which is what MMS uses anyway), with the sender address of a certain "special" service. Apparently, due to the way the providers' network had been designed, if the GSM service it believed these messages were "from" was not "alive" on the HLR [wikipedia.org], it would refuse to send these thousands of legacy messages to customers.
I always wondered what would happen if one of the cleaners came through afterhours and decided to help himself to a free phone...
Long ago hack (Score:2, Interesting)
The computer that was running their organisation had been struck by lightning.
They offered me a lot of money to get the system up and running by monday 09.00.
Naturally I accepted the challenge and made my way to the site, it was unbelievable.
The lightning hadn't actual struck the computer, it had hit a power line 100 yards away, but the damage was incredible.
I thought he was joking when he said it had melted.
Well over the course of the weekend I managed to either build, repair, or source replacements for everything except the memory. It was now about 03.00 monday and I hadn't slept, eaten, or had a beer since about 12.00 Friday.
Fed up by now I hacked up a little serial interface and connected it to one of the tape drives that had survived the blast, man those things were tough, then created a little board on vero that took a memory address and serialised it. This was hacked into the system and I punched in the boot code.
10 minutes later, a monitor sprang to life.
Well when I say sprang, it more like oozed to life.
One character per second, you had to hold a key down until it appeared on screen then release it, but it was working.
Amazingly all their records had survived as well! ( found out later that all the tape drives were run off a seperate mains feed with a surge supressor, at least one engineer must have known what he was doing.)
They paid me off and ran like that for two days before they could get a replacement ram board.
VHS backups (Score:5, Interesting)
They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.
And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.
At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.
I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.
Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."
Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.
The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"
Wireless LAN (Score:3, Interesting)
Well, the cable was connected both to the wall, and to the computer. Unfortunately, it was clearly CUT right in the middle. When I questioned that elder superhero, he stated that he found out years ago that you could use a copper cable as an TV antenna, and he received a memo the day before that WiFi was now available in all university buildings - so he decided to cut the Cat5 to serve as a WiFi antenna....
Fax transmittal of customer database corrections (Score:3, Interesting)
They didn't have the time at the office to look up all these records themselves. Their solution was for us, the company providing customer support, to (I'm not making this up) take screen captures of the needed screens on the old billing system, print them out, and fax them to the home office halfway across the country. This faxing had to be done after normal business hours as our machine had to be free for other use during the day. Anyway, to give you an idea how much paper this was wasting, we were measuring usage by the ream, not the sheet. All of this paper was being fed through an autofeed on the fax/copier that would occasionally jam up, too. I volunteered for this assignment for the overtime.
Anyway, things were moving slowly and bugging from the fax/copier hiccups (this was a 30,000 page/month duty cycle copier that was used at closer to 50,000 pages according to HR) at about 2:00am I came up with a better idea, why are we printing these out and wasting all this paper when we can send them the screen shots themselves? I was noticing the readability was quite poor from some samples we sent back and forth with the other side. So we pasted the screen shots into Word documents, and then we tried to send the enormous Word documents over email, but the company's email server was complaining about the attachment size.
Well a co-worker had a GMail account he had been playing around with (this is when GMail was brand new, so he had gotten an invite for it) so we decided to try it because of the 10MB attachment limit. We could just upload the files and give the other side the log-in info and they could download the attachments at their leisure. Well, that didn't work out either, I think some of the files may have been too big or there was an issue getting the files uploaded, I don't remember anymore. But it didn't work out.
My next idea was to go back to faxing, but paperless. There was one workstation that had a dial-up modem instead of a NIC card, it was normally used to test access numbers for dial-up ISPs clients we had, but we hadn't used it in so long we didn't remember the login name to get into it, the password we were pretty sure of. After trying to guess it for a couple minutes I got the idea of booting the machine using a Knoppix CD and looking in C:\Documents and Settings\ to see what user folders were there (as I'd spot the correct login name amongst them). After we got logged into the machine we used a flash drive to transfer the files from the other machines we had been using to compile the Word screen shot documents. Then we'd open a document and fax it using Windows XP's built-in fax capabilities to the fax machine at the home office. So soon we had an over 50 page Word file printing over fax to the east coast with several more just like it queued up. It was moving slow but seemed to be working. The idea was we could now leave and let the machine fax the rest of the night.
I was excused to go home about 4am. I came in the next day around 1:30pm and found the fax calls had been interrupted around nine that morning. Apparently the home office had called because the same page was repeating over and over again on their side (which they naturally claimed must be an issue from our end). I didn't hear how they finished getting the records transmitted, but I think they went back to paper faxing again.
Now that I think about this, It would have probably been a better idea (if we'd had more than one day to do this) to just to take the huge Word documents and burn them to a CD and then Overnight the CD to the home office.
Re:2 SUNs == remote control (Score:2, Interesting)
For a temporary installation we had bunch of PC computers installed on a remote location. Those were controlled normally over ethernet connection, but I decided to run serial cables between them in case something odd happens with management port, like running ifdown for a wrong interface. It was a cheap insurance, and if I recall right, it did save one 1500 km trip.
About wrong installations, I remember that when one designed thin ethernet cabeling, they did not remember to take into account cables between wall socket and computers: it easily add 10 meters for each desk PC. Thus if you originally were in limits (185 meters), after some years your network segment was 350 meters and had strange network problems on some hosts.
the 50 cent U-Bend problem solution (Score:1, Interesting)
Power problems (Score:3, Interesting)
Twelve years ago, I worked in the I.T. department of a small private college in upstate New York (around 4000 students). Our main server room was fairly meticulous and well set-up (we had a perfectionist geek as the main sysadmin at the time). One summer, that building was scheduled for new electrical service to some new science labs in the upper floors of the building. A few hours after the electricians showed up, they came running to our offices and insisted we follow them down to the basement. There, they showed us the wiring to our server room, installed just a couple of years earlier: it was not actually physically connected. The wires had a small gap, and the electricity was simply arcing over. One serious bump of the box would probably move them enough to cut our power.
So their first task was to fix this. They would turn off the power for 30 minutes while we ran all the servers on UPSs, then temporarily reconnect power for awhile to recharge the UPSs, then turn off the power again and work... took all day at this rate.
Using the heat (Score:5, Interesting)
I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.
As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.
Re:Server room heating & worker Safety (Score:5, Interesting)
Re:the U-Bend (Score:4, Interesting)
No need to fill it, a few drops are enough. Oil floats on water, it spreads and forms a thin film on the surface. You get a lid that efficiently prevents water evaporation.
Here are a couple... (Score:3, Interesting)
Trying to trace some problems with one of the voice circuits, in one wiring closet we found the voice circuit had 3 twisted wire splices in one 6 foot section of cable...
Later, as a contractor running the email system for a not very large, but well known US government agency...
A few months after the agency had moved a lot of people into the new Ronald Reagan office building in Washington, I noticed their GroupWise server didn't seem to be functioning.
I called the local GroupWise administrator and told her they needed to take a look at it.
She called me back an hour later to say they hadn't been able to locate the server and were still looking, but she's sure she had seen it recently...
Two days later they found it in a closet in the old building they had moved out of months before...Still running, but someone had finally shut down the network in the building...
Re:the U-Bend (Score:1, Interesting)
Re:Continuous wiring (Score:1, Interesting)
You are allowed to do joins in wiring at any point that is immediately accessible for service. Obviously, the outlets themselves are immediately accessible, and so, joins there would be fine.
It might be odd, but honestly, that electrician was probably quite smart, and probably didn't have to come back to fix his own wiring very often.
For a low voltage example, this is how would would wire a punchdown block to split signals if only regular punchdown blocks are available (that's why good punchdown tools let you turn the wire cutter off). You just wire the wire continuously, looping it around the one side of the punchdown block, leaving you with multiple places to tap it from.
Re:Dungeon radio (Score:4, Interesting)
Along a similar vein, in my last apartment, I decided that I wanted to be able to listen to music in the bathroom while I took a shower. Specifically, I want to listen to the MP3s stored on my computer. Digging through my junk boxes, I found an old battery operated FM transmitter from ~1999 and an old FM Walkman from ~1985. I also had several sets of extra computer speakers that I'd just managed to acquire over the years. The last component in this project was my old Marantz stereo receiver that I haven't used in a while because the volume potentiometer needs to be replaced -- it will only output sound when the volume is cranked to less than sensible levels and then it would only work for about 30 minutes, after which I think it got too hot to operate any further. 30 minutes should be fine for music in the shower, though.
Anyway, I had to run a stereo mini-jack to RCA cable from the rear speaker output on my PC to a pair of RCA female to female adapters to about 20 more feet of RCA cable which then plugged in to the input of my Marantz receiver. From there, I used a 1/4" to 1/8" headphone adapter to output the sound to a 1/8" to 1/8" male to male stereo cable feeding the FM transmitter. I did consider plugging the transmitter directly in to the computer, but the thought was that I would have a better chance of receiving the signal if I used the Marantz as an amp to boost the gain in to the transmitter. In the bathroom I then hooked the Walkman up to a set of 2.1 computer speakers, using a nail in the wall to mount the walkman as high up the wall as the cable would allow, since the cable also acted as the antenna. In theory, it worked, but in practice the signal from the transmitter was just too weak to reach the Walkman. The transmitter ran off AA batteries, so I decided to see what would happen if I connected a 9V battery to the leads. (Thinking that maybe if I upped the voltage, I could get more transmitting power.) The result was the release of the magic smoke and the end of my silly project.
Re:Other way round (Score:3, Interesting)
After the person who hooked up our phone lines informed us that there was no way to use an old phone with these connectors, no converter, no nothing, we would need to buy new phones (which Bell South had a fine selection of, of course), and then each of the offices would need to be wired professionally (which Bell South could do!)...
I found that we still had: 1) A crimping tool 2) Leftover CAT5 cable 3) more splitters than could possibly be justified by a single DSL line.
I know nothing about phones, or electrical anything beyond my kindergarden science fair project on Conductivity (pennies DO conduct, oatmeal lids DON'T!). This is a piss-poor job.
Every phone in our office plugs into a splitter. An RJ11 crimped onto a CAT5 cable (two dangling wires) plugs into the same [read:wrong] end of the same splitter, which then plugs into the wall's RJ45 connector.
At the other end is a nest of wires in which many small wires pulled out of a CAT5 cable connect every line of the terminal to the same connection on the terminal as the only "real" phone line in the building (which is located in the 4'^2 "lobby")
I'm pretty certain the terminal is not intended to have four wires stacked onto the same connection. But as I said, I'm not an electrical anything. For all I know, this is how it was meant to be done. All I know is: It works.
Re:300 wires with a conduit sawed off (Score:4, Interesting)
Heating effect in a given cable rises by the inverse square of the voltage for constant power, which is why power transmission is done at 25KV or more. I don't think there's much OLE on railways in the USA, but it would be interesting to see what voltages they selected for that.
ian
Two separate networks on a single hub (Score:2, Interesting)
I was given a new 100Mb hub, the theory being that 100Mb is faster than the 10Mb we already had, so it would solve the problem. Not so! Those large files would still collide with the audio streaming because we had no intellient routing, traffic prioritising or cash to pay for a decent solution.
I discovered that the new hub would auto-sense the 10/100Mb speed from the NIC at the other end, but had no internal 10Mb100Mb switch capability. In other words it was effectively two hubs in one package with separate 10Mb and 100Mb buses inside. That turned out to be advantageous in the end. I set up all the audio workstations to run at 100Mb, and all the administrative workstations to run at 10Mb, so effectively we had two separate networks, one for audio and the other for admin. So the secretaries could continue sending their large files around, printing and so on, and it didn't affect the audio operations. There was only one PC that needed access to both audio and admin, and I solved that by simply giving it two NICs (making sure they weren't bridging).
It ran that way for several years. I believe more recently they employed the services of an IT contractor, who promptly saw fit to replace my old 10/100 hub with a fancy new switch. Almost immediately the stuttering problems returned. I don't know what they did to remedy that, but it seems to be better now. Presumably they have a bit more of an IT budget these days.
Breaking and Entering, err "Social Engineering" (Score:2, Interesting)
Nuts, Bolts, and Other Errata (Score:1, Interesting)
I worked there full time for three years and still work there one piddly day a week because I know what I'm doing and it's easier than hiring and training somebody else for that one pissant little day. I'm the resident bithead, so naturally all computer problems get routed directly to me. I have no idea what these people did before I worked there. (To be fair, I work for a boring old computer-laden engineering company by day.)
This place has a computerized point-of-sale system. It's called ABC, and I guess the right person with the right background might know that this is the "old" POS system used by TrueValue, formerly ServiStar - the two merged in the early 90's. The "new" system is Triad, and it runs in Windows in a semi-modern manner. No such luck for ABC; It runs in MS-DOS 6.0 (not 6.22) over Netware 3.1. All of the machines in this system (there are eight in total) are IBM GL300 workstations. Pentium 3's, with their vintage copies of Windows 98SE installed which are not used. Instead, every single machine in this system boots FROM A FLOPPY DISK. The hard drives remain untouched; Everything is pulled from a mapped network drive through DOS and Netware, which naturally resides on the server. And is read from floppy.
The mind reels.
So one of the first things I did upon discovering this was steal all the floppies and run them through my laptop, burning bootable CD's on halfway decent media and saving the floppy images on my hard drive, to be squirreled away on my file server at home. I loaded up every machine's CD's and set the boot order to go floppy -> CD -> hard disk. Already one of the bookkeeping machines upstairs and a POS terminal have had floppy failures and nobody even noticed until I pointed it out.
But it gets better.
It is of note that the ABC system runs in DOS and therefore requries very specific Netware compatible DOS drivers for its ethernet cards (IBM EtherJect 10/100 PCI cards, if you must know) and the sytem will only work with that specific type of card. Which is, helpfully, not made anymore. We've had multiple network card failures on one machine for reasons I'll get to in a moment, so my solution was to hie myself hence to eBay and purchase a lot of 25 of the stupid things for about as many dollars, which now reside in a file cabinet upstairs in a cardboard box maked with black sharpie: "Spare ABC parts network cards. Do not meddle with, mangle, or misplace, or we keel you." I now have some spare ABC boot disks and CD's in this box as well; If it is ever lost the entire business will surely go up in flames not long after.
I should explain the proprieters of this establishment. It's owned by a husband and wife couple, and their son. Mom, actualy grandmom, is pushing eighty. Her husband is in a similar situation but his health dictates that he isn't involved in the workings of the company much anymore. They, and their son (pushing 50) are not technically inclined in the least; Grandmom does all of the bookkeeping and paperwork upstairs and refuses to use a PC; Instead she uses a vintage IBM Selectric typewriter (I kid you not) and has somehow, somewhere, managed to scrounge up a source for ribbons for the silly thing.
All of our receipts and invoices are filed, by hand, in an imposing wall of file cabinets behind the office upstairs. They're in chronological order, every single one of them from about 1963, from when they bought the place from another owner. And grandmom can track down and pull any one of them inside of three minutes, and she's had to do it to settle disputes.
The nightly backup is put on an ancient parallel port Iomega Jaz drive on cartridges that I'm not entirely sure even still hold data. We've never had to restore a backup and I've never pressed the point. Some day I will.
But this is the crown jewe
Re:The Curious Case of the Magic SCSI Clock (Score:4, Interesting)
Your problem may have been a matter of EM emissions from the monitor cable: when the clock was positioned in that way, the EM from the monitor and/or its cable were such that they didn't interfere with the SCSI cable's signal.
"Secure" data center (Score:5, Interesting)
The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.
Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.
Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.
I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.
Concrete Anyone? (Score:2, Interesting)
The building my company is in was never designed to house IT infrastructure - even new newer additions are not build to house IT infrastructure. But we've made do and put infrastructure where we can, even if it means sharing a closet with the cleaning crew. One such closet had and old catalyst 2900 24 port switch in it, with the ports facing upwards. The connection are nice and hardwired into the switch, and generally looks like a rats nest (it pre-dates my employment). Recently in the renovation of the floor above it, the construction crew needed to poor a new floor. It didn't occur to them to plug the holes that had been drilled through the old floor over time, or cap the stubbed in conduit that had been put in. As a result of this the switch with it's ports facing upwards was filled with concrete. Completely and utterly filled. The cleaning crew reported that the fans made an awful noise for days. The switch never failed and is still in operation, although we can't unplug any cables or plug any new wiring into it.
At a different company, the server administrator was deeply convinced that rack mount server cases were evil and caused problem. He didn't want to use tower or desktop cases either, his reasoning? They were ugly (his words). The solution? He ordered servers, removed the motherboards from them, zip tied them into the rack via their screw-down holes, and placed the hard drives directly on top with cardboard "protecting" them from the motherboard. 5 racks of servers mounted just like this. I quite after a week of working for them, as I didn't want to be anywhere near that place when a zip-tie failed.
Re:long ping next door (Score:3, Interesting)
Re:Bizarre IT setup seen around the country... (Score:3, Interesting)
Back in 1999-2000 time frame I was working at an office that had been acquired by a larger company.
Said larger company used Exchange for all their email needs. Our office used an old desktop running linux. It was shoved in a corner of the server room but mainly forgotten.
I had set a limit of 1MB on email attachments on the linux server after some secretaries decided to start swapping 40MB word docs back and forth via email instead of the shared drives. They pretty much killed the mail server simply because it didn't have the disk space or IO speed to handle 40 MB attachments.
One day the HR department in the larger company decided to send out a 20MB PR fluff make you feel good about working here newletter.. via email.. to all 500+ employees.. Completely crashed the exchange server. Our little email server happily rejected the garbage and kept on trucking.
About 9 months after I left that company I got a call to rebuild the linux server. The hard drive had failed and no one knew anything about it.
At another company there was an old Novell Netware 3.x server in a closet.. covered in about an inch of dust with an uptime in the 4 year range.
-Jerry
Old School Hacks (Score:2, Interesting)
I got you beat--Slaughterhouses (Score:3, Interesting)
-One computer room in a pork plant had its air intake on the roof... right next to the exhaust from the rendering exhaust. Now, while the smell itself was awful, the worst part was that the sulphur and other noxious chemicals would eat the computers alive. The IT group had to install a special device that monitored the decay levels of samples of metals such as copper, silver, and gold to see how long the machines would last.
-One computer room was made out of a new office on the third floor of one plant. But there were no elevators to get the hardware up into the room. So they cut a hole in the floor, used a gigantic crane, and lifted two guys with each fully-populated rack up to the room. And the crane was still three inches short, so the two guys would have to do a wheelie with 1000+ lbs. of equipment to get it in the room.
-The same plant with the hole in the floor was also keen on bringing in electricians who were severely brain damaged. How bad? Imagine a 110V cord strung across the air with no support. And the other end of the cord with the three prongs was "hot". And someone once plugged it into the wall socket. I hear it doubles as a cattle prod now.
-Two computer rooms needed air conditioning, so they simply carved holes in the walls and hung out Wal-Mart brand home air conditioners. With no other insulation. In the winter sometimes frost appeared on the machines.
-Many of the computer functions were relegated to plant employees with their own unique vocabulary. I had the privilege of speaking with one woman in Texas who sounded exactly like Boomhauer on "King of The Hill." Honest. Another one always tried one-upping the "smart" IT guys back at corporate. On one occasion, she said that her monitor was "tricating". This was the word she used to mean that the emulation was off. I informed her that there is no such word as "tricating," at which point she told me that since I was young I likely hadn't heard of that word before. Funny, neither had Merriam-Webster.
Iraq (Score:2, Interesting)
- Cat 5e running out in all directions from my tent - some partially buried in gravel/sand, some over roads, etc. At 1000ft per box, I must have strung nearly a mile of cat5e over there. (FYI, cat5e will survive a run over a moderately trafficked gravel road for about a month and a half.)
- Three different Wi-Fi access points of different models / configurations. Having no external antennas for the "primary" access point, improved signal and range was accomplished by shimmying up to the top of a tent pole and mounting the access point to a platform hung from the pole. The whole mess was then covered up with a cardboard box with vent holes cut in the side to try and keep dust from clogging the unit.
- Tons of splitters and hundreds of feet of network cable to share the few TV antennas available. Starting at the cable in my friend's trailer, I once attempted to determine which antenna he was actually hooked up to. A complete loop and a half around the entire housing complex later, I still hadn't found the actual antenna.
Re:Phone cables from modems (Score:3, Interesting)
This wasn't obsolete equipment- node boxes were $50 each. He spent $600 of the computer lab budget on node boxes to add one computer to the network. He could have just bought them a laser printer for that.
My friends and I found out he'd done it that way, and went out to Radio Shack and bought a 100-ft roll of telephone cord and some crimp-on ends and made a 100' cord, and replaced the chain of short cords and node-boxes in the ceiling. He was none the wiser, and we ended up with $600 worth of node boxes and about $50 in short, pre-made phone cables.
That's when we started having LAN parties. Previously, we couldn't afford the node boxes. After that, it was Bolo [wikipedia.org], Scepter, and other early network games all night.
Re:Nothing THAT bad... (Score:1, Interesting)
Posted anonymously for obvious reasons...
I am one member of a five-person security group at.... a well-known, uhm, Internet security - related company. We're a medium-sized company (around 800 employees) but significant in our market and growing fast. Pretty much anyone in IT would recognise the name, and a lot of people outside IT as well.
We (Security Group) do not have network diagrams. Partly this is because we're not trusted with them, partly it's because we've not been able to justify access to the satisfaction of the people who have that info, but mostly it's because... we don't have them. Oh, we know where the main WAN pipes are, and roughly the ranges allocated to our 30 sites, and we know the 192.168-space the desktops and live production servers are; but... well, actually, we don't even know that -- we probably could find some old diagrams that might include sketchy details on how it was intended to be, or how it was when it was built, but there's so little control over the network that in practice, we have NFC what we have, or where it is.
I'm looking on this job as a really comprehensive challenge to all my skills, from layer 1 to layer 8. If we can get this stuff under control, rebuild the airplane in flight well enough to walk away from the landing, I'm gonna know that nothing in my professional career need ever intimidate or phase me again. On the other hand, it often feels more like a challenge to my sanity...
Re:Security incident (Score:1, Interesting)
Early VoIP (Score:2, Interesting)
nightmare wiring (Score:3, Interesting)
The first notable point was that the outside of the building seemed to be polka-dotted. Those were the bullet holes. Nobody was firing at the building per se, it was just downrange from one of the local hot spots.
The telecom room for the building was on the second floor. My friend's company rented the fifth floor, and some other company rented the floor above that. There were no data risers in the building. Instead, there was a hole about a foot across that had been knocked through the wall on the second floor, and a couple of six-inch holes on the fifth and sixth floors. The data cables ran out through the wall on the second floor, up the outside of the building (among all the previously mentioned bullet holes), and back in through the walls on the fifth and sixth floors. My friend's company had gone to the extra time and expense of running conduit, but the company above just had a 40' swag of cable hanging there in the breeze.
The telecom room itself looked like the aftermath of a "will it blend" episode. Take 20 drums of assorted wire product and unspool it all, wad it all up into a 10'x12'x8' snarl, then start grabbing random segments and pulling until you can nail that particular chunk of wire to some point on the wall. That's what it looked like. When they needed to fix something, they put one guy upstairs with a handset on the line and a radio, then sent another guy with a radio down to the telecom room to wave a toner around until the guy upstairs started to hear noise.
My friend also told a story about renting a phone line from the Chadian national government. That's all it was.. the same kind of POTS line you have to the phone on your wall, for a cost of something like $10k per month. One of the things he had to do was install a modem on the line (which saw about 95% use since it served a whole office), but the line itself was so unbalanced that he couldn't get a decent signal. He mentioned this to the local telecoms expat, who said, "oh yeah.. come on. You'll enjoy this."
They got their security people and drove over to the phone substation (which was run by the Chadian military and had armed guards outside), and before the truck even stopped, the expat was out and stomping his way past the guards, through the door, and into the wiring frame. As my friend came in behind him (pretty much thinking, "okay, we're dead"), he realized that he could hear voices speaking English througout the building. The military had tapped their line, but since it was used so much, they were running the sound through a loudspeaker rather than just listening to it on a handset.
Halfway down the row of wires, the expat stopped, pointed at the wire that dropped down from the celing and tapped into the frame, and shouted, "IS THIS A WIRETAP? IS IT?" By that time, *everyone* in the facility was there watching, and the colonel who ran the place was saying, "no no! Is not a tap," despite the fact that everyone could hear the voices of the people using the line over the loudspeaker.
The expat yanked the wire loose and said, "damn right it isn't a wiretap.. it's rubbish. We spotted it two seconds after you put it in. Here.. let me show you what to do," then proceeded to do a quick lecture-demo on professional wiretapping techniques.
The "Clean" Room (Score:5, Interesting)
Back in '95, my father was a VP of research for a large manufacturer of transmissive and reflective coatings for various glass applications (think insulated windows for the simplest example of said product) in Palo Alto, California. I was 15 and in highschool at the time, and having spent many a year trying like hell to keep a series of shitty no-name x86 computers up and running well enough to play the latest games, I had a sufficient skillset (and my dad had sufficient clout) to get me a job in their IT department. I did pretty well, and quickly found that users generally only got mean-spirited when made to look stupid, so a small dose of humility coupled with an interest in details on their primary task - "While I fix these printer drives you accidentally deleted, I was curious, what does a spectral photometer do?" - kept me out of trouble. Long story short, next year when I switched to full-time for the summer break, my boss actually brought me for a one-day business trip to our plant in Tempe, Arizona.
Now, you've got to realize, a business trip for a 16 year old (this was '96 now) is freakin' AWESOME. I was nervous as hell, had been up since the crack of dawn to take a red-eye with my boss out to the plant, and was deathly afraid I'd do something to embarrass not just me, but my father for having recommended me. So it was pretty unnerving to learn that my first job involved going into a large clean room production area, kept free from particles that could settle on the film during that specific type of sputtering process. We're talking the full disposable "bunny suit" that covered everything but the eyes, even with little slippers, and an airlock-type blower to clean you of all particles before entering.
The problem was a simple fix, really. The brand of 486 motherboard we were using at the time had a tendency, in about 1 out of every 3 units, to burn out the CMOS battery much earlier than you'd expect. And for a manufacturing-floor computer, not having a correct internal clock was a bad thing, not to mention that the lab techs had to go through some errors at startup with BIOS setings no longer being saved. So I suited up, cleaned off the replacement part and my tools as ordered, and went to find the bad machine.
That took some doing, oddly enough, since these computers were rarely shut down due to a 24/7 production schedule, so I had to go through back records on hand to find the lab techs' notes during the last power cycle on which computer had the boot errors. But, once located, the terminal was taken offline and I was able - after being told I had 20 minutes for the repair, tops, before the company would start to lose money as they needed that terminal again - to drag it off to a quiet, out of the way corner for the swap.
But see, there was a problem in the planning stages when this plant was set up. The PCs they used to control the machines were pretty complicated to configure, and the machines run in the clean room were just slightly modified versions of those used in the full-on manufacturing area in the main plant in Palo Alto. It was actually only a pretty small fraction of these production machines that had to operate in a clean environment. So when it came time to set these terminals up, they carefully washed off the outside of the older computers - computers, mind you, that have been sitting on a 24/7 PRODUCTION FLOOR with 10+ lab techs nearby at all times and various debris kicked up from the manufacturing process - and shuffled them off into the clean room.
So picture the scene: our hero, an extremely nervous 16-year-old on his first business trip in full head-to-toe bunny suit gear in the corner of a white, immaculately clean production floor opens his target computer to find a system so full of dust that he can't even SEE the goddamn cards inside. We're talking full-