Overwhelming Bureaucracy in the IT Department? 591
Nedry57 asks: "I am in the somewhat unique position of being a technology worker, who lives outside of the IT department in my company (a very large organization in the US). By far, the biggest challenge I face is getting anything done due to the bureaucracy that exists, within IT. There are certain tasks (i.e. anything that happens in the data centers) that I don't have the access to do. Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration), thus costing many times more than it should. The lack of agility is maddening, because I know we are missing significant business opportunities. My management is extremely supportive and despite our excellent track record of success in creating robust/secure applications--our work has passed audit numerous times with flying colors--we get no support from IT. Even senior management can't break through the barrier. I am very interested in hearing the experiences Slashdot readers have had in similar situations." How do you get your technology work done, when your IT department is more hindrance than help?
Buy In Or Bail Out (Score:3, Interesting)
P.S. It sounds like you need to acquire funding for a development and testing lab that is not under IT however, do not expect to connect such a lab to IT's network.
It's all relative.... (Score:1, Interesting)
Work for Yourself (Score:4, Interesting)
I worked for the government... (Score:2, Interesting)
The military (USAF) had a very good IT setup (overall) that was basically setup the way you'd set up a good memory architecture.. you have a hierarchy of IT with the most used/essentialy tasks able to be done close to where they requests come from, and build upwards and outwards. Those local people were "fired" (in the government people don't actually get fired, but moved) if they didn't perform, and they were basically giving the keys to their kingdom. In the grand-scheme of things, it actually ran pretty effeciently, and we were never waiting on IT for more than a day, except in the most extreme cases.
That being said, the military has a pretty large vested interest in people being able to work and use their computers (ie, the cost of failure can be scrubbed missions which equates to huge amounts of money down the drain) so things tended to Darwin into a workable system. It sounds like your company's IT organization is just immature or flat-out poor (I don't mean in money... althought that could explain poor quality, also), and the powers that be don't seem interested in fixing it.
Re:Buy In Or Bail Out (Score:2, Interesting)
He hated me, or should have, but eventually resigned and was replaced by a new head of IT with company agreement from the board of directors that it was time to enter at least the 1980's in the computing world, if not the 21st century. But to make such a person resign you have to get the failures on paper: record these failures, get them on paper, get the data collected, and present them on a regular basis to your supervisor and theirs as appropriate.
"We're Not Freaking NASA" (Score:5, Interesting)
He and others in the IT department tried doggedly to get security noticed, only to be shot down by executive management. To paraphrase the CFO and strip out the gratutious profanity, "We're a meat company. We turn happy cows into happy steaks and happy pigs into happy bacon. We're not freaking NASA. We don't need to worry about our computers like Lockheed Martin does."
Several months later a virus hits the company and the phone system, which includes all sales offices, dies. I rush and get the tools to remove the virus in every hand possible.
Ultimately, as I was leaving the company, they finally hired a security manager. This was only because of Sarbanes-Oxley, and that person was given the role of a paper tiger--no authority to change things to be more secure, but a perfect picture for blame should something go awry.
When I left, I entered another office with other politics, but it is nowhere as bad as it was there.
Don't underestimate a bribe (Score:3, Interesting)
Re:No Exaggeration? (Score:4, Interesting)
Now THAT was funny... 3 months later I had a working application sitting on a shared server, and I had to go. We had about 1 week's worth of data in there, but that was almost 100,000 rows in most tables.
Make It Happen (Score:5, Interesting)
Because of the percived importance of uptime on this network, everything required mountians of paperwork. Installing and removing nodes from the domain required three administrators, setting up a new machine required a month on a private VLAN being monitored by a sniffer, memory and hard drives were obselete before they got to the customer.
Anyone who ever worked around an UPS knows how they die. They give plenty of warning. Having an UPS fail is a rediculous way to lose your backbone infrastructure.
My predicessor had done a wonderful job of installing an UPS for every router and switch in the datacenter. Problem is, both power supplies in the routers and switches were connected to the same UPS. In cases where an UPS was about to fail, he unplugged the UPS from the wall and plugged it into, you guessed it, another UPS.
He didn't do it out of ineptitude; it was done because the only option was to clash heads with the IT overlords. They would require studies about how many UPSs failed and if it failed before the MTBF, they'd want us to try and recover money from the manufacturer. They'd want contractors to come in and examine the UPS to bid on a UPS monitor and replacement contract.
In short, asking the overlords was like asking to be turked by a syphalitic bear.
So, some BOFH, overwhelmed by the prospect of repairing the power system, chose another path. He walked over to a failing UPS and simply turned it off. He was the only one with the access to turn it back on, so he had no reason to worry.
Within two hours, all in-progress meetings were cancled. The Supreme Overlords demanded from on high that this lowly tech was to get a blank check and a blank trouble ticket (approved by the Supreme Overlords) to do whatever he needed to do to prevent that from ever happening agian.
Electricians installed two seperate power feeds into every rack.
Each power supply got a seperate UPS.
Old equipment was updated.
Everything was strawberry fields and unicorn giggles after that for the infrastructure department.
Now, to answer your question: You have something that someone wants. Hold it hostage till you get what you need.
Yes! (Score:4, Interesting)
Now I run my own company with lots of production severs.. No paperwork required, and I've automated most stuff.
If you are stiffled, go out!
I find ways of circumventing the bureaucrats (Score:2, Interesting)
1) Circumvention: Recently, I needed a DNS change to point an existing subdomain to a different IP address. Our not-very-useful IT project manager told me they needed to come up with an LOE for changing the DNS entry. Three days later, they told me they hadn't had time to calculate the LOE and would not be able to complete the change by the following week's deadline.
I went to the head of our corporate marketing and branding group, asked for her help. Even though this is a very large corporation with more than 30k employees and a very significant IT organization, within 10 minutes, one of the staff members on the marketing and branding team physically made the DNS change herself.
2) Mockery: I needed our web team to add a link to the bottom of our company's homepage linking to my program's home page. Three weeks later, the guy who was going to do the change saw me in the hallway and asked me if I had lined up testing resources from our testing outsource company to make sure that the link worked.
I responded, very loudly and within earshot of the web developers: "open bracket a href equals quote h tee tee pee colon slash slash my dot domain dot com close quote close bracket My Site Name open bracket slash a close bracket and then click on the damn thing"
2) Fool them into wanting to do it for their own purposes: We decomissioned a website a few months ago, and it is no longer publicly available. However, we've kept it around while we make sure we got all the old documents. However, we are still getting monthly reports extracted from the back-end database. I contacted our IT reporting team and asked them to stop delivery of these reports since they're no longer needed. They sent me a form I needed to fill out justifying why I needed these to stop and aking for VP signatures and notarized copies of the marriage certificates for every gerbil I've ever owned.
I told them they had to be kidding, then I set up a rule in Outlook that automatically bounces back the reporting emails to them and deletes them from my inbox. I don't have to worry about it and once they start getting these every month they'll try to figure out what's wrong and fix it. Once we fully bring down the system, I imagine that the report engine will start throwing all sorts of error messages and they'll see fit to do it on their own if the auto emails don't do the trick.
It's sad that IT, something that shoudl enhance productivity, has become a huge obstacle for us to do business.
Re:IT (Score:5, Interesting)
The cost savings are barely %15 at the most and the Indian management companies take most of the cost savings away.
You need to spec requirements for any programming projects and you can't outsource business processing that far away. If anything efficiency eats in and costs actually go up.
There are a few companies that are %100 based here in the US where manufactoring, operations, and management are all in one location. Outsourcing to China will actually cost more because work wont flow seeminglessly or as easily with everything apart.
I wonder if this guy works for a government contractor or has the military as a customer? Such companies are required to do tons of checks and ballances and security.
Regulatory Compliance Nightmare (Score:2, Interesting)
You've missed the entire point (Score:5, Interesting)
Cynical? Yes, but also very true. The above is the root of the issue. I'll put it in the terms that IT would:
ITs job is to keep the servers running, smoothly, with as little interruption to daily work as possible. As with any complex undertaking, different users have different priorities. CxOs come first. Period. Internal needs come next (see: "servers running, smoothly," above). High profile departments are next - marketing, sales, accounting. The last one is mostly because it comes under a CxO (F - you can choose what it stands for) who is intimitely involved with the month-to-month operation, and through which everyone gets their pay checks (including previously mentioned CxOs). Development is pretty far down, as you can see. You must understand - you don't bring cash into the organization (sales), nor do your efforts directly affect the price of company stock (marketing), both of which are of top importance to the CxOs.
That does not mean that you are not essential. But you are essential in a way that is ongoing - like the janitorial staff. If they lose development, things will slowly start to degrade, but it will be a while before there is a crisis. Either way, its an expensive mess to clean up, but if you throw some cash at it, you can bring things back to livable.
Now, lets look at the flip side. If IT goes down for a day, there will be hell to pay, and heads may roll. Every IT person knows this. Anyone who has dealt with complex modern systems knows that it's a house of cards. There are so many things that can go wrong. One failure, if not just costing your job, is certainly going to make for a long night getting things back in order. That would be uncompensated overtime, remember. Also, ten years without a single failure will not make you a hero, like landing a new sales client, or scoring a great marketing campaign which lifts the stock price or sales. It will make the company think you're reliable, but boring. Bonus aren't given out for boring. One failure, on the other hand, makes you a villain.
Now, if you've made it this far, how much value is there - for the IT professional - in helping you get your job done faster. In case you've skimmed, I'll tell you: none. It's like playing russian roulette for fun. Unless you just happen to like the life-or-death thrill, or have nothing to live for, it's a fools game.
I wish I had better news for you, but if you have a large corporation, than you have an ingrained corporate culture, and IT subculture. And they don't drift your way.
Oh, I've never been in IT. They piss me off 'cause I'm an engineer and just want to get shit done, and they want to worry about making sure the CEO's internet never goes down. I've learned over the years that, in effect, that is their job. I've stopped fighting them and learned to either (a) work with them or (b) work around them. The latter is done carefully to avoid stepping on toes. Just as they are under the thumb of uper management, they like to exert their power where they can. That would be against you and me. You don't tunnel under a mountain if there's a reasonable way to pass around it.
Re:IT (Score:3, Interesting)
In a similar light, my own corporation has a complete clusterfuck of an IT department, since it is centered entirely on being as cheap as possible. Not only do we have the India effect, we have a security model centered on making sure every employees laptop runs a particular OS image that has a virus scanner fundamentally attached to the ON button (slight exaggeration) so that a virus does not somehow sneak into a factory image. This way there is only one network to maintain and thus we are "cheaper". (similarly websites are "banned" for being not work related, to save bandwidth you see) This also means no linux boxes, no stray equipment on the network (in a HW dev company, this is a real bitch since most test equipment can dump data to the network), etc. Try to convince them to build a better security model that relies less on "good citizens" and they try to get you fired.
Re:No Exaggeration? (Score:4, Interesting)
Re:IT (Score:5, Interesting)
I work at a large university. My start in IT began in a non-IT department, and I had to work with the IT people all the time. To them it was a game to try to stop any progress I wanted to make.
They would make me wait months just to add a column to a table in a database that only I used.
They took 2 years to 'investigate' moving from a flat table database (FoxPro) to a relational database (Visual FoxPro) but never migrated anything on the production server, because they were worried about incompatibilities. (FoxPro/Visual FoxPro were the only options they gave me)
I could list dozens of things- but their prevailing attitude was that I was an outsider, and only the IT group should be doing any IT work. I wouldn't have even started doing the work if they had been effective.
Well, now I've moved up, and I head a different programming department. The lessons I learned at my previous position have been serving me well. A little too well in fact- other people who have to deal that those other IT people are coming to me just to get a little server space...even from the other department.
I don't know, but I see IT (especially at a University) as a group that should facilitate others in doing their work- not hinder them.
Okay, so I'm bitching, but this stuff happens. And the sys admins get away with it because their boss doesn't understand what the job entails.
I feel your pain (Score:2, Interesting)
I am in the same situation at the moment working for, err... a major british telco. I've done it before as well, I spent 10 years working in the civil service. That was pretty tough.
I have a consistent approach to the problems of working this way: I do everything myself. If I need a server? I buy one, charge it, plug it in, install it, support it. If I need hosting? There are plenty of hosting companies out there selling services; I buy it and set it up.
I do try and keep the things I do clean and secure and "away" from the IT department.
I do try to point out, whenever I can, what a clueless bunch of losers the IT department are.
I do try to get people on my side by doing favours for them with "my" resources as quickly as I can. If you can save an important managers pet project by judicous installation then all well and good.
I tell as many people as possible about what I am doing; taking care to point out that if I relied on the IT department I would never have been able to achieve success.
Mostly this approach is getting easier. It's easier to buy powerfull servers that can host masses of virtual machines; it's easier to get the hosting you need.
Lastly, remember that IT departments are so swamped by their own dumb rules and ineptitude that they have very little time to concentrate on trying to stop someone who knows what they are doing.
IT is not the obstacle (Score:3, Interesting)
Re:"We're Not Freaking NASA" (Score:5, Interesting)
I work for NASA, and the IT on our office systems (NOT the production/mission critical stuff, thank God) is the worst thing I've ever seen.
My workgroup of 20 engineers has a shared server space of...300 Megabytes (that's Mega, with an "M"). Our actual needs are around 10 Gigabytes.
So...about 20 Gigs of spare drive space on one guys machine has gotten shared out and is now the de-facto server. It gets backed up every week or so to another machine, and maybe monthly DVD backups get burned.
This is a terrible solution, and I know darn well that the 2 or 3 man-hours a week it's taking to maintain this thing costs a hell of a lot more than giving us the correct server space we need. Let's not even mention how much it will cost if we screw up and lose something. But...IT is funded seperately, and they could care less how much labor we waste making up for their inadequate infrastructure (a big problem in any government org is accounting for wasted labor like this).
I won't even talk about the "improvements" to the mail server, which resulted in day long email crash to several thousand users yesterday.
Re:Recognize those things you cannot change.... (Score:1, Interesting)
> department with respect, give them credit and appreciate
> their work. They are the ones who save your ass when you
> type "rm -rf
> and see those 9 months turn to 9 seconds!
That goes not just for IT, but _anyone_ you work with. I was a lowly admin-assistant for a few years with three different bosses. Fun fun. These guys could have been part of a psychological study. One was very neutral. One was an utter ass. (Think "I am the superior and you are my secretary! File these beotch!" attitude.) One was sincerely polite. He never took me to lunch because I had to stay and hold down the fort while they were out but he would often ask if he could pick something up for me if I was having a particulary harried day. Also, at those times, the amount of work he had for me would level off a bit. I'd see him sending his own faxes and such. Little things, but every little bit helps when you're busy.
Take a wild guess who I'd bend over backward for at THEIR crunch time.
It may not be "appropriate" but hey, human interaction has been going on for many, many years. If you want treated well bring some nice berries back to the cave every once in a while...
A CIO's Perspective (Score:2, Interesting)
First, the reason that IT organizations typically don't like technical folks outside of IT developing their own internal business apps, building their own infrastructure, or buying their own gear is pretty simple: we're the ones generally charged with ensuring predictability and security in the corporate infrastructure and we lose the ability to mitigate risk and provide reasonable levels of support with each bit of control that we give up. It's the same reason that the Legal department doesn't let you write your own contracts and why the Finance/Accounting department doesn't let you make journal entries.
I can't tell you how many applications, systems, and servers that my respective IT departments have had to inherit because the well-meaning business employee who developed or setup the system had either lost interest or moved on. When this happens we find that nobody left in the department knows anything about what is inevitably deemed a "critical app" by the department head (and is usually running on a server under a desk in a vacant cubicle). This scenario also applies to self-setup infrastructure of all kinds -- We regularly find rouge wireless access points, PCs and laptops bought and 'expensed,' application-ready mobile phones attempting to attach to our network and on and on.
The only way to deal with an increasingly technology saavy workforce wanting to do their own thing, in my opinion, is for IT to set clear policies and processes that allow for a certain amount of 'self help' but only within the guidelines of an IT ecosystem-friendly arrangement. We need to know about hardware and software you buy or make and we need to know where these systems and sub-systems reside, what data is on them, how that data is protected, who has access, and who is responsinble for maintaining them. In this day of increasing scrutiny (SarBox, etc), its more important than ever that we maintain some level of control.
Aside from all this ranting, I'll say that IT Leaders who do not realize that they are service providers at the end of the day are doomed to be loathed by business users. CIOs who stand fast with their arms crossed saying "no" to everything are obviously not familiar with the way a service organization is run. Unless a service-oriented culture is fosted from the top of IT, things will never change in your organization. The most successful CIOs that I know spend a lot of time with business department heads ensuring synchronization of priorities while also instilling in their IT employees a sense that proper, measurable internal customer sat is a standard part of doing business. Take to your CIO to lunch if possible and talk about this. I think you'll very quickly be able to tell if you have any hope of a culture shift.
Re:IT (Score:2, Interesting)
Simple solution. (Score:2, Interesting)
Re:IT (Score:5, Interesting)
Most of the time O/S contracts are not negotiated by tech savvy people which results in ridiculous clauses.
The contract I'm working on at the moment only allows us to delay releases a certain number of times in a year and allows us a certain number of outages.
Fair enough you may think...
Now, if we're close to the limit on delayed releases but way ahead of the curve on actual outages what do you think we're going to do when we have to call go/no-go on a release with only a 50-50 chance of being successful? If we pull it we definitely get hit on the service level agreement; if we put it in we've got a 50% chance of taking no hit and a 50% chance of an outage which we can absorb easily. Is this the best thing for the customer? No. Is it the best thing to do pragmatically to protect the profits of the outsourcer? Yes.
Another outsourcer at my company is only contracted to create 30 (IIRC) user IDs per month. If you're new hire 31+ you're out of luck until the first of the next month & the company normally hires in big blocks (when the graduates become available). Somebody averaged the number of new users over 12 months without negotiating in the flexibility to overspend one month & underspend others. It can be created of course but that means big bucks... That outsourcer had used up all of their projected 5 years budget within the first two years with all of the incurred excess charges for stuff like that. Mind you they were SO incompetent that the failures in other areas of SLA incurred penalty clauses to partially counteract that...
I agree that entrenched IT departments can be really bad to have to deal with but they can be fixed if senior management has the will to do something - maybe the CEO needs to be told there's a problem instead of the usual "everything's fine".
If you have a LARGE IT department and you believe outsourcing is the answer - you probably asked the wrong question. Small-medium companies with limited and well defined requirements can and should outsource. I do not believe large IT departments can be economically outsourced because the increase in management overhead that is incurred more than outweighs any savings that may be made - you end up paying for the outsourcers managers while you have to keep your managers to liase with their managers... If you write a cast iron contract the outsourcer will have already charged you a shedload of money to negotiate said contract and you will have also spent a lot of money on your peoples time negotiating it. If you don't have a cast iron contract then you can open wide & say ARGH!!! because the outsourcer will ream you for every excess charge they can before you go bankrupt.
Re:IT (Score:3, Interesting)
Re:I may be taking you too literally (Score:5, Interesting)
The right answer is somewhere in the middle, not a bureacracy expanding to meet the needs of the expanding bureacracy, and not departmental IT Lords all deploying their own solutions, Linux here, Windows there. But I so rarely hear about anyone finding that middle ground. I've seen a balance at big tech companies, but a balance of centralized and departmental IT expending 75% of their energy in a tug-of-war. The departments and divisions of a tech company can sometimes effectively fight the bureacracy because there's geeks in all corners who know what they're talking about. At a software company especially, the product teams rule, they know it, and they can fight about IT issues on even footing with the IT bureaucracy. In most other industries, the key departments don't have that advantage, so at the end of the day the IT folks make the IT choices, always making noises about collecting and meeting business requirements, but free to say "no" without much effective pushback.
My basic point was about human nature. Even if you create that balance, with a central IT plus dedicated IT staff across the organization, eventually the centralized guys win because their chief sits at the table with the other C*O's and exerts more pull, making effective noises about standardization lowering costs. It's simple corporate politics. If that CIO sees the big picture and has some humility, s/he might end up leading an organization that does the right things. More likely, even with that CIO, the IT middle management underneath will still play politics and make arbitrary rules and decisions that benefit themselves and disempower everyone else.
On another note, I never meant to suggest that a NAS from Best Buy was a good choice for any office needs. It's just that 6+ month turnaround on upgrades or new solutions is what drives people to route around that crap and starting using things like that NAS, or worse, Microsoft Access. I guess it's kind of a similar phenomenon to the adoption of the PC and M$ software in big businesses in the first place, to route around the mainframe cult.
Re:Wrong Wrong Wrong (Score:2, Interesting)
I once left an organisation after telling everyone I could reach up the management chain what was wrong and being brushed off. The one time that people actually wanted to listen was when I handed in my notice. Management and HR then wanted interviews to ask why I was leaving. I told them that as I was already leaving, sharing that information was hardly in my interest.
size * freedom = K (Score:3, Interesting)
Re:Wrong Wrong Wrong (Score:1, Interesting)
It turns out that their phones hadn't worked at taking calls for three months since they'd switched offices so every other call left for them also got ignored, that they were all burning reams of paper every week printing out resumes, and that after I spent 2 months integrating all of their code lines into a properly source controlled repository with my manager's agreeement, I got laid off because someone else could be shifted to manage the integration, because I'd documented it. They were kept because their code was so unstable only they could hope to manage the old code, and they'd successfully pushed back every test of the new codelines because they "had no time to review it". Then two months after the layoffs, they resigned in masse to do a startup that failed within six months. (They failed to get the hint about the dot-bomb killing stupid business ideas of technical whizkids.)
Unfortunately, this nonsense is typical when you move development and support groups away from the main office. They lose track of background priorities, and they'll pull complete nonsense to protect their little remote fiefdoms.