Ask Slashdot: Biggest IT Management Mistakes? 341
snydeq writes: Sure, nobody's perfect. But for those in charge of enterprise technology, the fallout from a strategic gaffe, bad hire, or weak spine can be disastrous, writes Dan Tynan, in an article on the biggest management mistakes in IT. "Some of the most common IT gaffes include becoming trapped in a relationship with a vendor you can't shake loose, hiring or promoting the wrong people, and hiding problems from top management until it's too late to recover." What are some other career- and company-destroyers you've witnessed in your years in IT?
Pretty Much Universal. (Score:3, Insightful)
Focusing the department on nothing but fire stomping and not focusing on preventative design/administration.
Re:Pretty Much Universal. (Score:5, Insightful)
As soon as you outsource you are in the hands of the company you outsourced to and that company don't understand your business model, only profit for themselves.
Putting things in "the cloud" is probably the most dangerous thing you can do these days. It's like peeing in your pants, it feels warm for a little while.
Re:Pretty Much Universal. (Score:4, Insightful)
For many companies the cloud is only cpu power, and connectivity. ... because I back up my private stuff myself.
There is nothing wrong in putting stuff into the cloud if you have backups of your data and can reliable shift from one cloud provider to another one, for a company.
And for a private person it is just convenient to use cloud based back up storage. I don't
Re: (Score:3)
Focusing the department on nothing but fire stomping and not focusing on preventative design/administration.
LOL! Try doing anything else when the company you work for sees IT as nothing else than a cost-center!!!
frosty (Score:5, Insightful)
Implementing SAP
Outsourcing
Outsourcing your SAP implementation
Re:frosty (Score:5, Informative)
Implementing SAP
Outsourcing
Outsourcing your SAP implementation
Whenever I talk to someone from a company that uses SAP, I always ask if they are satisfied with SAP and would choose to use them again.
So far, this many have said yes: 0.
For comparison, this is the number that have said they are happy with Oracle's ERP: 0.
Re:frosty (Score:4)
Implementing SAP
Outsourcing
Outsourcing your SAP implementation
Whenever I talk to someone from a company that uses SAP, I always ask if they are satisfied with SAP and would choose to use them again.
So far, this many have said yes: 0.
For comparison, this is the number that have said they are happy with Oracle's ERP: 0.
I'll see your Oracle ERP and raise you one Peoplesoft, a true marvel of software engineering, where database tables have intuitive names such as PSPRSMDEFN or PSFLDFIELDDEFN (those are real names).
Runner up: anything from ASG.
Re: (Score:2)
Wow, the choices seem kinda limited then?
Re: (Score:2)
I work for a company that has implemented SAP - and yes we are happy (so there's a one for you).
Re: (Score:2)
Re: (Score:3)
He said implemented, he didn't say that it worked. Contractors make bank on SAP installs, I'd be very happy spending 5 years implementing the forever project with no downside save boredom.
Re: (Score:3)
It worked - for all but one of the companies I have worked for in the past 25 years. The one company where it did not work, the main reasons for it not working were catastrophically bad management; basically someone decided that a good salesman would also make a good manager (this turned out to be a bad decision). That lead on to multiple poor decisions, and in the end a failed project; whilst that SAP product that they were implementing, at the time was immature, but many other SAP customers have experienc
Re: (Score:3, Informative)
No I work directly for an SAP customer (have done for 5 years now, cannot say which one unfortunately). SAP is not an inherently bad system, it really depends on how knowledgeable your implementation team are. SAP has some really dedicated developers and some very good support. There are definitely SAP horror stories out there - that people seem to love repeating forever and a day, but I can honestly say that I have worked with SAP for 25 years, and 20 of those were as a consultant, and in that time I saw o
Re:frosty (Score:4, Interesting)
Being widespread doesn't make something good, people copy each other even when doing something stupid... Also all the examples you give are large companies which is a very important thing to consider...
SAP is a large, expensive system with many hidden costs in addition to the purchase price. You will likely have to buy lots of dependencies, lots of highend hardware, hire many expensive and highly trained staff to manage it and develop custom additions to handle your own business needs.
If you have the budget to do this, then it can work well... But many smaller companies go in blindly because they want to copy what these larger companies are doing... They get unrealistic quotes from greedy third party consultancies, or only see the ticket price and don't consider the true cost. They buy the software, but don't buy enough hardware to run it adequately, or don't hire sufficiently competent staff to manage it.
Many sales people will blatantly lie to you in order to sign you up for a large purchase, and then completely fail to deliver leaving you locked in with a huge bill and a big mess to clean up.
The end result is colossal failure and a big mess, or a system that limps along and still ends up costing a fortune.
A lot of people have a stupid mindset that "company X is huge and successful, if we copy them we will be successful too".
Copying a company 100x your size is not a good business plan, if you're a small company you should act like one and play to your strengths. You don't have the economies of scale or huge budget enjoyed by large companies, but you have agility that large companies lack.
Re:frosty (Score:4, Interesting)
A lot of people have a stupid mindset that "company X is huge and successful, if we copy them we will be successful too".
Copying a company 100x your size is not a good business plan, if you're a small company you should act like one and play to your strengths. You don't have the economies of scale or huge budget enjoyed by large companies, but you have agility that large companies lack.
Now try telling that to the legion of devs who blindly use products like Cassandra, Hadoop or Storm "because we need to be able to scale". Then they jump through hoops trying to get it to do what they need (and not always successfully).
Re: (Score:2)
I think you just gave the material for about 5-10 episodes of a hilarious sitcom.
Outsourcing (Score:5, Insightful)
People in India can't possibly know your business, your other employees or your customers as well as qualified, competent, real live boots on the ground in the US of A.
Re: (Score:2)
Outsourcing and offshoring... (Score:5, Insightful)
The two biggest mistakes I see is that a dev team which is doing fine gets cut and the people outsourced or offshored. Sales and quality hit the shitter, but management doesn't care one whit about that, since to them, the only people that matter are the S&M guys (sales/marketing), so more gets offshored.
Biggest mistake (Score:2, Informative)
Executives with no IT experience running IT departments.
Re: Biggest mistake (Score:5, Insightful)
Executives with no IT experience running IT departments.
NO. More likely the executive making promises to customers and PMs without checking with the IT department and then demanding a months worth of work be done in 72 hours to one week and fire them if they can't get it done etc.
If you do get it done then it will continue. You are screwed either way
Re: Biggest mistake (Score:5, Interesting)
A friend makes bank now after his CEO got pissed at sales and marketing doing this shit. His sole job is to go to sales meetings and keep sales honest. He knows the business well, and studies up on the customer to understand who they are and what they seem to need. Then he shoots down sales during the meeting when they start promising too much or trying to sell something that the customer doesn't really seem to want or need.
The marketing folks hate him, but the CEO, developers, and the customers love him.
News stories: Intel and Microsoft spyware. (Score:2, Offtopic)
Microsoft's history is filled with abuse [slashdot.org]. Important 3rd comment: Possibly some other person from India... [slashdot.org]
Windows 10 is possibly the worst spyware ever made. [networkworld.com]
Re:News stories: Intel and Microsoft spyware. (Score:5, Insightful)
Show me an OS that can replace W10 with everyday apps. There isn't anything out there that can handle Acrobat, MS office, Outlook, Exchange and other items, other than Windows. Oh, AD capability is a must out of the box.
Don't tell me what to use. Tell me what you are trying to accomplish, and I will find a solution. That's my job as an engineer. Of course you can't run Exchange on Linux without virtualization. But if your goal is to get e-mail, calendaring, and contacts I can do that for you.
Need Acrobat? Sorry. Try coming to me and saying "I need to read PDFs" or "I need to save this document as a PDF". Sure. I can solve those problems for you.
"I want Office on Linux". No, you don't. You want some of the *features* Office provides...on Linux.
Re:News stories: Intel and Microsoft spyware. (Score:5, Insightful)
What you're asking is extremely unfair, especially from users who are not used to think like IT-lawyers. To them "Acrobat" is not just "reading PDFs", it is a whole set of capabilities including things you won't find in free alternatives (like encryption, form support, etc.). Asking users to articulate those needs individually, when they can summarize with the phrase "I need Acrobat", which they already know and of which they already know it supports their needs, is both unfair and unrealistic.
And it's not just capabilities, it's also a general lack of hassle. I use OpenOffice a lot, but frequently it fucks up the layout of Word documents. As a user I neither want nor need that hassle.
Look at it from a users' perspective: what they have now works. What you are selling is a journey into a world of pain, hassle, and "sorry we can't do that". Why would they want that? What tangible (i.e. not abstract) advantage do you offer to make all that worthwhile?
Re:News stories: Intel and Microsoft spyware. (Score:5, Interesting)
And some of the things I hate most about Acrobats are these unnecessary additions to PDF files. It should be a read-only format, impossible to turn into a vector for malware. But no, Adobe screwed that pooch. I cannot open a PDF now, with Acrobat or Preview, without it thinking I have just modified the document and so it will ask me when I close it if I want to save my changes. I don't want encryption, if I wanted encryption I would encrypt the file. I hate vendors that send me "secure" documents and then require me to check in every 6 months to get new certificates so that I can continue to read it. I just want to read the document and I don't need the heavy weight Acrobat with it's strange UI doing this.
Re:News stories: Intel and Microsoft spyware. (Score:4, Insightful)
Many of those users also get by with a given application because it's all they know or all they have access to, but if they got used to something else they may soon get used to and benefit from additional features offered by that software.
Many users actually complain about bugs or deficiencies in software they use on a daily basis, bugs or deficiencies which are not present in other software. People put up with what they're given and get used to the bugs and workarounds required, then will complain if you give them something else because it now has a different set of bugs and workarounds even if its superior on balance.
Re: (Score:2, Insightful)
Software cost is minimal compared to (re-)training cost and process/workflow design.
Well by that definition Windows 10 doesn't meet your goals either, because it requires significant retraining of Windows 7 users. Of course Windows 7 doesn't qualify either as it required significant retraining from XP, which required significant retraining from 95/98.
By your specifications there are only two Microsoft products that meet your goals, DOS and Windows 95/98.
Since neither of those products are offered for sale anymore, that completely rules out all Microsoft products as functional for your pur
Re:News stories: Intel and Microsoft spyware. (Score:4, Interesting)
Software changes... Applications come in and out of fashion, even new versions of the same software change radically between versions.
When we were in school, we were taught wordperfect for dos because "thats what businesses use", by the time we left school there were very few (if any) businesses left using wordperfect or dos. They were using msoffice 95 on windows 95 or nt4, which is still radically different to the versions in use today.
The differences between 2 versions of msoffice or 2 versions of windows can actually be more significant than moving to linux or libreoffice, and the prevalence of people accessing the internet using smartphones and tablets has shown that people don't actually need (specific versions of) windows to do so.
As soon as you become familiar with something, that software will become obsolete and people will be using something else. Teaching specific software is stupid, you need to teach users how to accomplish their goals with a variety of different programs, and how to identify the functionality they require in any software capable of doing it.
People are not incapable of adapting to changes, they just complain about it because they don't like change. Usually they aren't given any choice, and just end up getting on with it.
Re: (Score:3)
There isn't anything out there that can handle Acrobat, MS office, Outlook, Exchange and other items, other than Windows.
If you list functional requirements instead of specific applications, then there are plenty of alternatives.
The "Big Five" tech companies are Amazon, Apple, Facebook, Google, and Microsoft. Only one of them uses Windows as their primary OS.
Re: (Score:2)
Microsoft is a given.
Maybe 15% of Google uses their version of Ubuntu for desktop. FB and Amazon are probably in a similar boat. Apple, well, would be stupid not to use their own OS.
If you're talking the Server OSes, you might actually be right
Re:News stories: Intel and Microsoft spyware. (Score:4, Interesting)
Windows as a primary OS is somewhat vague. Server or Desktop?
For 4 out of 5, that would be "neither".
Maybe 15% of Google uses their version of Ubuntu for desktop.
I have been to the Googleplex many times. I have seen plenty of Linux desktops, and plenty of people using Macbooks. I have never, not once, seen anyone working on a Windows desktop or laptop for anything other than testing.
Re:News stories: Intel and Microsoft spyware. (Score:5, Interesting)
There was a story a few years ago about how google banned the use of windows desktops on security grounds. If you needed a windows desktop as a specific requirement of your job (testing, dev etc) you had to be able to justify it.
But it does make quite a statement, the more technically oriented a company is the less likely they are to be using windows.
Re: (Score:2)
Which one? Asking for a friend...
Re: (Score:2)
I know now that I'm responding with a viable alternative you'll shift the goalposts but macOS ticks all of those boxes for you
Acrobat Pro - check
Microsoft Office - check
Exchange support - use either the in-built Mail.app or use Outlook 2016 from MS Office
AD - check - you can bind to AD straight out of the box.
What else did you want?
You didn't say within my own company. (Score:3, Insightful)
Biggest one I've seen is -- duh. [marketwatch.com]
Wow! Equifax. (Score:2)
Equifax Faces Mounting Costs and Investigations From Breach [nytimes.com].
The Equifax Breach Was Entirely Preventable [wired.com]
But still, the Intel and Microsoft spyware seems more destructive.
Don't be like Equifax (Score:2)
Don't be like Equifax. Don't hire a music major to be your CSO. Hire a cyber security major with 20 years of relevant experience. Someone whose knowledge is evident from years of posts about security you can see on your favorite tech site.
Btw, I'm a cyber security major with 20 years of security experience.
Re:Don't be like Equifax (Score:5, Insightful)
Don't hire a music major to be your CSO.
The best CIO I've met was a music major, he even played in a symphonic orchestra for a while. He got started in IT because he wanted to design his own DSP.
Some people are a lot more than their diploma.
Re: (Score:2)
Some people are a lot more than their diploma.
Indeed. But CIO != CSO. While a CIO is more a manager than a technician, the CSO job is much more specific and demands strong and targeted technical skills.
Re: (Score:3, Insightful)
No role starting with a C demands any skills beyond salesmanship.
Re: (Score:3)
Re: (Score:2)
Failing to manage customer expectations (Score:5, Insightful)
Practically every mistake in IT is recoverable, except for failing to manage customer expectations.
In particular the two ways in which I can specifically think of that lack of customer expectation management becomes a project killer are lack of solid requirements (e.g., constantly changing requirements), and mismatch between the developer's idea and customer's idea of what the deliverable should look like.
I think that the requirements one is the worse of the two because it is so easy to have this conversation:
cust: Can you just add in this one little change here? ....
dev: Sure thing
cust: While you're at it
Code Complete covers this pretty well with the analogy of building a house. "Moving" a wall is really easy when the house is just a drawing on paper. It is considerably more difficult once the foundation is poured, the walls are up and the roof is on. People building houses know that asking to move a wall in the later stages means lots of money and time on the project. However, because software is an intangible and you can't see it taking shape in the same way as a house it is much more difficult (for someone who is not a software developer) to appreciate that things that seem simple might actually be major architectural tasks for the project.
Re:Failing to manage customer expectations (Score:4, Informative)
Practically every mistake in IT is recoverable, except for failing to manage customer expectations.
Ok, let's see:
1) Threatening to expose hackers AND using the same password everywhere including in your unpatched CMS (HBGary Federal)
2) Botch manual deployment of a trading algorithm and lose $440 millions in 45 minutes (Knight capital)
3) Do not handle race conditions properly and expose patients to doses of radiation 100x higher than expected (Therac-25)
and the list goes on...
Sales Engineer (Score:5, Insightful)
Always make sure your sales team includes a dedicated engineer. They will help keep them in line, and mitigate situations in which a customer was promised something like running Internet Explorer on AIX.
Re: (Score:2)
Re: (Score:2)
promised something like running Internet Explorer on AIX.
Well, that's maybe not possible to do, but, having a customer requesting this insanity is also not possible.
Re: (Score:3)
If I can get you to sign that you won't be paid before it's implemented, rest assured I will not only request it but also make it non-negotiable if I notice that you're too dumb to know.
Re: (Score:3)
Make sure the 'Sales Engineer' isn't a lying weasel like the rest of the sales team.
Having one and having him do his job is all about how he's paid. If his commish structure is exactly like the other sales weasels, expect him to weasel with the worst of them. Only he's worse, he should know better.
Failing to expand with the business (Score:5, Interesting)
The big one I've seen with my current employer is that they've failed to expand their IT staff as the organization as a whole has expanded. The predictable result is that nothing but the most urgent requests gets handled promptly, and minor problems fester indefinitely.
Taking management advice from Slashdot (Score:2, Insightful)
Is definitely a mistake. Lol
Voting for Trump (Score:2, Funny)
Look at all the boobs in charge, and look at all the places no one is in charge. It's a fucking looney bin.
Hiring brogrammers who push out real talent (Score:5, Interesting)
I recently worked for a startup with a real big asshole brogrammer who never was in the office and always was misunderstanding what was going on. Eventually I caught on to these charades. He was also lying to investors about our startup actually containing AI when it couldn't have been farther from the truth. To cover my own ass I sent a text message to the CEO warning him that we were committing fraud. After that the mark was on my head. Our "brogrammer" calls me into a room and says that I'm toxic. I counter by asking him his working hours and if he understands what fraud is. Over the next few weeks regardless of what happened I was ripped into a room and told how toxic I was.
Of course I was let go shortly after because the dumb CEO who always called me "his brother" was well aware of the sham and apparently didn't care. The crazy sociopath actually thought that I would still be friends with him after being fired. He also thinks it's some point I will come back to work for him, likely whenever that brogrammer finally leaves. I wonder if he knows that I'm planning on telling the FTC and his investors about this "AI" company. It's a bunch of regex matching natural language to appear as if you are speaking to a digital assistant. They're actually telling customers that this is a real AI system.
I would call this a serious mistake because their entire future is essentially in my hands. Since this Psychopathic CEO thinks I'm his friend and going to keep the lid on this, he's just plodding along blowing his money on other endeavors. I'll just let him build a little bit more of a paper trail for me before I strike. That's what you get for listening to the brogrammer.
Re:Hiring brogrammers who push out real talent (Score:5, Insightful)
I hope to god you have some actual proof and can take it to the FTC. If you don't and you file a complaint, you might well be facing a lawsuit on any number of grounds. To be completely honest, I don't expect this to play out well for you
Re: (Score:3)
I wonder if he knows that I'm planning on telling the FTC and his investors about this "AI" company.
Nobody likes sour grapes. Learn from this and move on, otherwise you're just sentencing yourself to a lifetime of bitterness and angst.
"Whoever is righteous let him be righteous still
Whoever is filthy let him be filthy still
Listen to the words long written down
When the man comes around"
https://www.youtube.com/watch?... [youtube.com]
Too many to list (Score:4, Interesting)
I've seen them all, but "buying products or services from Oracle" ranks pretty high up. Or more generally, putting faith in a vendor because of a glossy ad in "CIO Magazine" or somebody in management getting kickbacks. Nontechnical managers are incapable of making these decisions, but want to feel like they're in control, so they try anyway.
"No. 1: Vendor lock-in" like buying Microsoft? (Score:5, Informative)
In about March, we started moving everything to Microsoft, and they audited us in August. About $250k worth of internal time later, they gave us the final bill. We didn't know, for example, you couldn't run Visual Studio Professional on Amazon on nondedicated hardware. Amazon charges $2.185 per hour for that which is $19,140.60 extra per year. We're paying $1,199 per year already for VS for every developer, so we assumed we'd be allowed to use it with no extra charges. We were wrong. I think the total bill after the audit was over $130k plus the extra almost $20k per year on Amazon. We don't even yet use Windows for production(customer facing stuff)!
Re: (Score:2)
It's way cheaper to run it on Azure. Gee I wonder why?
If I had to pick one (Score:5, Insightful)
Not implementing and operating a data backup system properly.
I have been bit by this myself and I thought I was doing a good job at it. (I'm not an IT manager -- I'm a software engineer who often gets shoved the IT manager's job for one reason or another.)
Almost every other failure can be mitigated but not this one.
Re: (Score:2)
Not implementing and operating a data backup system properly.
I have been bit by this myself and I thought I was doing a good job at it. (I'm not an IT manager -- I'm a software engineer who often gets shoved the IT manager's job for one reason or another.)
Almost every other failure can be mitigated but not this one.
I think data backup is very important. What is even more important is making sure you can do recoveries from the backups. (And keeping older backups off site) Lessons learned the hard way at my former employer.
Re:If I had to pick one (Score:4, Insightful)
My favorite quote is "Any idiot and his dog can write a backup program. However, it takes a real genius to write a restore program."
Yes, it's super easy to write a backup program. You can come up with one in 5 minutes in shell script. But if you want to be able to recover the data, it takes a lot more work and thinking. Heck, anyone can repackage Hello World as a backup program. (Make it say "Backup done!" instead).'
By testing recovery you're making sure your process works (make sure it's documented, because in the high-adrenaline environment of the server is down fix it fix it fix it, you want a solid monkey-dumb method that lets you get the server up and running again even if you have zero brain function left (distractions from people calling for a status update every minute, to maybe that alcoholic drink wasn't a good idea, etc), but you're also making sure the backup did something - there are way too many stories of everyone happy in knowing the backups were being done, only to realize that it broke six months ago because the tape drive jammed up or a staging server went offline, or other thing.
Re: (Score:2)
Almost every other failure can be mitigated but not this one.
Oh, we mitigated around it. Thank $deity that Bitcoin was only about $700 back then.
Focusing on Employees (Score:2)
When a company focuses on taking care of their Employees, then the employees can focus on getting work done, getting the client taken care of.
Part of that is keeping employees skilled. You might think the worst that could happen is that they outgrow your company and move on because they've become highly skilled. That's not true at all. They could be underskilled and stay right where they are ...until the company goes under.
Like this? (Score:5, Insightful)
Arriving at the new hospital data center (a room 500 ft from the old data center) before anyone else (including IBM), doors open, snowing a blizzard out at 25 degrees, calling the CIO and asking him if he'd been in the room lately.
Nope.
Asking him if he had the electrical contractor's number Call it. There is not s single power outlet visible. Not one.
Yeah, they closed up the walls and painted, the electrical sub never got called.
Extension cords. Frantic 220 installs. Mangled sheetrock. The AS/400 came up about 5:30pm. I was secretly pleased our NetWare cluster was in failover...
We got done about 10:30pm on a Sunday night. No one every asked if this was an IT blunder or or a contractor blunder, but I never discussed it with the CIO , ever. He paid the overtime. My boss was litersally, genuinely speechless, a first for him. This was the same client who had a Token_Ring network that would beacon furiously on a regular basis. IBM took three months to say they couldn't do anything with the CAUs/LAMs, and they should come out and be replaced with switches. Took me asn entire afternoon to find the loose DB-9 interconnect on an 8230 chassis, the ones that were welded on back then. Bolted it in place, problem solved, we did put in the T-C switches during the move. I credit Laura Chappell, her presentation on Token networks at Networks Expo when she was with Novell, and Lanalyzer, for making me a lot of money. Thanks you, Laura!
Now there was the client who, after much analysis, believed his app vendor and replaced our 16MB Token-Ring network with 100Base-T, since they were adamant that Ethernet would outperform Token. This required recabling, drops from the ceiling, because we had reused the existing Cat 3 PDS in floor trays, but that wouldn't do for 100Base-T. No, it made no difference. The vendor them blamed NetWare and AdvantageDB, and in came the NT 4 server. The IT supervisor was the owner's son, but that's not why I questioned his competence.
I don't know how that came out because they wouldn't use us for that, we were a 'NetWare shop', despite my finishing my MCSE. Fine. I know the new guys presented migrating NetWare to NT at our Novell user group two months later. That's how it was back then. Feh.
Re:Like this? (Score:5, Interesting)
A month or so after I was "promoted" from lowly developer to "Systems Infrastructure Manager" during a whole-scale move from an old green screen AIX based system to a brand new in house custom rewrite in modern tech, we had some of the new replacement hardware onsite and being built up (although the replacement applications werent ready to go, but thats not important to this story).
One friday, the UPS support contractor came in to do his servicing of the UPS - that went well, he finished up and switched it back from "bypass" to "protected". That triggered a surge on the electrical supply to both server rooms, which took the AIX box off line. Due to the nature of the green screen application, there was no way for it to be high availability - the data couldnt be replicated in real time, it didnt even talk to anything other than its own binary database files...
A few hours later, the corrupted AIX box was restored and ready to go - by this time, the company (a busy call centre) had been on manual processes for the entire afternoon. On the advice of the UPS contractor, who said the surge was probably the result of too much load on the UPS at the time, we decided to do a full shut down of the entire system, switch the UPS back over into "protected" and bring everything back up - so we waited until 6pm and did just that...
At 6pm, I threw the switch - and promptly looked over my shoulder at the comms racks behind me in the server room. The comms racks were billowing smoke. The comms equipment was burning. Before I could react, I heard loads of loud pops and bangs - both inside the server room and outside it.
Another surge. This one did real damage - a dozen network switches dead, over 40 PSUs in the servers dead, one server dead outright, and loads of call centre desktops went (loudly) pop.
Panic time. UPS contractor called back in - they gave the UPS a clean bill of health and promptly left, disavowing any responsibility.
The board of directors shat themselves - at that point we didnt know the ultimate damage count, but suffice to say the company was dead in the water to any observer.
Cue a desperate night of testing servers, pulling dead PSUs and swapping redundant PSUs between servers so that each server had at least one good PSU. Comms equipment was harder to solve, having to get some expensive switches from our local shop to tide us over. Desktops were bought from the local consumer PC store to give us enough desktops to run the company.
Ultimately, we were back up and running for 8am Saturday - it wasn't pretty, but it was up and running. 3 of us in the IT tech team worked through the night scraping the bare minimum together.
My predecessors DR plan was fleshed out to the point of "we have a DR site" (a commercial site a town over that we had a contract to use - no equipment there, no plans for how to fail over to it etc etc).
So, on to the management failure....
It just so happens that one of my things "to do" on the following Monday was to submit my DR plan for the "new world infrastructure" to the board, who were having their quarterly board meeting the following week (10 days after the company almost died). It was a modest one, but required some equipment outlay to make any DR event as smooth as possible - kept the same contract with the off site unit etc etc.
They turned it down, said it wasn't needed.
I quit the following week.
Mine... (Score:2)
...was caving and hiring the the admin by boss, the CEO, wanted me to hire, not the one I wanted to hire. Badddddd mistake, but she did end up letting me fire him after a series of highly visible fuckups.
Know your limits (Score:5, Insightful)
Does a medical database have to track phone calls, public and private databases to ensure every person who had a test got their results?
If a person had contact with a professional over a result? That person never got a result and never saw the professional after a test was done.
That a person actually got their results and did not move to another part of the country?
Bring in an expert who has worked with the exact problem around the world and who can make a database work in your country.
Real skills and the local experts get network and database they want.
Have political and gov move in and demand they be allowed to build the network with gov staff and other contractors who have no skills.
Thats how big gov can fail.
Take a project from the gov that has the skills and give to the politically connected private sector.
The contractors have no skills.
Thats how contractors can fail.
Stop using people with no skills. Stop allowing people to work on complex projects who did not pass their exams and got given a decade of social advancement.
IT can work if the right experts in the private sector, gov, mil or as contractors are found.
Stop advancing very average people with no skills into the IT sector every generation.
Find professionals that can understand complex problems.
Sunk cost fallacy (Score:5, Insightful)
Time and again, it's the sunk cost fallacy. A system that an organization might have spent a few million dollars to build is just not shaping up into anything they can use, but they keep at it rather than ditching it and seeing what they can do to change things.
What ironic about this is that I think agile actually encourages the sunk cost fallacy because teams will go "oohhh we can 'pivot' a little each sprint." Uh, no. If it's deep-fried dog shit for an architecture, and design you're not going to "pivot" out of this. It gets even worse when you have a management culture that doesn't understand refactoring; most of the agile teams I've been on have had managers who flat out don't care about technical debt and think they can default on it which reinforces the problems with the sunk cost fallacy down the road.
Re: (Score:3)
Time and again, it's the sunk cost fallacy. A system that an organization might have spent a few million dollars to build is just not shaping up into anything they can use, but they keep at it rather than ditching it and seeing what they can do to change things.
I'll second that. I have worked on a project where literally everyone knew that we were not going to deliver. We had bought a package that did not do what we wanted - not even close. Within the first week of the project we were saying "get something else, this was a purchasing mistake, or let us develop something". After a meeting with the manager and supplier, it was agreed that the supplier would customise the package - which was a surprise as this was almost to the extent of customising a toaster into a
Cloud computing ... (Score:5, Interesting)
... it was actually just RDP to a remote server.
At the first management/vendor meeting, I got to ask the first question: "How will response time compare to what we have now, with our servers in-house?"
"Oh, it will be much faster!"
It got worse from there.
I asked the owner if he knew that light slowed down in a medium and he said he did.
We already used RDP to the desktop and he KNEW about the latency.
Management ignored the red flags I threw on the play and put everything on the cloud against my recommendation.
We were a law firm and a time came when an elderly couple traveled from far away to sign some wills and the goddam cloud was down.
The family law practitioner blew her fucking top and confronted me and told me to implement Plan B.
I told her, "Ma'am, Plan B is Plan A."
It cost a fortune, but they paid termination fees and put everything back the way I had it before they went nuts.
Switching to Windows Server (Score:5, Interesting)
We were on Linux for our file server for a decade and ended up needing to switch to Windows. We were tired of not having a good consultant and Windows consultants were easy to find.
The mistake incidentally was not in switching to Windows per se; it can easily do the added tasks we need of it and Linux could not. The mistake was in thinking the issue was in finding good Linux consultants-- the issue was simply in finding good consultants period.
WWW.FUCKEDCOMPANY.COM (Score:3)
Biggest IT management mistake? Easy. (Score:3)
Going into IT management.
Selecting Hardware before software (Score:2)
I've seen many hardware projects run into trouble because the selected hardware platform is too small for the software. The temptation is huge to start the hardware project early. However, if you don't know how fast the software is, how do you know if the hardware will be fast enough?
If the software is larger than the initial estimate (it usually is), and the hardware started too soon, then the hardware needs to be redesigned (often with software changes) and this is expensive.
Keeping jerk managers (Score:2)
Not firing, correcting, or demoting idiot managers is a big problem. Some managers are excellent kiss-ups, but treat their employees like dirt. Get feedback from their underlings, and if you see problems, do something about it. Sometimes with pressure from above they'll mostly correct bad habits, but if they don't, bootem!
For all my Sun peeps... (Score:2)
IBIS.
Ink jet printer purchase (Score:2)
A place I worked bought thousands of ink jet printers.
Mistake #1. Not getting a supplies contract
Mistake #2. They were a name brand printer will known for clogging cartridges and breaking. (I don't want to get sued so no, I'm not naming names.)
Mistake #3. Bought a long term support contract and paid for it in advance.
Result - they rented warehouse space to store the printers until the warranty ran out multiple years later.
thinking testing and UI are easier (Score:3, Interesting)
There's a widespread belief in IT that test code and UI are easier than other coding tasks.
It's completely false. Both are harder than other coding tasks. Your senior devs tell you to assign these projects to junior devs because they don't want to do them. They don't want to do them because they're much harder than other coding tasks.
It's win-win for the senior devs: they get easier work, and when the junior devs struggle, it makes the senior devs look even better. "oh, man, they can't even write a test suite. Well, I guess I should get the big bonus this year."
Put your best devs on test and UI. Put your junior devs on the simple stuff: backend work.
Organizational problems (Score:3)
It's not the biggest problem, but it's one that I've seen a lot and isn't often recognized: Trying to fix organizational problems with IT systems.
What I mean is, if people aren't doing their jobs because management won't enforce the rules, you can't fix that by putting in an automated system that notifies people that they aren't doing their jobs. If people don't know how to do the job because they're dumb and untrained, you can't just give them a sign-in to online training courses and expect them to catch up.
Things like management and training take effort and attention, and until AI gets clever enough to make managers obsolete, an automated system isn't going to do it.
Practice restoring from backups (Score:4, Interesting)
Make sure you actually can restore. Do it regularly. Restore to different hardware. If using tape, restore using different tape drives. Make testing restores a routine thing. When I was a boss, we did daily backups onto tape and same day read the tapes at the offsite recovery site (about 30 miles away).
During my career, I've seen two restore failures where they'd been backing up for years but the backups were no good and they had no idea.
Biggest management mistake? (Score:3)
Doing business with Oracle.
And Microsoft.
Two words (Score:2)
phoenix pay system - Canadian federal employee pay (Score:2)
In Canada the federal government negotiates a pay contract with various unions to come up with the rules for how employees are
Outsourcing desktop with high change fees (Score:2)
SAP - Send Another Payment (Score:3)
'Nuff said
The funnietst (Score:3)
It didn't cause serious company problems but the funnietst management mistake I have seen was...
Manager buys demo copy of some "amazing" speech recognition package with a view to selling to cusomers.
Manager: Set it all up.
Me : I can install it but you will need to set up your voice on it.
Manager: No! You set it up and do that tecchie stuff.
Observation, manager has a strong southern English accent. I have a slight Scottish one.
When he started to use the system expecting me, it made out less than 75% of what he said. I ended up assisting with that successful sale. The manager then spent the time needed talking to the system and it worked for him as I predicted.
Apparently, the customer thought it was quite funny too.
Outsourcing, without a doubt (Score:4, Interesting)
The most expensive, wasteful, credibility damaging, productivity reducing, and sheer chaos producing IT management mistake in my experience was the decision by a certain freight company to outsource IT. The two-letter outsourcing company, who will remain nameless, in sales presentations (which I attended) painted a rosy picture of a "right shore" solution with capable vendor-trained personnel in several call centers across the world, so that no matter what the local time, the call center would be on local daytime, which would help them draw the best talent for the job, etc etc. They were offering best-in-class for a fraction of the cost of in-house IT.
Upper management, who honestly thought that the entire job of IT was to push a button whenever a light came on, bought it hook line and cancellation penalty. As is often the case, they shook their collective fingers at us and told us "you'd better document your job thoroughly before you leave". Devops, my department, maintained a fairly extensive knowledge base, so it was only a matter of printing out all of our written procedures and handing them over (with two hands, because it was a lot of paper)to management of the outsourcing vendor. And, they lost them. So we printed them out again and handed them over. And they lost the second batch too. I'm convinced that they never intended to keep them. (More on that below.)
Our last day was Friday, which was also the cutover day. I had transferred to a department that was being retained, so got to stick around and see the carnage. It was fascinating in exactly the same way a high speed head-on collision between two passenger trains is fascinating. You're retching, but you can't look away.
This was back when Blackberry was still a thing, and all the execs carried one. BES went down Saturday and remained down for two weeks. This was the sign to upper management that things were perhaps not going as swimmingly as advertised.
The helpdesk was a shambles. You couldn't understand them, they didn't know what to do or whom to contact, and major incidents would just disappear in the system and never get addressed. Employees would come to those of us who survived the layoff and BEG us not to make them call the helpdesk.
The outsourcing vendor shook their fingers at us and said that those damned former employees had not documented their jobs well enough. I had a (third) copy of our procedures, and the names of the managers I'd handed them to both other times, and argued that we had in fact made a good faith effort, just ask these people. Only to find out that those managers no longer worked for the company. Curious.
The vendor said they could recover from our former IT employees' incompetence, but it'd Cost More Money. And that was the other shoe dropping.
Some former IT personnel were rebadged, so occasionally stuff still got done. They worked long hours in very stressful situations. Most of them moved on as soon as the economy started to improve.
Promises of a "right shore" solution were absolute fabrications. The entirety of IT, except for those few overworked rebadged employees, was a single call center in lower central India, manned by people apparently plucked off the street, sitting at card tables with IP phones.
The company tried to improve response by sending a number of people over to India to train the personnel there, but ran into an interesting phenomena -- as soon as employees of the outsource service got training, they WENT ON TO BETTER PAYING JOBS. This training effort served to flush out the people with any experience, causing an influx of new people who couldn't find an "enter" key with a gun to their head.
A major plumb for people who got a little experience was getting off night shift, which was our day shift. So as soon as we'd built a relationship with an admin and taught him to do valuable things, he'd brag about how he's finally getting off night shift, and we'd never hear from him again.
Speaking of which, I don't think th
Re: (Score:2)
I believe in your message, especially the AI part.
AI should be used to analyze and predict the behaviour of malware regardless of source and just say, "No!"
You and I know the difference between bullshit and wild honey.
A computer can know that, as well.
Re: (Score:3)
I tell you that this type of attitude is PREVALENT today in IT. Apathy.
I don't think this is an IT problem, I think this is a people problem. Recently we paid an obscene fee to get a report from one of those big management consulting firms. The report had the logo of one of our competitors in the footer, and they made a typo in all instances of our company name in the text. And they sent it 2 weeks late.
Sometimes it feels like I'm in a video game and a majority of people around me are NPC.
Re: (Score:2)
The thing is, offering to pay money to a company should have been the only thing necessary to erase the apathy.
Re: (Score:2)
Maybe it's the old Domino files one has to spend hours on DOS command line to pull out of an Enterprise Vault system.
Now you see why I'm paid the nominal bucks!
Re:IT is NOT a cost center. (Score:4, Informative)
The biggest mistake any company makes? Treating IT as a cost center.
Unless your company sells IT services, by definition IT is a cost center. This is basic accounting.
There was a good Dilbert about this in 1997.
http://dilbert.com/strip/1997-... [dilbert.com]
Re: (Score:2)
perhaps it should be "treating IT as an easily-trimmable cost center"
Re: (Score:2)
perhaps it should be "treating IT as an easily-trimmable cost center"
Spot on. And the corollary for this one is: "treating IT workers as interchangeable."
How often have i heard: let's cross-train the DBA and email admins, or let's have developers specialize in both front-end and back-end work, that way we can move them around as needed. Every time it's a nightmare.
Re: IT is NOT a cost center. (Score:4, Insightful)
If done properly IT is a revenue center. Other departments must internally account for "paying" for IT services. Note it is easy to screw this up, but it can work.
Re: (Score:3)
If done properly IT is a revenue center. Other departments must internally account for "paying" for IT services.
That's called chargeback (or showback if there's no actual money moving around), and it does not qualify in any way or form as "revenue".
"IT chargeback and IT showback (memo-back) are two policies used by information technology (IT) departments to allocate and/or bill the costs associated with each department's or division's usage."
https://en.wikipedia.org/wiki/... [wikipedia.org]
We're not talking about complex financial statements here, this is just very very basic accounting. IT is a cost center.
Re: IT is NOT a cost center. (Score:4, Insightful)
YOU tell a C-Level that he's not going to get administrative privileges on his machine, I dare you. And while you're at it, tell him he has to use the company proxy with the content filter.
Re: (Score:2)
when faced with quality evaluation showing there is no benefit, refusing to not withdraw an opinion or commitment shows poor leadership and a stubbornness that wont benefit business at all.
That's just people being people. Instead of being in their face and expecting logic to win them over, as a consultant your job is to find a way for them to save face while walking away from one of their previous decisions - including jumping on the grenade yourself once in a while. If you become good at that, you'll always make shit-tons of money.
Re: (Score:2)
I once tried to ride a cheap bicycle without considering the gyroscopic forces of two bike wheels spinning at road speed.
The wheels came off, and I've learned my lesson.