Forgot your password?
typodupeerror
IT

How Would You Benchmark an IT/IS Department? 144

Posted by Cliff
from the what-metrics-does-the-bar-measure dept.
ferretworks asks: "Our IS/IT department has been asked by our CEO to find a way to benchmark ourselves against IS/IT departments from other companies with similar technologies (none specific). This sounds like an innocent enough request, but diving into it has made me realize that this is, not necessarily undiscovered country, but a desolate one and rarely visited. So, my poll to the community is: In your Opinion, what is best way to benchmark an IS/IT department and what categories/sub categories would you base your judgment and ratings on?"
This discussion has been archived. No new comments can be posted.

How Would You Benchmark an IT/IS Department?

Comments Filter:
  • by evileyetmc (977519) * on Wednesday May 02, 2007 @07:57PM (#18965333)
    I think the first facet you should look for is uptime / accurate data rates (eg. 1% lost data etc). Beyond that, while being nearly a crapshoot, I think the satisfaction that the rest of your company is getting from your department is paramount - perhaps having a anonymous survey given company-wide to see how you're doing. Also, your upper managers may want to hear numbers such as ROI as well as IT costs as a percentage of revenue brought in...maybe even what revenue would be lost without the department.
    • I'm highly dissatisfied with the IT department in my workplace.

      They recently did an anonymous satisfaction survey across the 4000 employees in the company.

      The survey was USELESS!
      - All of the questions were True/False,
      - Many of the questions were not even applicable to me, (but being T/F, I couldn't put a NA answer)
      - There was no way to adequately express my dissatisfaction on most issues.
      - Many questions were ambiguous.

      If a survey is done...
      1. Get someone else to write it, NOT just the IT department.
      2. Grab
      • Careful, some surveys are not done or controlled by the IT department even if the subject matter is the IT department. This happened to our department and we weren't moved by the development of survey.
        • 1.) Make a random survey, asking insightful questions 2.) Have several emails go around the company stating how important it is for everybodies voice to be heard 3.) Pull metrics that make you look good out of your ass and use those instead of the answers 4.) profit!!
    • by Mattcelt (454751) on Thursday May 03, 2007 @09:00AM (#18971023)
      This is actually a HUGE industry.

      Forrester, Gartner, and IDG all offer advanced comparison data from industry surveys against which you can measure your own company. (Help desk costs per call, fully-loaded employee per hour, etc.)

      Then you implement some sort of metric to use in the comparison. ITIL, COBIT, ISO 17799, and a host of others are available as frameworks you can use, or you can design your own. So you start taking measurements, compile the metrics, and compare.

      [How much does a password reset cost? How much does it cost to terminate a sysadmin? How many staff hours are required to clean a virus infection on one machine? On all machines? &c, &c.]

      I think you'll be hard-pressed to find a Fortune 1000 company that doesn't use some form of this - it's how most companies compare themselves against the industry.

      Hope that helps.
      • by velkro (11)
        I just finished doing a bunch of paperwork for one run by Hackett (we also looked at the other 3 above). There are really too many things to compare, and your question was too vague. Do you want to compare performance (uptime, mttr, etc...), costs (cost per support call, costs per server, etc...), maturity (Standard Operating Procedures, compliance to ITIL, COBIT, etc... ) or maybe all 3?

        In any case, the big 4 consulting companies can help with this... like legal advice, Slashdot isn't a good source for c
  • Downtime (Score:5, Insightful)

    by Conception (212279) on Wednesday May 02, 2007 @08:00PM (#18965353)
    One metric you could use is downtime/loss of work due to IT. This could be because you don't do backups right, to you don't have a test/production setup, to you upgraded to vista without training first, etc etc. Though you'd be reporting stuff you do badly, you can use this for a lot of justification.

    "Email was down about 25 times for up to a day over the last year. This is because we don't have to budget to buy a redundant system. If you give us more money, we can increase uptime." etc etc.
    • Related to this is average time to trouble ticket resolution- assuming you use a trouble ticket system to keep track.
      • by rah1420 (234198)
        As soon as you start measuring ticket resolution time, you won't believe how fast your call center people will find creative ways to close tickets without actually resolving problems. They'll close a ticket when it gets bumped from functionary to functionary, for example.

        "See, we close tickets 50% faster than before!" But the work gets done at the same pace.

        Don't do it.
        • by Sigma 7 (266129)

          As soon as you start measuring ticket resolution time, you won't believe how fast your call center people will find creative ways to close tickets without actually resolving problems. They'll close a ticket when it gets bumped from functionary to functionary, for example.

          That's only the case if you look at the raw data (hence, why things such as 90% service level within 30 seconds for inbound calls is considered bull.)

          The solution is to use adjusted data - anomalies are corrected or prevented before being reported. For example, you can force agents to use a "case closure queue" where another agent verifies that the issue is resolved (or if that's not possible, check if the ticket was legitimately closed.) Other tactics to prevent Georges [google.ca] from disrupting statistics (since

      • by TheLink (130905)
        You better ensure that average time isn't too highly weighted.

        Otherwise the popular solution could be "replace drive with one with a pristine image", "reimage old drive if old drive works".

        Alternatively there could be one of those hardware/software solutions that can rollback to a "pristine snapshot". Reboot machine, select "rollback" during the BIOS bootup, and voila, fresh installed system with the usual patches.

        Sure the symptoms get fixed quickly, but...
        • All users ever want fixed is the symptoms. So fix the symptoms and flag the machine for replacement in the next cycle. For a user machine, that's the most that should EVER be done.
    • Worst. Idea. Ever. (Score:3, Insightful)

      by mcmonkey (96054)

      One metric you could use is downtime/loss of work due to IT. This could be because you don't do backups right, to you don't have a test/production setup, to you upgraded to vista without training first, etc etc. Though you'd be reporting stuff you do badly, you can use this for a lot of justification.

      Why would you do such a thing to yourself? With those metrics, the absolute very best you could do is, 'we didn't cost the company anything...other than salaries and resources.'

      Why not use metrics/goals th

      • Yes... we usually schedule the "maintenance" 5 minutes after we realise we're doing "maintenance"...
      • by TopShelf (92521)
        Nobody said downtime was the only metric to use, only that it is one that has value.
      • by DougWebb (178910)
        IT doesn't add anything to the bottom line. IT takes away from the top line, just like HR, management, administrative positions, the mailroom, etc.

        IT should be measured three ways (from the CEO's perspective):

        1) To what extent is the IT department reducing risks to the business? This covers backups, software updates, choice of software and hardware, ticket resolution (NOT closing) times, etc. All of these should be presented as "To prevent X, we did Y."

        2) To what extent does IT improve the productivity of o
        • by Phillup (317168)

          IT doesn't add anything to the bottom line. IT takes away from the top line, just like HR, management, administrative positions, the mailroom, etc.
          Um... dude... we are a programming shop.

          (Sounds like you've been to too many business classes.)
          • by DougWebb (178910)
            Then you shouldn't call it IT, and get lumped in with the guys who write database Apps for the finance group.

            I develop software for customers, but my team doesn't "add to the bottom line" either. Nor the top line. My team produces the products that the sales and marketing groups sell, and they're the ones that being in the revenue. My team plays a critical role in that, and our cost is justified by that role, but it is still a cost that enables another group to being in revenue. My team cannot, by itself, g
            • What if the same people who do for-profit development are also maintaining internal systems? The two groups are sometimes heavily-crosslinked in smaller companies.

              Out of curiosity: What do you consider for-profit software development if it isn't Information Technology?
              • by DougWebb (178910)
                A mixed group is a mixed group, and should be represented as such. If you spend 80% of your time on external software development, and 20% of your time on internal systems development, your costs should be allocated proportionally. Your accounting department probably has different codes for these different kinds of costs, because they can be treated differently in financial and tax reports.

                I consider for-profit software development as product development. The fact that it's software may make it a subset of
            • by mcmonkey (96054)

              I develop software for customers, but my team doesn't "add to the bottom line" either... We treat the develoment effort of producing new software as a capital investment rather than as a sunk cost;

              The above is why bringing such questions to Ask /. is such a waste of time. What do you think a 'capital investment' is? It's spending money to make money. If your team doesn't "add to the bottom line" then you are a sunk cost.

              Just because the company has a sales team doesn't mean the devs don't add to the

  • by MeanMF (631837) on Wednesday May 02, 2007 @08:01PM (#18965379) Homepage
    If there are benchmarks, then the terrorists win.
  • I think Futuremark has a benchmark for that! Although I hear some IT/IS Department are hiring custom techs just to get high results on it.
    • by mobby_6kl (668092)
      The trick is to set the benchmark to the most demanding mode and run it at least a few dozen times to ensure the most accurate results possible.
  • by m0rph3us0 (549631) on Wednesday May 02, 2007 @08:02PM (#18965385)
    I would start a consulting firm, get the CEO to hire your consulting firm. Spend a lot of time compiling a bunch of numbers, then because the other companies won't want their data revealed by name sort them into averages based on Fortune 20, Fortune 100, Fortune 500.

    Make up an "average" for these three sets in which your company does better in most metrics, take the $250,000 you got from this consulting gig and live on it while you go around with your initial report selling it to other companies.

    This idea is so stupid and useless that only a consulting firm would offer this service.

    • by m0rph3us0 (549631)
      I forgot, once the report is done, do a white paper or case-study on how much money they saved after changed after firing key people in the first month. Neglect to mention that the company crashed a burned 2 years later.
    • by walt-sjc (145127)
      That's enough out of you, Dogbert!
  • TPS Reports (Score:3, Funny)

    by Joe The Dragon (967727) on Wednesday May 02, 2007 @08:02PM (#18965387)
    did you get that memo?
  • One trend out there is if an IT project doesn't really to appear to have lowered costs, didn't really improve profits, and didn't really improve productivity, what was the over-all point? Was it a failure or is tehre some other measurement of success?

    But another measure of success is to measure the increase in Consumer Surplus the IT project/dept adds to the firm. Consumer surplus is one possible way to measure a competitive advantage (because it measures the increased value to your customers).

    Estimating c
    • by jafiwam (310805)
      If yours is like my company, IT projects dont do any of those things, but rather can be catagorized is "filled the gap between the pack of lies the sales department created and what the developers actually said it would do".
  • Survey the users (Score:4, Insightful)

    by Kohath (38547) on Wednesday May 02, 2007 @08:14PM (#18965513)
    I would survey the users.
    • Re: (Score:2, Insightful)

      by wild_berry (448019)
      When you're surveying the users, be careful: remember that the squeaky wheel gets the grease -- in these surveys the people who have nothing to complain about rarely say anything. Do it with the intent of making it easy to propose fixes for issues raised and also to present the data in terms that flatter you and enable you to do a better job. The game should be: work with management, and work with your internal clients.

      It can't hurt to have your report include a count of preventative maintenance issues and
    • by Degrees (220395)
      Similarly, see if your help desk ticketing system can be connected to a survey system. In our organization, every time a help desk ticket is closed, an email goes out to the end user ("please click this link to take the survey").

      The survey includes questions about responsiveness, courtesy, time-to-resolution, number of interactions required, technical proficiency of the service person. It's important to know where you are great, and where you are less-than-great. Over time, you can show how the organizatio

  • my 2 cents... (Score:2, Insightful)

    by asdef (261823)

    Having worked on both sides of the IT vs Business user / programmer fence, I think any measure of the success of IT needs to include some form of customer satisfaction. This is important because IT's customers tend to be internal and most customer satisfaction queries are focused on external customers.

    I've seen battle lines drawn between IT and everyone else, and nobody wins because energies are focused on battling with the other side instead of finding ways to help the company make money. While I und

  • >> How Would You Benchmark an IT/IS Department?

    1. Coversheets per TPS report
    2. PHBs per employee
    3. Salesmen per server

    Higher is better for all metrics.
  • If you are being asked to do that, then the PHB's are looking for a cheaper option. Work on your resume and get out NOW. Otherwise, you will training your replacements.

    Been there, done that, don't have the t-shirt.

    Seriously, they are looking at cheaping labor costs in India/China/Godknowswhat3rdworldhellhole to send your jobs to. If they are asking for a lot of procedures on your work and maps on how it gets done, leave faster.

  • by bergeron76 (176351) on Wednesday May 02, 2007 @08:28PM (#18965685)
    Operations(IT) isn't like sales. It's not exactly easy to quantify achievements, etc. As long as your teams hit their deadlines within the allotted time frames, your department is doing better than most.

  • The less the IT department is noticed that it takes care of things behind the scenes the better. It should be perfectly fluid. They shall fix problems BEFORE they happen. If problems happen out of the blue there should be fast replacements/workarounds that have minimum impact on users and the entire systems.

    And remember EVERY second counts!
  • If you have a little money,ok more than a little, to come up the results from a third party call Gartner (or another big firm) and ask to participate in a study. They will find other companies your size in and out of your industry to compare you against. They use a common and repeatable criteria that you can use in the future again.

    We had one done in the IT department I work in, and while we came in above average in one or two areas it gave us a place to start from in proving we don't just throw money away.
  • Tracking by uptime, as a lot of other posts describe, simply is not enough. If you want to compare success by uptime, your management might as well just outsource your department to a datacenter company.

    A Service Level Agreement can truly monitor your performance as a customer service organization. This idea would measure the soft side of IT support as well as uptime. Also if you do uptime measures, monitor the services, not just ping times. That is simply a waste.

    HEAT is a very expensive system but can
    • by nologin (256407)
      I have to agree here. I have worked as a technician and as a manager within IT/IS departments and I certainly recommend that every IT department understands the basics of a Service Level Agreement.

      If your IT/IS Department doesn't have a Service Level Agreement, then benchmarks are generally irrelevant as they can be easily taken out of context to make your department look incompetent or as a waste of money.

      The good thing about an SLA:

      1. It basically defines what your group is responsible for. It's the perfe
  • Management always want things like this, unless it is a subsidiary its unlikely that you would get any meaningful data you could compare against in another company. Unless they agree to benchmark following the same rules and be completely honest with you... Who would tell you if they had a bad IT department?

    A managers real question is how much money have you saved me or how much easier has my life become because of the IT department.

    Someone suggested asking the users, thats not a bad idea... A simple
  • by Cervantes (612861) on Wednesday May 02, 2007 @08:38PM (#18965791) Journal
    I got a benchmark for ya. "Are things running well?" Yes? Yes they are? Then shut the fuck up and stop asking me to waste time comparing myself to other people.
    (This method of managerial interaction is derived from the groundbreaking book "Shut the fuck up" by Dr Denis Leary)

    Seriously, it's not an "innocent" request. It never is. First, they just want to know how you compare. Then, they want to know why you're not the absolute bestest in the whole universe. Then they compare you against departments with fewer people. "Hmm, seems we could get comparable results with 2 fewer bodies". Then each employee is evaluating their specific performance trying to justify their job. Or you're filling out "Time code sheets" to tell them what you do in each 6 minute block of time. Pretty soon you're hearing about their buddy Steve, who has his entire office and web presence run by one part time summer intern who happily works for $6/hour plus tips.

    Evaluate your performance based on internal expectations. You want great uptime on the web server? Why, we were only down for 2 hours last month. Compared to previous months, that's great! You want quick response to employee problems? Our average response time for properly filed tickets was down %4 compared to last quarter. That's the way you evaluate. If you start giving these fuckers examples of how other companies are doing, they'll cherry-pick and start challenging you to match. "Why, there's a place in Wisconsin that only has 1 tech covering 400 people! I don't know what this 'Wyse terminal' thing is, but surely if just one guy can cover all those people, we're overstaffed here!"

    I know it seems like it could be good. "Why, we're outperforming most other companies, surely they'll see this and use it as criteria to give us better raises/more benefits/better perks". NO. NO, they won't. Just NO. Come review time, you could be leading 90% of the field, and they'll go on about the top 10% and how those guys earn less than you, so they shouldn't give you a raise. Or they'll go on about how other companies have their helpdesk techs do server support too, so they don't need your $80k server admin. If you're underpaid, and everyone else makes more than you, you can show them that too, and they'll nod in an interested fashion, they'll hum and haw, and they'll end up giving you some bullshit excuse about budget constraints. If you make more than other people, expect a pay freeze or cut. Even if fall in the middle of the bell curve, expect them to find a reason to lump you with the folks on the bottom of the curve.

    Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you. Don't let them, don't help them, don't give them any excuse. No matter how much you train them, or explain to them, or draw them little fucking flowcharts in Visio with pretty colours, what you do is still voodoo to them. You push keys on the magic box, typing 10x faster than they can, and you get it to do things they couldn't even dream of, and you do it really easily, and that makes them feel dumb. And just like stupid bullies on the playground, they want to get back at you, subconciously perhaps, but they still do. They want to sit there across from you, at their big fancy desks in their spotless shiny offices, because that's their one time to feel superior to you, to be in control of you, unlike all the other times when you're the magician who just makes miracles happen.

    Don't fall for it. Rephrase their request, turn it around on them, avoid it at all costs. Does your accounting department have to compare themselves to other businesses? Do your managers have to go around finding out how other managers in other companies are doing? Fuck no. Don't let them treat you any different than any other department. They're just doing this because they don't understand what you do. Don't explain it to them, just give them numbers based on internal statistics so they can sit back and go "phew, IT is performing well, I'm happy".

    Just remember folks, any time anyone in management opens their mouth, echo these words in your head:

    "It's a trap!"
    • Re: (Score:3, Insightful)

      by EmperorKagato (689705) *
      Yet comparing yourself to others is how companies improve. This is how positive competition is created. This is how problems get fixed.

      When I first started working at my company there were no solid metrics in place. We had a backup system not being tested. We had a NAS system growing out of control. What happens when we started placing metrics in place? Things begin to change.

      I recall someone wondering why the director would never approve a project until I went to the director with numbers and history (DATA
      • Re: (Score:3, Interesting)

        by bitty (91794)
        It sounds more like your company benefited by following common sense procedure and industry standards, not comparing yourselves directly to other IT departments. Remember, the article submitter is talking about direct comparisons to other departments, not standard procedures or best practices.

        No two companies are even remotely close to identical. Every company has different goals, different organizational structure, different programs they're using, different needs in general. The moment you start compar
        • That same information can be vital especially if you're trying to get an idea on how much your project could cost the company.
      • by Cervantes (612861)
        Yes, Businesses compare themselves to other Businesses. This isn't a business, this is a department in a business, a service department at that. If the services are being provided in an acceptable fashion, that's what should count.

        Additionally, even when comparisons are occurring, management isn't in a position to compare. They simply don't have the expertise. They can compare sales, or throughput, or numbers, but asking them to understand that a backup server with RAID and a DAT tape drive is fundamentally
        • I think this is an excellent opportunity for this guy. With that charge, he can go around on business trips around the world, spending a week at each competitive site to evaluate what their performance is like. Naturally, you'll want a large enough sample size and I think 52 companies ought to do it. One year later, having burned through half a million dollars of travel expenses, hand in your report.
           
          • by Cervantes (612861)
            As much as I dislike external comparisons, your ideas intrigue me, and I would like to subscribe to your newsletter...
        • There's numbers behind the differences in a RAID + DAT solution compared to a using random pc solution.
          • "There's numbers behind the differences in a RAID + DAT solution compared to a using random pc solution."

            Yeah, sure there are. The problem is where are they? A company with a "random pc solution" is almost surely not making numbers out of them (they are el'cheapo after all), and even if they are, you are probably going f* as well as not. Hey, I had this rsync script and a single hughe SATA drive and we haven't had a problem in five years. Total cost of ownership? 1000US$, beat that.
            • by Cervantes (612861)
              Amen. Additionally... management is going to ask you "Well, will this technically work?", and yes, technically this is one possible way to back up data. It's one of the worst possible ways, but it is one way... and as soon as you admit that it's feasible, even though not desirable, they seize those words and suddenly "crappy x86 with a huge HD" becomes the benchmark, and you have to JUSTIFY spending an extra grand on RAID and DAT.

              And then, the next day, you're installing x86 w/ big HD and a script, because
    • by bitty (91794)
      Parent beat me to the punch. It's very close what I was going to say, only said better. Excellent post.

      Don't ever let them railroad you into that benchmark bullshit. It only creates trouble.
    • Re: (Score:3, Insightful)

      by Malfourmed (633699)

      Does your accounting department have to compare themselves to other businesses? Do your managers have to go around finding out how other managers in other companies are doing? Fuck no.

      In my experience, all functional areas are compared to other businesses. It's what benchmarking is all about and has been all the rage for at least twenty years now. IT is no exception. Sometimes it can even be useful! If you're doing well, then it can help demonstrate you're doing well. If not, then it can act as an incentive

      • by Cervantes (612861)

        "Are things running well?" Yes? Yes they are? Then shut the fuck up" [...] Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you.

        And this embodies the attitude that wants to maintain the priesthood at all cost - keep IT mysterious and scary, rather than enlightening the business ... cause that way it makes it easier to justify our toys and inefficiencies.

        Most of us have spent many, many years learning the ins and outs and the finicky little details of IT. You simply can't communicate that properly to someone who's spent their life focusing on a different area. Almost every time I explain a process to a manager, I explain "this is how things go when they go well. If something goes wrong, it throws the whole process out the window". And then, when something goes wrong, they come to me...
        Them- "I don't understand, you said this usually only takes a day...."
        Me

    • by dustpuppy (5260)
      The sign of an incompetent department/person is one who doesn't want to be benchmarked.

      Look at it this way:
      If you are better than the rest, you should make sure everyone knows about it and ask for a payrise.
      If you are doing worse than others, then you better improve.

      It's the latter situation that most people such as the parent end up fearing (and hence ranting about). Simply, they are scared to change their current work practices - they are content doing a mediocre job. They are the sort who spend enormous
      • by Cervantes (612861)

        The sign of an incompetent department/person is one who doesn't want to be benchmarked.

        Not quite. An incompetent person wouldn't want any guidelines or deliverables at all. Nothing to live up to, nothing to fail at. I've got no problem having deliverables. If my boss was even moderately technically competent, I might even trust him to make some limited judgements in comparing things to outside situations.
        But most managers aren't. Most managers have no idea why it would take longer to install and configure Win2k Server than it does to run the Recovery Disk on their HP at home. That's why inte

      • by Scarblac (122480)

        Look at it this way:
        If you are better than the rest, you should make sure everyone knows about it and ask for a payrise.
        If you are doing worse than others, then you better improve.

        Except it isn't that easy, is it? For one thing, the test that gives a correct benchmark of my entire job does not exist; at most it'll measure some small parts of it, badly. Secondly, nobody else in the company has exactly the same job, so it'll always be comparing apples and oranges, but that won't stop them. And thirdly, t


      • The sign of an incompetent department/person is one who doesn't want to be benchmarked.

        But the act of benchmarking takes TIME. And ENERGY.

        And the last thing that people want to part with - people who are doing useful, productive, mission critical work - is time. Or energy.

        Bean counters & suits have plenty of time and energy to devote to nonsense like this. People who actually do something productive with their days do not.

        Consider classical computer coding.

        Say that writing the code to satisfy
    • Re: (Score:2, Insightful)

      by GovCheese (1062648)

      Management doesn't understand you or what you do.

      That's your fault. Once a day I go to a different office, a different section or division head, and to their staff offices and ask, "Hows the IT today? Everything working alright? You have everything you need?" And I listen and get back to them. If they trust that you care, then they'll start listening to you. And if they trust you, they won't slam you the first time there's a mini-crisis on their desktop. The best survey is done over and over at least on a weekly basis face to face. It's not a wri

      • " Once a day I go to a different office, a different section or division head, and to their staff offices and ask, "Hows the IT today? Everything working alright? You have everything you need?""

        Then some PHB realizes that all you do is wandering here and there and that your wages will look much better on his anual bonus.
      • by Cervantes (612861)
        I disagree. I know everyone in my division (300+ employees in 5 cities) by name. I speak with almost everyone on a frequent basis. I, too, spend time stopping by people I haven't seen in a while, catching up, checking to see if their computers are behaving properly, or if there's something that could be better. Same with the managers. It's not a question of effort or explanation.

        Let's put it this way. If my mechanic explains to me why my car isn't running, I can nod, I can pick up on keywords, I can even un
    • Re: (Score:2, Insightful)

      +5 Insightful? Sheesh. I considered not responding to this because I felt it was an obvious troll. However, with the mods not thinking so, I reconsidered. I hope I was right the first time.

      I got a benchmark for ya. "Are things running well?" Yes? Yes they are? Then shut the fuck up and stop asking me to waste time comparing myself to other people.

      Exactly. Because an organization should never strive for better.

      Seriously, it's not an "innocent" request. It never is. First, they just want to know how you compare. Then, they want to know why you're not the absolute bestest in the whole universe. Then they compare you against departments with fewer people. "Hmm, seems we could get comparable results with 2 fewer bodies". Then each employee is evaluating their specific performance trying to justify their job. Or you're filling out "Time code sheets" to tell them what you do in each 6 minute block of time. Pretty soon you're hearing about their buddy Steve, who has his entire office and web presence run by one part time summer intern who happily works for $6/hour plus tips.

      There no doubt are places like this, and you would do well to exit such an organization sooner than later. Trying to avoid accountability or fighting audits only make you appear aware of and afraid of your own incompetence. If you are providing no be

  • by blhack (921171) on Wednesday May 02, 2007 @08:38PM (#18965801)
    (Monitors on Desk * cups of coffer per day) - (downtime of mailserver + vista installs) / (|the temperature of the server room|) = Productivity

    x2 multiplier for Linux install on desktop
    x5 multiplier for BSD install on desktop + 1 free very much needed hooker
    x10 multiplier for *nix install on managements desktop


    Lather rinse repeat for each member of IT department.

    Add them together, if the department has somehow manipulated these values to = the number 42, give them all a free car If all of these values are stored in text files encrypted with 4096 bit encryption and spread across a 3x redundant cluster of 10 load balancing servers so that this value can be calculated constantly to the 10 millionth digit....

    ....pray
  • Take a look at how mature the strategic alignment between IS/IT and the business is. Unless your business is actually IT, IT and the business usually sit on opposite sides of a fence. It seems to be a reasonable way to rate both IT and the business. http://cais.aisnet.org/articles/default.asp?vol=4& art=14 [aisnet.org]
  • The more your staff makes the better you are. By paying the highest salaries you attract the most qualified and talented IT staff around. Just tell them it is the free market at work. That's economics, CEO's understand that stuff.

    Also explain that another useful metric is the numbers of hours worked. If salaried IT staff are working fewer hours it is because they are operating more efficiently and doing a better job of earning their salaries.

  • You cannot benchmark things that are nstill evolving fast. If you have the good fortune to have some really good IT people on your staff, do anything to keep them happy. They may just save your neck.

    Typical example: A really good dystem administrator catches most problems before others notice. Consequentially these people are often sacked by clueless management, because "'there was no problem''. Face it: IT is expensive and still experimantal today. And you absolutely need it to work.

    If you really need benc
  • Metrics (Score:2, Informative)

    by realperseus (594176)
    At the firm I'm employed by, we measure on several items in several ways. We have many departments under the IT "umbrella" so I'll try and break them out and briefly explain each + expectations.. BTW, we employ 1000 users.

    Helpdesk (4 employees): Their mainly measured on how many helpdesk "tickets" they can close without sending the problem off to a 2nd level tech. They can reset passwords, add printers, etc.. They can remote also into users desktops to provide assistance. Their resolution rate of about

  • and then compare yourself to it.
  • A meta-metric on this issue is easy.

    Start your stopwatch when you start working on it, and stop it when done.

    Do that for the duration of the project then add the total time all up.

    File the report under "waste of fucking time"

    Last time I checked, RAID arrays don't fail in sympathetically because the one next door did. They do or they don't. Your metric should be "did shit break?" and "did it get fixed fast or rendered irrelevant due to failover"? "Yes/No".

    What the guy next door does is irrelevant and tryi
  • by Whuffo (1043790) on Wednesday May 02, 2007 @10:55PM (#18967167) Homepage Journal
    When management is looking for benchmarks / SLAs / metrics to judge IT performance by, you can be sure of two things:

    1: Management is clueless as to what the IT staff actually does. 2: Management is looking for ways to reduce headcount.

    When you see this nightmare raise its head at your place of employment - be worried. Especially if you're at the top of the pay scale - you'll go first.

    If you see this at the same time as there's handwringing about the cost of health benefits, don't just get your resume in order, start looking for another job. Even if your job survives the reorg that's just around the corner, you'll be working twice as hard.

  • IT is a service department (something many bigger IT departments forget) and their primary goal should be to keep the customers happy, within obvious budget constraints. If I were benchmarking IT (or HR, or accounting, or any other internal service organization) I would simply benchmark the level of satisfaction by their internal customers.

    I think comparing with other companies is a waste of time simply because no two companies have the same requirements or specific networking setup - usually for historica
  • Use something like Remedy for problem and change tracking. When users report problems, you write and document an audit trail and mark the problem resolution and put the root cause for the issue and the downtime suffered and impact of the issue. Then you can take fun statstics to see your downtime, how important it was, the components in question, and whose problem is was.

    Obviously the hard part then is saying when everything's good enough, but knowing most corporations, it never is. There's always somethin

  • If your janitors are working hard all night long, the execs will come in, see how clean the office always is, and wonder why they have the janitors. They will fire them, (or replace them with cheap idiots) and only after the office turns into a filthy whole will they realize their mistake.

    If IT is doing its job right, the normal day-to-day users should almost forget its there! Updates and upgrades are smooth and invisible, problems are fixed proactively, instead of reactively, and people will wonder why t
    • Sounds like the I.T. department of your company may need to sell themselves better.

      Good I.T. helps them make money and not just save it by working computers.
  • by Organic Brain Damage (863655) on Wednesday May 02, 2007 @11:37PM (#18967515)
    If the CEO's asking you to benchmark your IT group, do not JUST measure your uptime or other simple availability metrics, that's just silly.

    Either you're reliable or you're not. But reliability, while important, is only part of the picture.

    For struggling, barely competent IT guys, maybe reliability is all they can hope to achieve, but let's assume that's not you and yours. Let's assume you've got reliability down pat. Why do I make this blithe assumption? Because we can purchase reliability off the shelf today for not very much money. Windows XP boxes on programmers desks regularly go 2-3 weeks without requiring a reboot. Servers can run even longer if properly configured, fed and cared for, even running Windows.

    So, where do we go from 99.999% reliable with an IT department? We start consulting with our business managers. We work with them as a partner to help select and adopt technology that will make the business more efficient and/or more effective. And we benchmark these efforts by measuring the productivity of the business function before and after the new technology/system is deployed. This is where we show the strategic value of IT. Not in uptime.

    And, we take into account cost. If we can get 99.999% reliable and your competitors IT group can get 99.999% reliable, the budget becomes the tiebreaker and the winner is the one that does it for less money.

    For instance, if a business is indexing large document sets for the litigation support market and we can figure out how to move that function from Dallas to India thus cutting labor costs by 60% without a loss in quality or responsiveness by cleverly deploying networking technology and low-end PC's into Madras, we have provided an easily quantifiable cost structure improvement which can give our company an important competitive advantage. This is an example of the big payout from IT. The other big one is quality improvement. And the really rare one is opening up a whole new way of doing business...disruptive technology.
  • BogoMips. What else?
  • by NeutronCowboy (896098) on Wednesday May 02, 2007 @11:50PM (#18967627)
    I do this for a living, so you'll probably hate me for some of my suggestions....

    Number one thing I'd like to get out of the way is that the people who qualify this request as a waste of time are idiots who shouldn't be in an IT department. I've been to too many places where no one can tell me what they're running, where the servers are, what patch levels they are at, which machine is up or down and what they need to keep pace with growing demand (your business IS growing, right?). IT departments like these are the root cause for bloated IT initiatives, downed web servers and crappy customer service.

    Number two is the realization that the IT department is in the business of the keeping the rest of the business up and running. This means two things: your responsibility is to keep the rest of the company happy and productive, and you get to charge them (system doesn't matter, as long as you keep track of what it means to service 200 requests a day for lost passwords) when you perform work to keep someones servers up, the mail flowing and the network lit.

    Once you understand the place of IT in a business, metrics are easy to come by:
    - how well you're doing is measured by how available your servers and your apps are.
    - how efficient you are is measured by how much money you save the rest of the business in avoided downtimes and increased productivity.
    - Bonus if you keep track of how long it takes to service requests and what their resolution was.
    - Extra Bonus if you can do performance forecasting and figure out when you need to expand your infrastructure.

    The performance of an IT department is NOT measured by any of the following:
    - budgets of IT departments in other companies
    - how many new products you've rolled out
    - how many people you fired (or hired)

    Anybody who suggets those needs to be smacked around and told not to speak in public anymore.

    With that in mind, there are a million products to track your statistics. My company will be happy to quote you anything from a 4-figure to a 10-figure deal for this. There are open-source tools that will do similar things for free (though in my opinion, you get what you pay for). There are in-house and hosted answers to any of these questions. But the one thing you need to remember is that you need to know the answers to the CEOs questions about what the status of your machines and your people is. If you can tell him that your hardware uptime runs at 99.99% for mission-critical servers (the Oracle RAC that holds your financial information, for example) and your app availability is 99.7% during business hours, that it takes you 4 hours to respond to a priority 1 request and 3 days to a priority 3 request, and that based on current response time trends, you need to double the processing power and database space in 6 months, you're golden.

    If you can't tell your CEO these things, you will be replaced by someone who can. Or even worse, your job will be outsourced to my company, and I get to work out these things while you're flipping burgers. Yeah, I'm being a jerk. That's because I'm flabbergasted at some of the comments (and their moderation) and how their authors still have a job. IT metrics are simple, and don't have to start with complex SLA measurements and other crap. You can start with a basic ping monitor of your critical servers, and go from there. But for heaven's sake, do something. You'll be a hero if you play it right.

    Oh, and just to repeat - do not benchmark yourself against other companies. You don't have access to valid data, and you won't find an identical business. Instead, find out what your other departments need from you, and benchmark from there.
    • If you cannot measure, you cannot improve... or at least you cannot prove you improved.

      One thing to watch out for: System uptime and application uptime are different things! A good program on a squirrely system is no good, and a solid box with flakey software is just as bad. Network is also different altogether, if that is unstable nothing else is stable. Again, networking is a well understood problem and you should not have to worry about it if you have good people.

      Making the OS stable is fairly well under
    • Despite my statements elsewhere under this topic, I agree with most of what you say. We measure all that stuff. For example, we answer priority 1 requests within an hour and every hour one goes unresolved results in a personal, automatic memo to the next higher level of management. Leave a priority 1 open and unresolved for 5 hours and it's literally on the immediate to-do list of the CIO. One more hour and the Commissioner gets involved. Priority 3, an individual work stoppage, is addressed in 4 hours
      • Eep - SLA discussions are the hardest part.... which is why I really didn't mention them. I don't have any reference material for that either. The best I can do is point you to standard negotiating techniques: things like understanding first that this is a common effort, that there are personal issues at work as well as professional issues which all need to be resolved, and that there needs to be a give and take. Basically, what I would recommend is to first find common ground, and then work from there. In
  • First off, as many people have commented, you don't want to blindly give management what they are looking for here, but you also don't want to ignore such a situation, as they are probably trying to justify the cost of IT.

    You need to turn this around in favor of your department, and it would be a very good idea to take a look at what metrics you can apply that will serve a twofold purpose - to set a baseline of current performance, and to set a moving target for constant improvement. In other words, the thi
  • by Billly Gates (198444) on Thursday May 03, 2007 @01:16AM (#18968275) Journal
    How much money does your inhouse ERP apps bring? How about your wharehouse database that can be used to cut down misplaced inventory? How about your accounting apps that help the CFO find out which areas are hte most profitable?

    Forecasting supply/demand apps?

    These all make money and help streamline business processes and work with the CEO and his company.

    Thats what CEO's want to hear and what a good I.T. department does in a fortune 1000 company.

    Not that your high tech janitors like another poster mentioned, which in this case would mean its time to either move to India or update your resume because your a cost center.
  • First of all what is your SLA? You should measure yourself firstly against the SLA in place. If you are way out you need to figure our why and/or maybe adjust the SLA. Do you offer a standard 9x5x5 (9am to 5am 5 days a week)? You need to have a clear cut definition of what services and at what level you will supply those services. Does your company outsource code infrastructure such as core network and telephony to another company? If so how does their SLA measure up to yours? Your SLA should be lower their
  • Do they keep your shit fixed?
  • I can think of a lot of ways to do it. The best practice (yeah, guess what...I'm an IT consultant) is to look at the ITIL library, and ITSM in general. ITSM is essentially a set of practices around helping business units tell IT what they want, in useful terms, and measuring the definition of success in how IT fulfills those wants/needs. ITIL is a library of information that relates to ITSM.

    But even if you get all that working at your end (and it's not a tiny effort), how are you going to possibly compar
  • by BenEnglishAtHome (449670) * on Thursday May 03, 2007 @09:57AM (#18971741)

    First, all the posters who are saying "It's a trap. If they're delving into this, you're gonna get shafted down the road." are absolutely right. Pay attention to that.

    Second, different tasks get measured different ways, so I'm going to concentrate on one aspect, front-line support. That's what I do.

    When my upper management got on a kick about assessing skills and measuring performance, my local manager starting doing her best to protect us from the crapola and just let us do our jobs. She's great. Still, I felt the need to pinch-hit for her so at a recent meeting with the big executive who's far enough up the org chart that we almost never actually see her in the flesh, I gave a little speech that went, approximately, like this:

    "You want to measure our performance? Fine. For the last several years, you've made a big deal about how we're a support organization and our job is to make sure other folks can do their jobs. So measure that. Assign me a hundred employees (The target user-to-support tech ratio in my organization is currently 113-to-1.) with roughly similar jobs or, for smaller posts of duty, just assign me the entire office. Tell me my job is to keep those people happy. Then randomly and continuously survey those people and ask them if I'm keeping them happy. If I am, I'm doing my job. Nothing else matters. I don't care how many numbers you aggregate off of how many closed trouble tickets and how you compile them and present them in fancy charts and feed them into service delivery models. None of that means anything. All the numbers you're seeing are presented to you by analysts who've never actually been in the field fixing things and dealing with users; you can't trust a thing they say because they don't know what they're talking about. If my customers and my manager think I'm doing a good job, then I'm doing a good job. What I want to know is: Do you have the testicular fortitude to abandon all those meaningless performance metrics and actually start measuring performance by, you know, actual performance? "

    Two side comments: My manager is about to give me a perfect performance rating for the year. I know she's going to do that because she had me write the thing. I'm getting it less than a month after that little speech. Also, for you young whippersnappers who think everything in government is bad and anyone who spends more than 6 months on a single job is a loser, I'd like to point out that the speech cited above is a perfect example of why it's great to work under civil service protections and even better to work there long enough that even if management tried to fire you, you could just tell 'me to FO, retire, and take your pension.

    I love my job and I do it well; that's why I'm willing to stand up for it. Stand up for yours, if you've got the balls. NOW is the time to start; once management starts measuring the wrong stuff in the wrong ways, you're screwed a dozen ways, none of them involving a happy ending.

    • by illumin8 (148082)

      I love my job and I do it well; that's why I'm willing to stand up for it. Stand up for yours, if you've got the balls.

      That's nice that you love your job and are supposedly willing to stand up for it, but I think you're giving some bad advice.

      Also, for you young whippersnappers who think everything in government is bad and anyone who spends more than 6 months on a single job is a loser, I'd like to point out that the speech cited above is a perfect example of why it's great to work under civil service prote

      • Excellent points. Really excellent points. You made me stop for a minute and consider just how fortunate I am. And, also, how much I need to add a little more detail so that people don't get the wrong idea.

        So you just said you are willing to stand up for your job, but you also said that you're a civil servant that can't be fired

        No, I can be fired for doing something outrageous. I can be escorted right out the door and lose my pension. But I can't be fired for pissing off some exec by saying thing

        • by illumin8 (148082)

          Excellent points. Really excellent points. You made me stop for a minute and consider just how fortunate I am. And, also, how much I need to add a little more detail so that people don't get the wrong idea.

          You know the reason why I wrote the response was that I see a lot of the same personality traits in you that I see in myself. There is some saying that I can't remember right now about seeing the same faults in others that you have in yourself :-) I too have been known as the go-to guy on my team on mor

  • There were some good points made in the responses, but they still missed a lot of the 'how to really do it".

    Everyone talks about benchmarking or comparing themselves to others. The first step in this scenario is being able to communicate what your organization does, then how well it does it. If you can't do that... forget benchmarking, and forget being able to validate your' IT organization's value.

    It is most important to first look at the work being done. Even if your processes/activities aren't for

  • I'm currently involved with benchmarking a county government IT department. There has been a change in the leadership, and we now have a boss who cares about using technology effectively. Here are some things we are looking at:

    Do employees have access to reasonably modern technology? Answer: probably not. We're still using a 10-year old version of lotus notes, and we don't have any standard procedure for deciding who gets the new desktops.

    Can the different department computer systems work together? No. Beca
  • My tounge-in-cheek response is that you need to look at how fast your hardware is compared to how fast it should be.

    Specifically, I've been in many situations where my company-provided hardware was too slow due to the company-provided security software. McAffee is the biggest offender, it'll needlessly make me wait for minutes while it trashes my hard drive.

    You know that utility that automatically updates everyone's desktop? Guess what, making everyone reboot in the middle of the afternoon cuts away hal

Luck, that's when preparation and opportunity meet. -- P.E. Trudeau

Working...