How Can You Measure a Wiki's Worth? 78
moldar writes "I have been involved in a major project to migrate documentation from multiple sources to a wiki (Media Wiki, if you must know). Now, the PHBs are all asking questions about how organized the data is. I've already googled for various wiki metrics ideas, but mostly they focus on page counts, average page sizes, rates of edits, etc. Can anybody suggest better ways of measuring the quality of a wiki? Things like uncategorized pages, articles that are too small, etc? Any help or fresh ideas would be appreciated."
Impossible! (Score:3, Interesting)
Organized? (Score:2, Interesting)
The whole point of a wiki is that it doesn't have to be organized. It doesn't have a table of contents, it has a page list -- which are not organized in any sort of temporal or incremental way, but rather alphabetically.
You don't read a wiki start to finish like most PHB's think documentation should be read. Instead, try to make him understand that relevant links (internal link count metric) and search indexing (search count metrics) are what make a wiki organized.
Which is the whole point. You go from an organized list of things describing X and Y in word documents, to a wiki describing X with subpoints X{a,b} and Y with clarifications to Y{c,f,g,h}, etc. Some people don't care about Y{f}, but some people actually do. And since everything describing Y should be in the same wiki, then all of the users can go to their respective pages.
Its more of a culture change than anything. [Now, talking about wiki uptake... that should be something interesting.]
ask the users to survey it ... (Score:4, Interesting)
Depending how much of a BOFH you want to be you could ask users to rate the wiki, either a rating for any page they use (hard to enforce): MS have something like this as too do IBM, IIRC, "rate this article on ...".
Alternatively you could do an exit survey, when a user leaves teh site you pop-up a survey to ask about their experience.
Perhaps you could simplify adding templates and get users to add class markings (I think that's what they call them) for things like "this is a lame article", "this article needs editing for brevity", etc. and markings of "this is a B grade article", etc., then run a test of how many of what grades are given and how many templates are used.
Note that templates describing failings will be more prevalent as the wiki gets more use (per user and overall user count) a PHB might not twig this and will think quality is going down.
You could also measure uptake versus the previous sources.
Connectedness (Score:2, Interesting)
Re:Metrics (Score:4, Interesting)
That's actually a pretty useful suggestion. Think about it, the real question the PHB is asking is, why should we continue to fund this thing? Are we getting our money's worth? The best way to answer that is to see how much, if at all, people are using it, and what they think of it. Your survey may just be the metric the bosses are looking for.
Re:There are two possibilities (Score:5, Interesting)
A good PHB is mostly concerned about 3 things:
1. A decent ROI. ( From the business as a whole down to the paper-clip supplier and the birthday party commitee.)
2. Making the people above him/her look good.
3. Control. (So #1 and #2 above can be maintained over time and change. You have to stay organized when you delegate or it's impossible to manage anything)
If you concern yourself with the exact same things in the exact same order then it should be easy for you to figure out what to do and what buzzwords to say.
The doing might be hard, but the "what to do?" should become easily apparent.
In other words answer these questions as asked by your boss:
What's the ROI on your wiki project?
What's the weakest link to me (PHB) not worrying about this..meaning what person or machine or database do we need to protect and make redundant?
Who can I trust (to make me look good) when they say it's a success. I will keep asking dumb questions until I find a leader I can trust.
Who should I scold if the ROI on this project starts tanking?
In 2 or 3 short sentences, what does it do so I don't look stupid when people ask me. And when I ask "what does it do?" I do NOT mean how does it work. I mean what Return does it provide on what Investment?
Careful though, thinking like this will get you promoted FAST.
Re:Similar Situation (Score:3, Interesting)
I completely agree with you. This was one my biggest concerns when the idea of using a wiki was brought up at my company. We know that the real power of a wiki comes from the users, but management likely won't see it that way. You didn't quote the rest of that point where I indicate that I'm pushing for the user populace to have edit privileges ;-)
Well, yes and no. The idea behind the wiki is two-fold. It's not just that users should be able to quickly and easily edit documents, but also that they still be accurate. Consider a hypothetical situation in which procedures may be changing semi-frequently in a customer-facing department, such as a call center. You want all procedures to be up-to-date, and it shouldn't take a lot effort to publish and re-publish the content, but you also don't want Joe McNewbie making changes when he may not completely understand what he is talking about. From the time he makes the (poor informed and/or possibly incorrect) change to the time at which someone spots the mistake and corrects it any number of other reps may have read the same material and used it during their calls. So now you have reps who are providing incorrect information, and possibly forcing the company to spend time/money/resources correcting those mistakes. User involvement will of course go a long way towards finding and correcting mistakes, but there is still the possibility that those mistakes will cause problems in the mean time. Of course, you could also argue that a wiki may not be the best solution in a case such as this.
Ok, so I realize I sound like I'm arguing against myself here, but what it really means is I can see both sides of the fence. Personally I'd rather give more power to the users, but I understand management's point of view as well. I suppose what it really boils down to is how much you trust your users to self-manage the wiki, and how much risk you're willing to take on by giving them the ability to make changes.