Forgot your password?
typodupeerror
Businesses Software

How Can You Measure a Wiki's Worth? 78

Posted by timothy
from the weigh-the-electrons dept.
moldar writes "I have been involved in a major project to migrate documentation from multiple sources to a wiki (Media Wiki, if you must know). Now, the PHBs are all asking questions about how organized the data is. I've already googled for various wiki metrics ideas, but mostly they focus on page counts, average page sizes, rates of edits, etc. Can anybody suggest better ways of measuring the quality of a wiki? Things like uncategorized pages, articles that are too small, etc? Any help or fresh ideas would be appreciated."
This discussion has been archived. No new comments can be posted.

How Can You Measure a Wiki's Worth?

Comments Filter:
  • Turn it off. (Score:5, Insightful)

    by zonky (1153039) on Thursday August 14, 2008 @08:24PM (#24608837)
    See how many phone calls/emails you get complaining.
    • by Fluffeh (1273756)
      Or...

      See how many edits there are on the "Who shot first, Han or Greedo?" entry.
    • by Anonymous Coward

      The problem is that he's trying to measure the wrong thing. He's trying to get metrics from the wiki itself. He should be getting them from the users.

      So the simplest solution is to do a user survey about what's good and what needs improvement.

  • by Red Flayer (890720) on Thursday August 14, 2008 @08:25PM (#24608851) Journal
    I really don't know the best way to go about measuring a wiki, but I'd suggest not taunting it. It might get pissed and rip your arm off.
  • It would be worth whatever you can get someone to buy it for.

    Nice editing on the part of whomever took the firehose article. Changing the title changed how anybody is going to read the article.

    Getting a single metric for the quality of a wiki is going to be really hard, since the number would just about be made up.

    If you want to say what percentage of pages are uncategorized, below a certain size, or orphaned, then do that.

    • How much would you pay for a gallon jug of air? You might value being able to breath but you'd probably be very reluctant to pay for air until your head wasunderwater?

      People are very resistant to pay for something that they feel should be free. The same people that would pay a hundred dollars or more for MS Office would feel screwed if they paid $10 for OpenOffice.

      Value to the economy is very different to the value marketeers attach to something. Wikipedia, Linux and many other free products are very valuab

  • your wiki measures you!

  • Metrics (Score:5, Insightful)

    by Enderandrew (866215) <enderandrew&gmail,com> on Thursday August 14, 2008 @08:34PM (#24608945) Homepage Journal

    PHBs will always insist on metrics for things that can't properly be quantified.

    What I would do is suggest that he task a team from another department to run a survey on satisfaction amongst wiki users. Has the wiki helped them? How often do they use it? What would they like to see changed.

    This should distract the PHB for a while, while diverting your responsibility to come up with a metric.

  • by MarkusQ (450076) on Thursday August 14, 2008 @08:35PM (#24608949) Journal

    It depends a lot on what your wiki is for, but usage pattern statistics might be a good metric. For example, repeat visitors are people who are getting some value out of the wiki. Trampoline pages (pages people hit and then bounce elsewhere from a link in the page) might be another good thing to track. As well as abort-and-retry pages, where instead of following a link the user goes to the search box and tries something else.

    You get the idea.

    If the wiki is valuable to the people who are using it, you should be able to tell this from the logs--even if no one ever mentioned wikipedia in public, they could tell how much we care by noting that we keep coming back.

    --MarkusQ

    • by aero6dof (415422)

      It depends a lot on what your wiki is for, but usage pattern statistics might be a good metric. For example, repeat visitors are people who are getting some value out of the wiki. Trampoline pages (pages people hit and then bounce elsewhere from a link in the page) might be another good thing to track. As well as abort-and-retry pages, where instead of following a link the user goes to the search box and tries something else.

      One can transform these types stats into quantified estimates of savings vs non-wik

  • Impossible! (Score:3, Interesting)

    by Gewalt (1200451) on Thursday August 14, 2008 @08:37PM (#24608965)
    You cannot "measure" anything until you have defined measurement criteria. So until you define the "worths" of a wiki, you cannot measure them.
  • Organized? (Score:2, Interesting)

    by tmarthal (998456)

    The whole point of a wiki is that it doesn't have to be organized. It doesn't have a table of contents, it has a page list -- which are not organized in any sort of temporal or incremental way, but rather alphabetically.

    You don't read a wiki start to finish like most PHB's think documentation should be read. Instead, try to make him understand that relevant links (internal link count metric) and search indexing (search count metrics) are what make a wiki organized.

    Which is the whole point. You go from an or

    • The whole point of a wiki is that it doesn't have to be organized

      Very good post!

      At my company (one of the major international technology SI's, population >40k peeps) we've just moved our entire culture across to a global Wiki (Confluence) with a lot of MediaWiki imports from various wikis (our company's wikis, that is) around the world.

      The existing knowledge base portal just didn't quite catch on, was too bureaucratically structured and just never achieved critical mass -- behind the scenes, a lot of individual Wikis had spread up to address the content shortfall

  • obvious (Score:3, Funny)

    by Brain Damaged Bogan (1006835) on Thursday August 14, 2008 @08:59PM (#24609145)
    by using metrics of course
    • by dunnius (1298159)

      by using metrics of course

      Or you can use imperial units if you want to torture yourself.

    • by HTH NE1 (675604)

      by using metrics of course

      But imperialistics have been working so well.

      "`What's the Wiki worth?' It exists and it's useful. That's enough! Off with his head!"

  • by CorporateSuit (1319461) on Thursday August 14, 2008 @09:09PM (#24609269)
    Many Moons [tripod.com] (I apologize for the hosting company in advance, the site is not mine) is a childrens' story everyone in IT, Sales, and Marketing should familiarize themselves with.

    The truth of the matter is: You're asking the wrong people. You should be asking the suits what they would imagine in a report and what kinds of numbers they're looking for. If they're general and obtuse, tell them straight up they'll get a general report, or just make up numbers to keep them happy. Create a widget they can access from their desk that shows them numbers generated from the database that hardly matter. Does it matter what you deliver if they don't know what they want?

    They're trying to imagine some great power-point presentation you'll be showing with pie charts and red/green/yellow/blue graphs popping out and wowing them? Make it so. They're trying to imagine a spreadsheet? Easier and more concise. They just want a status report at the end of the day on what percentage of the documentation has been migrated? You can probably get away with a few page count written on a napkin.

    Remember the Alice's conversation with the Cheshire cat:
    "Would you tell me, please, which way I ought to go from here?"
    "That depends a good deal on where you want to get to," said the Cat.
    "I don't much care where -" said Alice.
    "Then it doesn't matter which way you go," said the Cat.
    • Create a widget they can access from their desk that shows them numbers generated from the database that hardly matter.

      I have an RRD graph titled "Processed Internet Packets" that just graphs /dev/random every 5 minutes. The Manager I built it for contentedly uses the graph for all sorts of things, and I can get back to my work.

      • by ais523 (1172701)
        Use /dev/urandom instead, save entropy! Some day it's possible your system will be low on entropy for whatever reason and /dev/random will block.
  • Simple (Score:3, Funny)

    by Dan East (318230) on Thursday August 14, 2008 @09:17PM (#24609363) Homepage Journal

    eBay

  • by pbhj (607776) on Thursday August 14, 2008 @09:24PM (#24609399) Homepage Journal

    Depending how much of a BOFH you want to be you could ask users to rate the wiki, either a rating for any page they use (hard to enforce): MS have something like this as too do IBM, IIRC, "rate this article on ...".

    Alternatively you could do an exit survey, when a user leaves teh site you pop-up a survey to ask about their experience.

    Perhaps you could simplify adding templates and get users to add class markings (I think that's what they call them) for things like "this is a lame article", "this article needs editing for brevity", etc. and markings of "this is a B grade article", etc., then run a test of how many of what grades are given and how many templates are used.

    Note that templates describing failings will be more prevalent as the wiki gets more use (per user and overall user count) a PHB might not twig this and will think quality is going down.

    You could also measure uptake versus the previous sources.

    • It may just be me, but if I were one of your users, constant pop-ups asking me to rate the site would just result in low ratings of the site.
  • 1. Pretend to invent your own metric by lack of pre-existing satisfactory metric
    2. Describe it using esoteric terms you know your audience won't grasp/buzzwords
    3. Make up phony results
    4. ???
    5. Ask for a raise, errr, I mean, profit!

  • I was on the same boat not that long ago; here are a few ideas:

    1. Wiki documentation to non-wiki documentation ratio. Compare how much documentation there is between the wiki itself vs. other project documentation like your word documents and pdfs (assuming they are not just copy/paste stuff from the wiki). A simple word count as measure will probably please your PHBs.
    2. Assuming that this is a programming project that you are in, you can also compare the % growth of the wiki vs. the % growth of the codebas

  • Similar Situation (Score:5, Informative)

    by SirLurksAlot (1169039) on Thursday August 14, 2008 @09:45PM (#24609593)

    I'm actually in a situation which is very similar to the OP's. The company I work for has a number of training/procedural documents which currently exist in .mht format (of all things), and we're in the process of migrating those documents into the wiki which comes built into MOSS (the latest version of SharePoint). I won't go into detail about how horribly crippled the MOSS wiki is, except to say it is. Anyway, I was asked to write a "Wiki Best Practices" document and I tried to include a few ideas on what makes a successful wiki because I knew the business users were pretty clueless about them (not that I claim to be an expert here). These were some of the points I tried to address:

    1. First and foremost are the users able to find what they are looking for? If they can't find it in the wiki they'll go somewhere else for it, and the wiki will fall on its face.

    2. Is the information accurate? If you're not meeting these two criteria your wiki is probably worthless, and your users will almost certainly resort to another source of information.

    3. Does your wiki exhibit signs of growth? By this I mean new pages/sections/categories being added when new information becomes available, and edits being performed to improve the quality of existing information. I think one of the biggest hurdles companies will have to overcome for this criteria is the fact that there tends to be a very top-down approach to the distribution of new information (This is definitely the case in my company). In order to allow the wiki to grow there must be enough users with create/edit privileges to keep it up to date. The more users there are who are actively involved in the wiki, the more likely it is to be successful. Of course, if management doesn't want to allow the user populace to be able to edit the wiki the task will probably fall to a select few, and they will inevitably become involved in their other tasks at which point the wiki will fall by the wayside and die a quick death. This is the battle I'm currently fighting, and I'm not entirely sure I'll win.

    4. Ask your users what they think. This one is pretty obvious, but easy to overlook if you're doing things like checking page hit counts and sizes. Put together a survey and ask them if they find the wiki to be easier/more useful than the previous method, if it saves them time (and if so how much), and what they think should be done to improve it. Ask them if the information they find is accurate or not. Finally, ask them what they did/didn't like about the previous method, and whether or not that issue has been addressed by the wiki.

    • by Conficio (832978)

      Does your wiki exhibit signs of growth? In order to allow the wiki to grow there must be enough users with create/edit privileges to keep it up to date. The more users there are who are actively involved in the wiki, the more likely it is to be successful.

      Does it not defy the purpose of a Wiki in the first place if not everybody has edit privileges?

      After all the ability to see all edits in public and to see the edit history is deterrent enough in a corporate organization where everybody (theoretically) knows everybody.

      If management is concerned about editing important documents, then they should encourage authors/owners of documents to keep tabs on edits and try to "investigate why the change has been made and if it might be actually a welcome change." (if

      • Re: (Score:3, Interesting)

        by SirLurksAlot (1169039)

        I completely agree with you. This was one my biggest concerns when the idea of using a wiki was brought up at my company. We know that the real power of a wiki comes from the users, but management likely won't see it that way. You didn't quote the rest of that point where I indicate that I'm pushing for the user populace to have edit privileges ;-)

        If management is concerned about editing important documents, then they should encourage authors/owners of documents to keep tabs on edits and try to "investig

        • by vrmlguy (120854)

          Consider a hypothetical situation in which procedures may be changing semi-frequently in a customer-facing department, such as a call center. You want all procedures to be up-to-date, and it shouldn't take a lot effort to publish and re-publish the content, but you also don't want Joe McNewbie making changes when he may not completely understand what he is talking about. From the time he makes the (poor informed and/or possibly incorrect) change to the time at which someone spots the mistake and corrects it any number of other reps may have read the same material and used it during their calls. So now you have reps who are providing incorrect information, and possibly forcing the company to spend time/money/resources correcting those mistakes.

          I'd thinking out loud here, but do any wikis support a policy where anyone can edit, but a one set of users have approval rights on any edits, and a different set can only see approved pages? Or even anyone can approve, but you can't approve your own edits. Wikimedia already supports part of this, where users can watch pages for changes.

          • Ah, good idea! Actually the MOSS wiki has this functionality; the "Workflow" feature allows you to do exactly that. I said the MOSS wiki is crippled (and I still stand by that), but that is one of the nice things about it. That would definitely open up the possibility of allowing all users to edit with an approval process in place.

            • by vrmlguy (120854)

              Ah, good idea! Actually the MOSS wiki has this functionality; the "Workflow" feature allows you to do exactly that. I said the MOSS wiki is crippled (and I still stand by that), but that is one of the nice things about it. That would definitely open up the possibility of allowing all users to edit with an approval process in place.

              It took me a minute to find (but I learned a lot about Moss, the plant, on various wikis), but I finally identified it as Sharepoint [microsoft.com]. Yeah, this is probably the way to go if (a) you're a business, (b) you've bought into the whole MS ecosystem, and (c) you want an otherwise crippled wiki that you can't improve.

              I dug into Wikimedia's database schemas and it shouldn't be too hard to add an approval mechanism. I'd add just one table with pointers to a particular edit, the approving user, and the date approved

          • by ais523 (1172701)
            There's an extension for MediaWiki (FlaggedRevs) that does this, IIRC the German Wikipedia is using it at the moment. If you go to a random page at http://de.wikipedia.org/ [wikipedia.org], you're likely to see "Gesichtete Versionen" on a bar at the top, showing that that version is "sighted" and approved to be shown to the public by someone with sighting privileges (a special group of users over there, but based on the way Wikimedia wikis normally work I imagine it's quite large). Anyone can view either the sighted versio
  • Connectedness (Score:2, Interesting)

    by RockMFR (1022315)
    The best measure of a wiki is connectedness, in my opinion. Do all pages have incoming links from relevant articles? Can all the pages be found by casual clicking, or are there some that only have links buried in the middle of paragraphs? Is everything categorized? You should have at least one well organized category hierarchy that contains everything, then add further categorizations from there.
  • There are metrics that can be applicable to the wiki itself, but probably for what they want to know what they have to measure is the community, the people that will feed it, not the tool. Is like measuring the quality of a book by the quality of the paper or ink that was used on it.

    You can use page size, amounts of edits, amount of pages, etc, but that will not imply that things are right or wrong. Sometimes a single word in the right context could say all you need. I remember that a page on wikipedia had
  • I'd say the best measure of a wiki's effectiveness is probably a thorough analysis of the access logs for the web server where the wiki lives. Run them through something like awStats [sourceforge.net] (personal favorite) and see what you get. Are people in your organization actually visiting the wiki? Which pages are most frequently accessed, and by whom? How many users are hitting the Edit or History portions of a page, as opposed to simply viewing static content? These are solid, quantitative numbers, and presented in a "d

  • I can understand why you'd be confused - trying to measure the quality and worth of information is a big problem (just look at Google and AI), let alone how it is organized.

    There are a lot of problems with Wikis - even the benefits, that you can grow your base of information, result in problems: who is going to reorganize it so it is usable again? What are good lists and bullet points to divide gobs of text into into usable chunks of information?

    Someone has already mentioned accurateness, but what that rea

    • Someone has already mentioned accurateness, but what that really means is correctness + point of view + timestamp of information, and these things are rarely if ever tracked normally, let alone in a wiki. Sure wiki's usually track all changes, but that doesn't mean the information is correct at the time of entry, nor that you necessarily know who's point of view is being described.

      I agree with most of your post, but you are basing this statement off of the assumption that the content of a wiki is open to in

      • Well, this is where we start crossing the fold.

        Opinion and interpretation are factors in every case of communication. That's why EULAs and company policies have official rewrites every year, quarter, or incident. That's also why there is always an HR department whose significant responsibility (50% of time?) is to respond to questions from employee, customers, etc.

        Granted some of that time is because people don't bother reading documentation or the search doesn't work, but one thing many, many documents d

  • by RGRistroph (86936) <rgristroph@gmail.com> on Thursday August 14, 2008 @11:19PM (#24610375) Homepage

    Look, the pointy haired boss doesn't really want or expect a numerical metric of organization. Immagine asking him if he has a function in Word that will tell him how organized an outline or document is. He doesn't, and if he never needed it for Word, he doesn't need it for a webpage.

    Possibility 1: he wants to kill the project, and came up with this as a way to find an excuse. The best response to this is to get your resume out there on monster right now, and walk into his office tomorrow morning and say "do you want to kill the wiki, and is this metric a way to get an excuse ?" . Let that conversation go where it will.

    Possibility 2: he read in USA Today that anyone can add stuff to wiki's and they are chaotic, so he's worried. He's never actually looked at any wiki including this one in his life. This is the most likely possibility. If you still have a job after asking the question in Possibility 2, go for this, or maybe start with this.

    Here, you want to talk to him for about 5 minutes using the following words as much as possible, without actually lying: "review", "control", "process" and "catagorize". If you want to can reveiw everything he has ever said and come up with your own set of buzzwords, based on the words he himself likes to use, but these will do. Then you say things like "We have a PROCESS to CONTROL the QUALITY of employee submissions. I occasionally REVIEW the most visited pages in the wiki, and make sure they are ACCURATE and properly CATAGORIZED. I also REVIEW the least visited pages, and if they are not visited because they are improperly CATAGORIZED and linked, I fix that." Actually talk like that, and kind of shout the buzzwords at him. He's either stupid enough that he needs it, or smart enough to figure out what's going on and move on to something important.

    All the geeks in this discussion who started actually talking about measuring connectedness of graphs and crap are totally off in the weeds.

    • by Strange Ranger (454494) on Friday August 15, 2008 @02:49AM (#24611641)
      I was going to mod you insightful, but there's an easier and more positive way to apply those buzzwords...

      A good PHB is mostly concerned about 3 things:

      1. A decent ROI. ( From the business as a whole down to the paper-clip supplier and the birthday party commitee.)
      2. Making the people above him/her look good.
      3. Control. (So #1 and #2 above can be maintained over time and change. You have to stay organized when you delegate or it's impossible to manage anything)

      If you concern yourself with the exact same things in the exact same order then it should be easy for you to figure out what to do and what buzzwords to say.
      The doing might be hard, but the "what to do?" should become easily apparent.

      In other words answer these questions as asked by your boss:
      What's the ROI on your wiki project?
      What's the weakest link to me (PHB) not worrying about this..meaning what person or machine or database do we need to protect and make redundant?
      Who can I trust (to make me look good) when they say it's a success. I will keep asking dumb questions until I find a leader I can trust.
      Who should I scold if the ROI on this project starts tanking?
      In 2 or 3 short sentences, what does it do so I don't look stupid when people ask me. And when I ask "what does it do?" I do NOT mean how does it work. I mean what Return does it provide on what Investment?

      Careful though, thinking like this will get you promoted FAST.
    • English Language
        |_ American Spellings
              |_ Misspellings
                    |_ C
                          |_ CATAGORIZED, replace CATEGORIZED, British CATEGORISED, not shouting "categorised"

      If I spelt anything wrong I'm sure you guys have my back ...

  • Install some analytics tags and look at the data. How many people are using it? How many pages are accessed every day/week/month? etc.

    If it's public-facing, put ads on it and see how much money you make. :) Can't beat that for a "how much is this worth?" question.

  • ... your bosses are demanding citations, and you're pissed about their lack of NPOV.

    Slug it out in a good ol' fashioned edit war.

  • As the person who started a wiki which has a small potential audience (10 million or so worldwide by some estimates) is that after a growth spurt at the start, it has now slowed down. Currently it gets about 100 hits per day - 3 or 4 of those are spammers adding junk pages and the same number is me deleting them, so around 90 hits per day on the front page. The metrics I use are : is the number of contributors increasing?, Are any pages being edited and how often? Are people returning? Is the number of pag
  • I manage a large intranet wiki at a university, based on TWiki. We also do some research on its use. The main metrics I would suggest :

    * number of pages (cumulative graph over time, and per month/day)
    * number of edits, idem
    * number of views , idem
    * Number/size of attachments, idem
    * a histogram of edits/views per person
    * a histogram of people who only read and never contribute.

    you can obviously anonymize the graphs if

  • When a user leaves your site, just pop up one of those irritating survey windows with a 'how useful was this wiki to you?' (You might have to say 'web site' if 'wiki' means nothing to them)..

    Log some answers, or just plain old make them up, and put them in an Excel spreadsheet which you then print out and give to your PHB.

  • Use mind mapping software... they make professional mind mapping software for businesses.

    http://www.thebrain.com/ [thebrain.com]

    And it organizes information much better.

  • If your organization is anything like mine, the most useful part of the wiki is how much simpler it makes life for the support team. Anytime a particular issue reaches a critical mass, a wiki article can go up and enter the revision cycle, and from that point forward all calls related to that issue can be answered with a single hyperlink.
  • I'd look at a few measures:

    • Number of pages (and their growth over time) as well as the pages/employee ratio
    • Number of WikiWords ==> links between pages
    • Ratio WikiWords/Page
    • Number of visits (and visits/employee ratio)
    • Number of bounces (and bounces/employee ratio)
    • List of top bounce pages and look if they are info pages, that stand on their own, such as company directory.

    Also, institute some meetings where people go through the Wiki and try to eliminate "old news" that has only historic value. This keeps the

  • by undantag (1345293)

    See point #3 on the 10 Tips for a Successful Wiki [flickr.com] poster from gdc;

    Invest in your wiki

    Nothing is perfect off the shelf and your wiki will be no different. Most wiki packages are designed to be customized and extended, so take advantage of that by investing in a few days of developer time to modify it to suit your needs. Re-evaluate wiki usage at regular intervals to gather feedback on the most common gripes and dedicate resources to adjusting the wiki accordingly. The few hours or days spent tweaking it will come back tenfold in time saved by users.

APL hackers do it in the quad.

Working...