Slashdot Log In
Conducting a Unix Desktop Usability Study?
Posted by
Cliff
on Thu Dec 15, 2005 07:05 PM
from the not-something-you'd-generally-find-in-the-HOWTO-db dept.
from the not-something-you'd-generally-find-in-the-HOWTO-db dept.
cyclop asks: "I am a close friend of a Ph.D. student on human interface usability. She's now working to tailor a KDE-vs-Gnome usability study (a pretty hot topic these days), and I have been called to help, as a long time GNU/Linux desktop user. What kind of advice -- both technical and theoretical -- would you give us on conducting a deep and objective study on the Unix desktop, that can be useful for the developers and the OSS community?"
"She has installed GNU/Linux and used both KDE and Gnome to get to know them, while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects. However, I feel that it's not enough and that there are a lot of potential pitfalls and misconceptions that wait for us, me being a geek and she being a Linux newbie. Moreover, she found that most of the previous studies on the web were quite sloppy, in comparison with common usability research standards."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Allow power-users to tweak settings first. (Score:4, Insightful)
The out-of-the-box setup is a compromise at best; and shouldn't be used to judge the overall usability for people who use the system more than once.
Tainted vs Ignorant users. (Score:5, Insightful)
#1. The ignorant users: These have never used a desktop before. These aren't as easy to find anymore. I worked with one woman back in the mid 90's who could not even use a mouse. She had to hold it still with one hand while she clicked the button with her other hand. After a week of solitare, she had the necessary muscle coordination to start learning the system.
#2. The tainted users: These have experience with systems other than the one you're testing. If your system isn't 100% like the one they're used to, they'll waste time clicking around where the functions are on their systems.
#3. Friends: These have worked on the system that they're being evaluated on.
Now, a system that is easy to learn for the "Ignorant" class may be incredibly un-friendly for more advanced "Friends".
Determine what functionality you want to measure and what GROUP you want to measure it for.
The real "ease" on an interface comes down to 2 things:
a. Can you quickly guess where a function is based upon your existing experience with it?
b. Once you know where a function is (you guessed at it before, you asked someone, you went to training), how easy is it to remember that 24 hours later, 1 week, 1 month, 6 months later?
Parent
Re:Tainted vs Ignorant users. (Score:3, Interesting)
I totally disagree with this. what percentage of your potential audience has "never used a desktop before". Very close to zero, and getting lower all the time. MSFT did do one thing, and that was teach everday people what a desktop was, and how to more-or-less get around a computer.
There is no reason to try to design anything for this class of user, it is such a small portion of the populace that it can be said to be zero.
In the USofA, that is mostly correct. (Score:4, Insightful)
Not to mention that an interface with settings 1-10 (1=brand new user, 10=expert) would make a lot of non-experts more comfortable with their computers.
An ideal interface would evolve with the user's experience level. Not trap an experienced user with a pre-school interface nor confuse a new user with expert-level options.
Tailor the choices available to the level of the user and let the USER choose how complex the interface he uses is. Just like books. When you started reading, you didn't read the books you read today. Those books followed very careful patterns on what words were used and how often they were repeated.
But since none of the interfaces out there are doing that yet, it really doesn't matter for this discussion.
Parent
Re:Tainted vs Ignorant users. (Score:4, Informative)
Parent
Re:Tainted vs Ignorant users. (Score:3, Interesting)
My sister's owned a computer since 1998. She's a whiz when it comes to email, using Word, or manipulating her photos in Photoshop. Her first machine was a Gateway monstrosity that she paid way too much for. Now she's got a G5.
A couple of months back, she was absolutely flabbergasted when she saw me switching between open windows... all this time she's had no idea that one can minimize a window to the taskbar and continue on in another.
You missed one last (Score:5, Interesting)
(In fact that was my case when I started learning both KDE and Windows 95 in parallel)
The best system ever should be as easy as possible for #1 out of the box, but need to be very easily configurable to whatever complex system #3 and #4 need.
If the system is newbie-friendly but can't evolve you'll end with Clippy and this kind of stuff that gets in the user's way with pointless tips (tips that would have helped a total beginner, but sorry now I know exactly what I want). Attracts #1 users, but repels #3 and #4.
If the system is configurable to extreeme you end up with emacs or vi : the most powerfull tool around you can't ever dream of in your wildest dreams, but you can't do anything without unless you've spent the first year learning it the harsh way. #3 only are interested, #4 must ponder if they want to re-learn everything once again (albeit they do it faster), #1 will prefer to commit suicide.
#2 are pointless, they won't accept anything that isn't their original system, they're the one that will never switch to MacOSX or Linux because it's not Windows+Office, and they'll cry each time MicroSoft revamps the interface and everything is moved around (Windows 3.11 -> Windows 9x -> Windows 2k -> Windows XP -> Windows Vista and same for the Offices). Just wait until the next "GUI is completly changed one more time" period, and they'll be as good as #1 users (or #4 if it's not their first change around).
Parent
Re:You missed one last (Score:3, Insightful)
I disagree. Group #2 is probably the most important group. Keep in mind that being tainted does not only mean being exposed to Windows+Office, but also to any other desktop environment, including Linux, OS X or others.
This includes for instance the Linux FVWM users who will be frustrated that the default GNOME window manager does not
Re:The key would be selectability. (Score:5, Funny)
I'm a level 137 Microsoft Win-zard, I can now wield the CLI of fortitude and I've unlocked the 5th ring of the registry.
That will surely make the level 1s in this world eager to learn.
Parent
Re:The key would be selectability. (Score:5, Funny)
Parent
Re:Tainted vs Ignorant users. (Score:3, Interesting)
Its not just the users who are dummies.
I was TOLD to take out passwords and logins on a database connected to the internet because "its too much bother for the workers to log in."
So if you know the server and the directory, you can "administer" it. Add inventory. Reprice inventory. Remove inventory. Make a contract selling a $10,000 item for $1.99. Change descriptions. And since it also rebuilds the web site, change the product pictures for pr0n and add links to competitors.
Re:Allow power-users to tweak settings first. (Score:3)
You don't want to compare GNOME to KDE to Windows because GNOME and KDE aren't operating systems. You should compare *Fedora* against *Ubuntu* against *Mac OS* against *Windows*.
Fedora and Ubuntu make customizations to GNOME because they feel they are doing a better job of usability than upstream. Fine, let them. Tell us how they compare.
No one (sane) will give a newbie an uncustomized Gentoo box or Debian setup, so compare realistic things
Re:Allow power-users to tweak settings first. (Score:3, Insightful)
Simple (Score:5, Insightful)
Re:Simple (Score:5, Funny)
No, but "KowboyNeal" [slashdot.org] is.
Parent
Good Lord (Score:2, Insightful)
Ha. Ha ha. Ha ha ha heh heh hee hee hee hee ha ha HAA HAA HAA HAAA HARRRRRR. You might as well do a study on whether apples are better than oranges, or settle the One True God question once and for all. It's hardly possible to do a deep and objective study on the merits of the Linux kernel vs. the FreeBSD kernel even though that's reducible to a purely technical inquiry.
It *does* sound like perfect academic paper fodder.
for meaningful results... (Score:5, Interesting)
3-6 months perhaps?
"Usability" imho, in the usual meaning of the term, is a load of wank.
Who cares if the first time someone uses the environment that it takes a little orientation to get used to? In the real world, if a couple of weeks of pain makes you much more productive after that, it's a net benefit imho - the remainder of your time using the environment outweighs the significance of the learning time.
I'm not saying that initial learning is not important, but I think that these studies need to show both sides of the equation...
smash.
Depends on Intended use... (Score:3, Insightful)
A work desktop must be quickly learnable and facilitate productivity of intermediate to advanced users.
Learning should be built into the system. (Score:4, Insightful)
How about, how many repetitions of instruction does it take for an average user to remember how to perform one action after a week of not using it?
And, once one function is explained, how quickly can the average user deduce/guess at related functions? This is how you select "bold" text. Then let them find "italics" and "underline".
Parent
Exactly. Spend a few months with KDE, and... (Score:3, Insightful)
Exactly. On the face of it, KDE and GNOME can appear to be similar to some people. You may even dislike KDE's Qt interface for a while until you get used to it. However, KDE has amazingly powerful technology underneath. Spend a few months doing all your work on KDE, giving yourself time to discover (discovery learning is good) its hidden strengths, and you'll never look back.
I don't often agree with Linus, but what he said about GNOME and KDE was spot-on: when you take the time to try them both, you'll s
My thoughts (Score:5, Insightful)
It is probably also worth noting that most people (apparently including Linus) consider KDE more powerful, so KDE is kinda at a disadvantage.
Re:My thoughts (Score:5, Interesting)
most people have used windows at one time and expect things to have that type of layout.
children who havent gotten used to what windows is like might find it a bit easier/harder to move around in.
you could have 2 groups of children
set one group of children to use gnome for the first week/month/year and kde for the second week/month/year whatever
and set the other group to use kde for the first week/month/year and gnome for the second week/month/year
and compare there reviews of how easy it is to move around.
however it might be better to test it out on teenagers are they will be able to take more infomation in.
Parent
Re:My thoughts (Score:3, Insightful)
Also, a lot of the tasks you'd be interested in testing a child wouldn't know about. If you were to ask, for instance, the user to write a formal letter in a word processo
Re:Will they be testing as Linus uses it? (Score:3, Informative)
Open Parent -> Alt+Up
Open Location -> Ctrl+L
Close Parent Folders -> Ctrl+Shift+W
Close All Folders -> Ctrl+Q
Close -> Ctrl+W
Home Folder -> Alt+Home
Notice that we're using Alt key combinations, Ctrl key combinations, and Ctrl+Shift key combinations. My biggest problem with using nautilus effectively is mixing u
The Nipple? (Score:3, Funny)
Go for it!
(If you're wondering about the subject of this comment, the nipple is one of the most intuitive human interfaces btw).
Re:The Nipple? (Score:4, Funny)
maybe on a woman... but I don't consider my own very intuitive; I can't figure out what its purpose is.
Parent
Re:The Nipple? (Score:5, Funny)
You're a geek, so just use the process of elimination. All you have to do is remove your nipples, then wait and see what stops working.
It's not rocket science...
Parent
Re:The Nipple? (Score:2)
Long-Term Efficiency (Score:5, Insightful)
Re:Long-Term Efficiency (Score:3, Insightful)
As for doing a usability study, you first have to decide what to measure, then to decide what your users are. "everything" and "everybody" are hopeless non-answers.
On what to measure, you could focus on several things: time spent dealing with the desktop rather than your work; number of desktop related problems run int
My advice? (Score:5, Insightful)
My advice? Don't have someone who's been a long time GNU/Linux user assisting her. Chances are, you're fond of either KDE or Gnome. Before the study has even started, I'm alarmed by potential bias. Let her do the study on her own, gather the facts and come up with her own conclusion. Isn't that what Ph.D.'s do?
"[...] while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects."
To me, the study is already flawed. You've dropped a load of information onto her lap, while a complete "newbie" doesn't have that same luxury. How can a usability study be unbiased in this manner? Who's to say you didn't provide her with REALLY good links to KDE information, while giving half-assed links to Gnome?
Re:My advice? (Score:4, Informative)
You didn't understand. She will conduct the study, but that's not she will judge what's more usable and what's not. This would not be a usability study, it would be a -1,Flamebait article. She plans instead to put categories like WinXP-proficient people,MacOSX-proficient people and total computer noobs (if any still exists) in front of Unix desktop enviroments and see their reactions and if and how they can be proficient with them. She's using them to understand them and for obvious curiosity, and I gave her info to help her tailor the study.
Parent
Not good. (Score:3, Informative)
No. All she is "testing" there is how closely the desktop they're being "tested" on resembles the one they're used to.
... where's Outlook? Wh
Novell did this already.
http://linux.slashdot.org/article.pl?sid=05/10/11/ 146202&tid=223&tid=106 [slashdot.org]
So, to send email
Re:My advice? (Score:4, Insightful)
The researcher won't be a subject. You can't do a usability study that way. You need to recruit a bunch of people who match the kind(s) of user you're studying, get them to do a range of tasks, and observe various aspects of their performance. If you're your own subject, you're not doing research. You're just airing your opinion.
Speaking as a PhD student, one of the most important things we are expected to do is a literature survey. That means we have to go out and read studies that are relevant to our research topic, and critique them. If the researcher fails to discuss them, her supervisors should ding her for not having done her reading. If she can't judge the worth of the studies for herself, she's not working at PhD level yet. She should have a good grasp of what constitutes good research by now.
I know I'd be thrilled to have a load of pointers to relevant studies dropped in my lap. I'll judge their degree of assedness myself, thank you very much.
Parent
objectivity schmuctivity (Score:5, Insightful)
Some examples:
* A novice might look for how obvious it is to do a certain task, whereas an expert user might instead prefer what can be done fastest (e.g. notepad vs. emacs).
* Related: How much time does this person use a computer/this application can be an important factor. If I rarely do 3d design, I want to be told how to do everything, and have obvious controls (i.e. > 3 parameters might boggle my mind). However, if I work for Pixar, the verbose messages, and dumbed down controls (i.e. 30 parameters might just not cut it for what needs to be done).
* Certain paradigms might make sense to people who are used to using certain types of systems. Files and folders make perfect sense to many people, but certainly not to everyone (e.g. my mother). We think these simplified analogies work better for novices, but that isn't always the case. People think differently, and different analogies will make more/less sense dependent on their world view.
Some suggestions (Score:3, Interesting)
*) Count the times you had to look in a manpage, in google, and how often you had to fire up a console for doing simple things (like creating a shared folder, browsing the internet, installing some plugins like flash etc.) Keep in mind: SIMPLE things! Trivial tasks done by the casual user.
*) For each system you need to learn how to use it, thats a fact. Unix users have to learn the concept behind the filesystem (nothing too fancy, but basic knowledge about what mounting is for example). This is comparable to the knowledge about the drive letters in Windows, the usage of backslash for separation in paths, that
*) Review the help system and documentation. Among other things, look for technical mumbo-jumbo. This is a common error. Stuff like SSH, SSL, CORBA, FUSE, pthreads etc. should never occur in enduser documentation.
*) Have a look at the menus. Are they cluttered or usable? How long did you have to search something in the menu?
don't do it! (Score:5, Insightful)
Instead compare either or both against Windows or Macintosh for tasks that your _specific target userbase_ would do. [If you haven't defined one or more use cases you've already lost.] This would be much more valuable.
Better yet, switch your topic to focus exclusively on accessibility (a11y). Every DE out there needs some accessibility love.
Personally I think Apple has... (Score:2, Interesting)
Currently they have it working on two different processor families (the IBM Power series, and Intel).
Yes, it is proprietary, but that does NOT mean that "Aqua" is not a GUI desktop running on a UNIX system.
Why not compare the other UNIX desktops with what may be the best UNIX desktop running?
Don't get excited, it's just an honest question. After all, just because it was done by a comme
Re:Personally I think Apple has... (Score:3, Interesting)
> core (the Darwin flavor of freeBSD).
Well, this is no real option.
Because for Apple users it is irrelevant whether it is build on foosys or Unix. When you run a C64 emulator on Linux which is distributed as a game console to play games you cannot say "this is a real Linux" only because a techie can open a linux console. For the average use it is C64. And for the manufacturer it it was a rather technical choice wh
I used a Solaris CDE Desktop from 1992 to 1999 (Score:2)
Frankly, I don't see why this is needed...people have been using it for years.
There...Can I please have my doctorate now?
Don't assume. (Score:3, Interesting)
Some suggestions: (Score:4, Interesting)
Get some novices (people who have never used the interface before) in, and a list of tasks to complete. Let some of them muddle through it on their own, and give others pointers on use of the help system, google, and man pages. (One of the tasks for the first group could be -finding- help on some of the things they won't be able to complete on their own.) This will help represent the range of people coming into it-some will have absolutely no idea what they're doing, others will have at least some support from other knowledgeable users who can at least point them in the right direction.
You also might want a broad cross-section of users-some who rarely or never use a computer at all, some who use one relatively frequently, and some "power users" from other operating systems. This could lead to a very interesting picture-do those that already "know how" on a different interfacee have a harder time with something new, or are they able to translate most of their knowledge and pick up more quickly?
As to a comparison between the two, you may wish not only to time how long it takes the users to complete their task lists, but also include feedback from them-were the help pages actually helpful, or did they just confuse the issue more? Was the experience relatively smooth and welcoming, or aggravating and frustrating? Was there anything the user expected to be/work a certain way that did not do as expected? Did the user find it necessary to work in CLI at any point, and if so, was this disorienting or frustrating, or relatively smooth? Did they ever think they had done something right when they really had screwed it up, and were any clues/warnings given them to this effect if so?
All these are factors in usability, and I'm sure anyone can list plenty that I missed. In the end, usability is determined by-well, the user. Since it is somewhat subjective (I find working in a command line far easier and more convenient then use of a GUI most of the time, but there are many that would disagree!), focus on what the end-user, presented with the interface for the first time, thinks of it overall.
improve apon betterdesktop (Score:3, Informative)
Improve apon betterdesktop [del.icio.us]. The site is a collection of usability data with a focus on Linux apps. The front page gives more detail
The Study should be Double Blinded (Score:4, Insightful)
Also, make sure to use more than Gnome or KDE! Use XFCE, Fluxbox, and other XWindows managers.
And don't forget to make sure that the study has the appropriate "power"!
And make sure that everyone is using the same system configuration (motherboard, processor, underlying flavor of linux)
The most overlooked item is (Score:4, Interesting)
Using the OOMA method, let's say it takes a user on one system 2 minutes to figure out how to send an email, and 1.95 min on anther system. What the hell does that mean?
If you use other items as some sort of bench mark, people might begin to get a feel for the numbers.
Lets say it take 45 seconds to figure out a new blender, 5 minutes to use a new remote, 20+years to set the clock on the VCR. Now people reading your study have a reference they can relate to.
It would also help companies trying to make applience computer to know where they stand in relation to appliances.
Don't get my hopes up. (Score:4, Interesting)
Well, since Unix has *NEVER* had an objective study of it's desktop done, you will make history as a pioneer. Since it's survived so many smear campaigns, yours will, unfortunately, just add to the hot air. What, exactly, is the *point* of such a study, anyway? What does it change? I have yet to read a single such study that swayed my choices one iota.
Sadly, you're off on the wrong foot already. KDE-vs-Gnome. Hey, Dr Kinsey, there's just a few other test subjects you're failing to interview: http://xwinman.org/ [xwinman.org]. So actually, you're flunking already. You are not doing a "Unix desktop study". You are doing a "KDE-vs-Gnome" study, and your results will no more be applicable to Unix in general than a study of Coke-vs-Pepsi would apply to all beverages.
It does not go without saying: Don't be paid Microsoft shills. Don't be paid by *anybody* for that matter.
Now, if I studied dogs, I wouldn't start with everything I know about cats and try to fit it all around that by comparing dogs with cats at every possible point. Similarly, Unix never gets taken as an operating system on it's own right. Everything is instead stated "It is not as good as or just like or better than Microsoft." How about judging something just once based on it's own merit, the way anybody studying anything else is expected to do in any other field? Consider your subject as if other operating systems did not exist. God knows, Microsoft is talked about in this manner.
Unfortunately, the focus will of course be on KDE and Gnome, the Heckyl and Jeckyl whose sole point of contention is "I'M a perfect clone of the Windows environment!" "No, I am!" "No, me!" "NO, ME!" So in fact, you're not the least bit interested in considering even KDE or Gnome on it's own right - this will be a Windows-impersonator contest. Never mind that counting from the invention of computers: http://ei.cs.vt.edu/~history/Babbage.html [vt.edu], computers have been around for one hundred and eighty-two years, and only the last 20 years http://members.fortunecity.com/pcmuseum/windows.ht m [fortunecity.com] has seen the existence of a desktop system known as Windows. For a ratio of 0.10989011 of computer's history, you are going to compare the one system whose sole claim is that it made a lot of money in the United States to two other desktops expressly written to mimic it.
I'm really sure the world will be enlightened.
Problem, Action and Result (Score:4, Funny)
Action: Perform UNIX GUI usability studies every few days, post repeatedly on Slashdot.
Result: UNIX GUI usability studies improve, UNIX GUI usability stays same.
KDE, Gnome and other (Score:3, Insightful)
Phrasing may be important too (Score:3, Interesting)
- "Send an email"
- "Create a new email then send it"
- "Write an email"
The answer from my point of view is "I don't know", but I'd be interested to know. Of course you need a load more people and a load more tests. But this is science, it's always toughTell her to check back her notes. (Score:3, Insightful)
Human interface usability is properly tested one way and one way only. By watching real people interacting with them. You can use muckups (drawn or computerized), storyboards, etc, but nothing beats putting the fat asses of a few users in front of a computer and collecting their impressions.
If you possibly can you tailor your investigation to a particular group of people, ideally one that would make the study useful to you (if you are testing usability for software in kindergartens you don't want to do your usability test in a bank's trading floor).
The laughable suggestions to use children only, experienced users only or unexperienced users only as the correct or more accurate way of gauging usability is, as the British say, a load of pants. People suggesting this should jump of a clift like the gerbills they are.
One would do such a think only if there is no choice or if one has particular reasons for doing so, but never as the preferred criteria for a generalized useful study, what may be good for children may be crap for old timers and viceversa, experienced users may find some things annoying that new users find useful and viceversa.
The first thing that many people fail to understand is that usability is a wholy subjective thing. Linus loving KDE (yeah, that Linus) is only probe that for Linus (yeah, that Linus) KDE is more usable. All the KDE zealots implying that this is the God given truth regarding usability in Linux should be forced to use Gnome untile the know better.
Usability should be studied only on groups with similar patterns of usage for it to be any useful. The wider you make your target group study, the more difficult it will be to find meaningful results.
If you target all Linux users, then you are in for the most subjective, meaningless, most likely useless study.
If you target Linux users with less than one year experience using Linux then you are into something. If you target Linux users with more than 10 years experience degrading penguins you would also find more useful results.
Target your audience and you will find good results for that group of people.
Make your sample too wide and be welcome to the scrapyard of useless studies.