Continued debate from: http://www.iallenkelhet.no/slik-maler-du-effekten-av-nettstedet-ditt/
@Gerry McGovern and @Bjørn: I have to say that even though a webguru like McGovern uses this method and argues quite well about its advantages, I don’t trust the results and the TPI number will be… well… useless?
You both say that the “optimal time” for the task is the most difficult part of the equation. The way you calculate this number is a “black box” of mystery as @bjørn said earlier – you use the customer, the fastest participant, your own expertise…. “We take a number of issues into account” (McGovern). I’m sorry but this doesn’t seem like something that would lead to a credible result. If you are using your expertise to decide optimal time, then you apply qualitiative factor into the quantitative method that makes it actually less trustworthy as an indicator.
This week I did a usability test with only 1 task – It’s basically 4 screens to fill out if you do it right. The fastest participant used less than 6 minutes and did it without any trouble. The slowest participant did it in 23 minutes. I wouldn’t try to make any conclusions about time on task from these results (median = 882 seconds | average time = 877 sec):
The result you get from the test is largely depended upon the successrate. If the successrate is low, then the TPI will be low.
Successrate is a number I have dropped from my usability analysis alltogther. Why? Because its a number depended on a large number of factors and even though 10 of 10 actually completes a task, that says next to nothing about how easy it was, or how many in the “real” world that would manage to complete the task, or how good the website really is.
Usability testing is a qualitative analysis and its wrong trying to mask it as something quantitative by introducing this magic number called TPI.
With 15-20 users you will be able to see a (strong) recurring pattern, no doubt about that, but its a long way from seeing a pattern and to grade someones website from 0-100 and calling it Task Performance Indicator slapping the grade on the report and force the client to improve the website so the number goes up!
*EDIT* Gah, so @josmag is complaining about wrong use of errormargin – even though I posted my disclaimer 2 seconds after my main post :D Let’s fix the errormargin thingy and see where that leads me:
OLD POST:
(TPI for this website would be 1(360/882) = 0,40 = 40%. Ok and then apply the error margin for +/- 19% so I can trust that my result is really somewhere between 21% and 59%.
That means i get a TPI that either give me: “TPI under 30 er ganske dårlig. Du har et stort problem.” OR “TPI på 51-70 er bra. Fortsatt mulig å forbedre nettstedet ditt.”
Doh? Does it suck or not? Well, the number doesn’t give me any indication really, but from what I saw in the test I would say to ignore the TPI and just fix the obvious problems.)
NEW POST:
TPI for this website would be 1 (360/882) = 0,40 = 40% or 1 (360/714) to 1 (360/1050) which gives me a number between 50% and 34%. Not a great difference from my original post. You (@josmag) are also correct that the median will be more trustworthy if I have more users in my test – we don’t know if the number will go up or down – and if we get users that don’t complete the task I will have to adjust the successrate from 100% with 5% for each user failing the task (with 20 users).
I think that the money spent on testing 20 users would be better spent if you split it up in more than 1 usability test and use the money more wisely. And as @magnusrevang points out – use analytics to get the “magic numbers” for time on task and successrate.
/End NEW POST
Actually, knowing the competence from both Netlife Research and G. McGovern I would trust their (expert) judgement a lot more than the actual number they get out of their magic blackbox :)
Information Design & Usability 