Archive | October, 2007

Why using 30 observers in the same room as the participants is 99% bad*

(Thanks for taking time to reply to my post Jared Spool. I have huge respect for your position as a usability guru thats not afraid to go upstream with your opinions. In this case though I think that your article Usability tests with 30 observers points the usability testing community in the wrong direction)

You have valid points in defense of your method, but I still don’t think 30 observers is a good idea. My main concern is (still) for the participants. But I have also some other arguments that supports the two-room testing setup:

The ethics! (once again)

I still claim that this method puts unnecessary strain and stress on the test situation and on the participants. People tend to agree on any kind of disclosure you present to them, that doesn’t mean that they look forward to sitting in front of 30 observers.

(One great scene in the movie “The Elephant Man” comes to mind where the kind doctor shows off the elephant man to his collegues in a big auditorium. Sure it was full consent from all parties and John Merrick did it by his own free will, but I think that when he get the reactions from the other doctors I think he rather would not have come).

I think that a big audience has a bad effect on the users performance. Usability professionals can probably agree that a nervous user is the worst thing that can happen to a usability test next to a poor written test script (?)

Jared says in his post that “a silent room is a room that is paying attention”. However, any slip-up by the observers (laughing, coughing, moans) at the wrong point can have direct impact on the test. I’d rather have a observation room filled with cursing (tech-)monkeys than anything that will disturb and influence participant behavior :D

The value of having observers separated from the test participant

Sometimes you have observers saying something like: “oh, this user is so stupid that we can’t blame the website/interaction/design”, but after seeing the 3rd participant doing the same thing they usually shut up and takes the lesson. My point is that statements about the user doesn’t mean that they don’t pay any attention to the situation. The passing of notes between the observers could (for all you know) be the same comments in writing ;)

My best argument to keep the observers in a separate room is the possibility to ask questions to the observers (but via the facilitator!) while the test is going on without disturbing the participant at all. This can give valuable feedback to the facilitator sitting next to the user. (We use MSN-chat between the facilitator in the observation room and facilitator in the testlab to get this communication going – I see that I forgot to mention this in my previous post).

How distracted are the participants by the cameras vs. 30 observers in the same room?

In response to Daniel Szucs question about how quickly the participant forgets he is being observed Jared answer that “They always seem aware of their observers, though, with the right facilitation, they can prevent that awareness by focusing on the task at hand.”. But later in the same reply he says that “Ideally, we’d give observers 5 minutes after each task and about 15 minutes at the end of the session”. This must surely be a fireproof way to remind the participant that he has audience – if he actually manages to block out the 30 pair of eyes on him during the test, he will certainly be reminded every 5-7 mins. about the size of the audience :)

I have only had 1 – one – participant in all the tests I’ve done that were aware and seemingly distracted by the camera standing next to the screen. I remember it well because we forgot to turn off the VNC control feature that resulted in the facilitator in the observer room to take control over the mouse for 10 secs by a mistake… resulting in a overly suspicious participant that probably wondered when the candid camera crew would appear. This is also a reminder that devil is in the details when it comes to making participants comfortable during the test.

I’m not a big fan of one-way mirrors either. I’ve only been invited as observer on one of these tests and my impression is that the sound isolation when using a one-way mirror is poor so that any loud discussion might be heard by the test participant. This might be a trick by the facilitator in order to keep us silent in the observation room though…

What effect will the observers comments and questions have on the participants actions (and ultimately on the result of the test)?

I think that allowing direct contact between the participant and the observers is generally bad. The observers will in many cases have a vested interest in a good result and a satisfied user. That may result in biased questioning from the observers: “Why did you click that button, when the red big button next to it was the logical choice?” This question will probably have a direct impact on how the user will perform and express himself for the rest of the test.

My advice for the best possible usability test situation:

  • Don’t intimidate the participant. A nervous participant is a useless participant. Keep the observers out of sight and out of mind!
  • Let the observers make comments in the observer room but try to keep it constructive and informative. Teach them to defend the user and blame the system (always).
  • Use instant message communication between the two facilitators to make sure that the questions from the observation room are presented to the participant in a non-biased manner.
  • Use two facilitators always.
  • Try to keep the test situation as similar as possible for each participant. How can you draw conclusions about anything if each test is unique?

* actually it doesn’t seem 99% bad, maybe only 60% bad. But it is bad, for sure!

Leave a Comment

Comment on Jared Spools post “Usability Tests with 30 Observers”

This is my response to Jared Spools post: “Usability Tests with 30 observers“:

I’ve been conducting usability testing for several years (5+), but I have never even considered mixing the observers with the participants.

The reason for this is twofold:

1. I can’t see the real benefit from doing this. You say that it seems to keep the observers alert and quiet while conducting the test and that they don’t get so easily bored when they are in the same room. In my experience its often good to let the observers express some emotions for whats happening. Sometimes they try to downplay or comment on whats happening and those comments can sometimes be just as enlightning as the participants actions. Other times you can easily explain why things happen so that the observers doesn’t jump to wrong / pre-determined conclusions about what he sees. A silent room of 30 observers? Why?

2. Usability test ethics. You say that:

“Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.”

This must be a cultural thing. I think that if you put an average Scandinavian test person in a room with 30 observers you will have one seriously nervous participant. Nervous participants are (as you well know) not of good use for the client you do the usability test for. I don’t know if this will be just as severe with American test participants.

So I think the participant will be nervous and reserved throughout the whole session. And what if he/she really messes it up in the usabilitytest and does something really funny or stupid? No matter how well informed the 30 observers are – you will have a problem with laughter (or even anger?). This creates a very bad situation for the poor test participant and in worst case u get a person that is a bundle of nerves for the rest of the session.

“Having the observers in the same room as the participants means they can interact.”

This is another thing I wouldn’t like to see. I can imagine this would create a few situations where you get observers that goes on with “why did you do that?” and then constantly reminding the test participant that there are 30! people he have to explain the error to.

I think both situations are unethical because it makes the situation akward for the participant.

I’m not pretending I’ve got THE answer on how to do it but here is our setup:
We use 2 cameras and live transmission of the screen/mousemovements projected on canvas in the observer room. A similar setup like the one you have with loudspeakers and also headphones to the main observer than takes the (main) notes. The cameras are connected to 2 TVs about 25″ big. So the real focus is not on the person but on the big canvas showing the actual website in action + sound. The expressions and facial tells is mainly for the trained observer and not for the bulk of the observers anyway.

I can see the entertainment effect of 30 people watching in the same room, but I can’t see the real benefit here.

Leave a Comment
Follow

Get every new post delivered to your Inbox.