This is my response to Jared Spools post: “Usability Tests with 30 observers“:
I’ve been conducting usability testing for several years (5+), but I have never even considered mixing the observers with the participants.
The reason for this is twofold:
1. I can’t see the real benefit from doing this. You say that it seems to keep the observers alert and quiet while conducting the test and that they don’t get so easily bored when they are in the same room. In my experience its often good to let the observers express some emotions for whats happening. Sometimes they try to downplay or comment on whats happening and those comments can sometimes be just as enlightning as the participants actions. Other times you can easily explain why things happen so that the observers doesn’t jump to wrong / pre-determined conclusions about what he sees. A silent room of 30 observers? Why?
2. Usability test ethics. You say that:
“Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.”
This must be a cultural thing. I think that if you put an average Scandinavian test person in a room with 30 observers you will have one seriously nervous participant. Nervous participants are (as you well know) not of good use for the client you do the usability test for. I don’t know if this will be just as severe with American test participants.
So I think the participant will be nervous and reserved throughout the whole session. And what if he/she really messes it up in the usabilitytest and does something really funny or stupid? No matter how well informed the 30 observers are – you will have a problem with laughter (or even anger?). This creates a very bad situation for the poor test participant and in worst case u get a person that is a bundle of nerves for the rest of the session.
“Having the observers in the same room as the participants means they can interact.”
This is another thing I wouldn’t like to see. I can imagine this would create a few situations where you get observers that goes on with “why did you do that?” and then constantly reminding the test participant that there are 30! people he have to explain the error to.
I think both situations are unethical because it makes the situation akward for the participant.
I’m not pretending I’ve got THE answer on how to do it but here is our setup:
We use 2 cameras and live transmission of the screen/mousemovements projected on canvas in the observer room. A similar setup like the one you have with loudspeakers and also headphones to the main observer than takes the (main) notes. The cameras are connected to 2 TVs about 25″ big. So the real focus is not on the person but on the big canvas showing the actual website in action + sound. The expressions and facial tells is mainly for the trained observer and not for the bulk of the observers anyway.
I can see the entertainment effect of 30 people watching in the same room, but I can’t see the real benefit here.
Information Design & Usability 