This is my response to Jared Spools post: “Usability Tests with 30 observers“:
I’ve been conducting usability testing for several years (5+), but I have never even considered mixing the observers with the participants.
The reason for this is twofold:
1. I can’t see the real benefit from doing this. You say that it seems to keep the observers alert and quiet while conducting the test and that they don’t get so easily bored when they are in the same room. In my experience its often good to let the observers express some emotions for whats happening. Sometimes they try to downplay or comment on whats happening and those comments can sometimes be just as enlightning as the participants actions. Other times you can easily explain why things happen so that the observers doesn’t jump to wrong / pre-determined conclusions about what he sees. A silent room of 30 observers? Why?
2. Usability test ethics. You say that:
“Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.”
This must be a cultural thing. I think that if you put an average Scandinavian test person in a room with 30 observers you will have one seriously nervous participant. Nervous participants are (as you well know) not of good use for the client you do the usability test for. I don’t know if this will be just as severe with American test participants.
So I think the participant will be nervous and reserved throughout the whole session. And what if he/she really messes it up in the usabilitytest and does something really funny or stupid? No matter how well informed the 30 observers are – you will have a problem with laughter (or even anger?). This creates a very bad situation for the poor test participant and in worst case u get a person that is a bundle of nerves for the rest of the session.
“Having the observers in the same room as the participants means they can interact.”
This is another thing I wouldn’t like to see. I can imagine this would create a few situations where you get observers that goes on with “why did you do that?” and then constantly reminding the test participant that there are 30! people he have to explain the error to.
I think both situations are unethical because it makes the situation akward for the participant.
I’m not pretending I’ve got THE answer on how to do it but here is our setup:
We use 2 cameras and live transmission of the screen/mousemovements projected on canvas in the observer room. A similar setup like the one you have with loudspeakers and also headphones to the main observer than takes the (main) notes. The cameras are connected to 2 TVs about 25″ big. So the real focus is not on the person but on the big canvas showing the actual website in action + sound. The expressions and facial tells is mainly for the trained observer and not for the bulk of the observers anyway.
I can see the entertainment effect of 30 people watching in the same room, but I can’t see the real benefit here.
Information Design & Usability 
Thanks for your thoughtful response.
To talk to your points specifically:
1) A silent room is a room that is paying attention. All too often, when observers are behind the mirror (or tv monitor) they start discussing what’s going on. Sometimes this discussion is positive. Often it’s derogatory to the user (“Oh, what kind of idiot is this?”). In either case, the team has stopped paying attention to the user and is now in their own world, doing their own thing. Important things could happen in those subsequent moments and the team would not see them.
Instead, I encourage the team to pass notes, which participants have no trouble ignoring. When a team member is writing or reading a note, they also aren’t paying attention. But the rest of the team is, so most of the information is still being retained.
The other advantage of having the participants in the room with the observers is a single facilitator can make sure both groups are getting what they need from the session. When you have a separate room, either you need two facilitators or one side goes without the facilitator being present.
2) With regards to the ethics:
We use full disclosure for the participants. They know why the observers are there and the observers are trained to behave respectfully. Having done this thousands (literally!) of times, I’ve never once had an observer act inappropriately in a session.
One could argue that having an unknown group of individuals behind a mirror (or tv camera) that the participant doesn’t see or know why they are there could be unethical, since the participants are being observed without their full understanding.
I wouldn’t knock the method until you’ve had a chance to try it. A lot of people have had similar reservations to yours, however, once they try it, many of them decide it’s an ideal approach that solves many of the current problems with testing.
Jared
I know the fact, that the audience must be not more than 12 members to grasp information properly. So, based on scientific approach an experiment can’t be conducted with 30 peoples the same time.
Brilliant post!, bro