User Test: Preparations


We, at Broken Rules, are in the hot phase of continuous user testing. Instead of building a level and hoping that it’s going to work and be fun, we’re building test levels first, to see how some of the ideas work out when played by real people. Some of the ideas work so well in our minds but when played, prove to lack something. To find this “something”, before it’s too late, we prefer doing user tests.

This is the first time I am responsible for the whole process. Back at Rockstar Vienna the test area was already there, everything was connected and running and I didn’t even have to find test users and set appointments with them. All I had to do was hunt down the latest deployment (not as easy as it sounds), set up my test lab, make sure everything’s running and then wait for the user to knock on my door so that we could start the usability test. The lab was great – two separate areas. One for the tester to sit in and play, and one for me, where I had my two PCs, my monitors, my TV and joystick to remote control the video camera. In the end, I was occupied enough with writing down every single thing the testers did, then analyzing the results and writing the protocols.

Now, I have to do everything myself. I did some research on what software to use, since we couldn’t afford such a high-end setup as was used back at Rockstar. We also lack the opportunity to have two separate areas, so I needed to find a solution that allows us to use as few equipment as possible, while not cutting back on the end material. Silverback was the software I always returned to. I can use it on the Macbook Air with no issues, which was one of the main advantages. Silverback records everything that happens on-screen. It also records everything from the built-in iSight, which is of extreme importance for us. Since the Playstation controllers can be simply plugged into the Macbook Air via USB (no drivers required), I quickly found my test setup: Macbook Air with controller, external HD (for capturing the videos) and TV set attached. With this arrangement I’m still quite flexible, we do not need a fixed test area with special equipment and all in all, the costs for the Silverback license were minimal.

The testing area was a bit of a problem, since the setting requires the tester to be “alone” with the game. With the room divider at the Rockstar lab this was no issue. I wasn’t there, practically, and the testers didn’t feel watched or supervised. Our new office is great – the only drawback is the lack of rooms. We have one big room that is thematically separated but not via walls or curtains or dividers. We’ve now put up a makeshift room divider that allows some privacy for the tests. Nonetheless, every discussion in the background and every comment can be heard. Of course, the level designers are curious to watch the test as it happens, which also means that not only I am watching the test but 2 or more people. And even if everyone tries to be really, really silent – a group of 5 can still be heard and felt by the tester, which interferes with the test results. They will, subconsciously, start playing differently. They will also hold back on their comments, or, even worse, start to please the developers with their comments. The filter, that is usually applied to every word we say, and which I ask them not to apply when they test, is re-installed and the test results can be rendered almost useless. The setting worked so far, though.

Another interesting part of the preparation is finding the right test users. At this stage of testing it’s essential to have “fresh” players, who haven’t played the game before and approach the game and its elements with a fresh mind and view. Previous play tests were mainly focused on the multiplayer part of the game, which, of course, requires more than one player to see if everything works. These sessions were less focused and mainly resembled a fun, social gathering. While the results were still useful, the approach of having two or more test users present for the single player part of the game would not improve the test results. Quite the contrary. So we had to disappoint some of the players who were excited about the social gatherings – since they had already played the game and knew of certain elements and mechanics, we couldn’t use them to find out if changes we had made worked, or didn’t.

So we sat down and brainstormed about our main target audience. With this in mind, I contacted those who fit the profile, added some testers we knew where not our main target audience but offered other qualities that were important to cover. Fans of the genre, skilled players, casual players and even test users we knew would not like the game but could give us valuable input otherwise.

Round 3 is up ahead and so far, we’ve had great results. So, we must’ve done something right. 😉

More about conducting the test will be coming up soon!


3 Responses to “User Test: Preparations”

  1. Would it help to have the tester wear a set of noise-cancelling headphones whilst testing?

  2. You’re raising an interesting point here. On the one hand, it would definitely improve the sound situation tremendously, since we can’t stop communicating altogether while tests are being conducted. On the other hand, would the headphones be considered as “disturbing” for some testers? And since I need them to tell me everything they’re thinking and doing, would the headphones interfere with the one-sided communication?
    But a very good point and I’d be interested to give it a try. Finances get in the way (can’t afford the noise cancelling headphones now) but as soon as I have the money that might be an interesting solution to our noise-problem.
    Thank you!

  3. Btw, we did use headphones this time. Not noise-cancelling ones but with big ear shells. Worked like a charm. Only drawback – I couldn’t hear the sounds and there was some kind of sound bug, which slipped our attention. Good thing – a splitter is cheap and easy to use. So the next time I can listen in!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: