User Test: Preparations
We, at Broken Rules, are in the hot phase of continuous user testing. Instead of building a level and hoping that it’s going to work and be fun, we’re building test levels first, to see how some of the ideas work out when played by real people. Some of the ideas work so well in our minds but when played, prove to lack something. To find this “something”, before it’s too late, we prefer doing user tests.
This is the first time I am responsible for the whole process. Back at Rockstar Vienna the test area was already there, everything was connected and running and I didn’t even have to find test users and set appointments with them. All I had to do was hunt down the latest deployment (not as easy as it sounds), set up my test lab, make sure everything’s running and then wait for the user to knock on my door so that we could start the usability test. The lab was great – two separate areas. One for the tester to sit in and play, and one for me, where I had my two PCs, my monitors, my TV and joystick to remote control the video camera. In the end, I was occupied enough with writing down every single thing the testers did, then analyzing the results and writing the protocols.
Now, I have to do everything myself. I did some research on what software to use, since we couldn’t afford such a high-end setup as was used back at Rockstar. We also lack the opportunity to have two separate areas, so I needed to find a solution that allows us to use as few equipment as possible, while not cutting back on the end material. Silverback was the software I always returned to. I can use it on the Macbook Air with no issues, which was one of the main advantages. Silverback records everything that happens on-screen. It also records everything from the built-in iSight, which is of extreme importance for us. Since the Playstation controllers can be simply plugged into the Macbook Air via USB (no drivers required), I quickly found my test setup: Macbook Air with controller, external HD (for capturing the videos) and TV set attached. With this arrangement I’m still quite flexible, we do not need a fixed test area with special equipment and all in all, the costs for the Silverback license were minimal.
The testing area was a bit of a problem, since the setting requires the tester to be “alone” with the game. With the room divider at the Rockstar lab this was no issue. I wasn’t there, practically, and the testers didn’t feel watched or supervised. Our new office is great – the only drawback is the lack of rooms. We have one big room that is thematically separated but not via walls or curtains or dividers. We’ve now put up a makeshift room divider that allows some privacy for the tests. Nonetheless, every discussion in the background and every comment can be heard. Of course, the level designers are curious to watch the test as it happens, which also means that not only I am watching the test but 2 or more people. And even if everyone tries to be really, really silent – a group of 5 can still be heard and felt by the tester, which interferes with the test results. They will, subconsciously, start playing differently. They will also hold back on their comments, or, even worse, start to please the developers with their comments. The filter, that is usually applied to every word we say, and which I ask them not to apply when they test, is re-installed and the test results can be rendered almost useless. The setting worked so far, though.
Another interesting part of the preparation is finding the right test users. At this stage of testing it’s essential to have “fresh” players, who haven’t played the game before and approach the game and its elements with a fresh mind and view. Previous play tests were mainly focused on the multiplayer part of the game, which, of course, requires more than one player to see if everything works. These sessions were less focused and mainly resembled a fun, social gathering. While the results were still useful, the approach of having two or more test users present for the single player part of the game would not improve the test results. Quite the contrary. So we had to disappoint some of the players who were excited about the social gatherings – since they had already played the game and knew of certain elements and mechanics, we couldn’t use them to find out if changes we had made worked, or didn’t.
So we sat down and brainstormed about our main target audience. With this in mind, I contacted those who fit the profile, added some testers we knew where not our main target audience but offered other qualities that were important to cover. Fans of the genre, skilled players, casual players and even test users we knew would not like the game but could give us valuable input otherwise.
Round 3 is up ahead and so far, we’ve had great results. So, we must’ve done something right. 😉
More about conducting the test will be coming up soon!
Filed under: target audience, Usability | 3 Comments
Tags: broken rules, play test, rockstar, rockstar vienna, silverback, usability testing, user test