Beta Testers and Their Bug Reports

03Feb09

Finally, the beta is out, a first glimpse of the final product can be caught, being one of the first to play it and being able to actually be part of the developing process. How exciting!

The good part is – usually, everyone who manages to sign up until the deadline is in and becomes a beta tester. The bad part – everyone who manages to sign up until the deadline is in and becomes a beta tester.

No prior experience needed. No training required. Just a working machine, the ability to download and install the client. And that’s it, most of the time. This usually shows in about 70% of the bug reports that come in.

Depending on the report system, it is often required to answer a few basic questions (to isolate the area of drama and damage) which not only help QA finding and reproducing the bug but also the developers to get rid of it (which is why the bug is usually reported). Deciding not to answer these questions usually lowers the chances of the bug getting fixed. If QA is unable to isolate the bug, the developers cannot work on a solution to fix it.

Furthermore, the beta tester often has the opportunity to give a more detailed description of what happened when and where and before/after/during something went horribly wrong (or simply looks weird). This is usually the part that really helps QA and the developers locating and isolating an anomaly and getting rid of it as soon as possible.

Somehow, beta testers/bug reporters love to ignore both ways of communicating with the developers. Neither questions are answered that could help locate the bug, nor is a description given as to what happened and maybe how and where. Instead, questions are skipped and one of the two most popular replies is given:

  • No:
    No…what?
    You see, in the end, QA will receive a bug report that consists of exactly that:
    no
    Nothing else there to work with. Pretty much useless. This sort of bug report is immediately marked as “Not a bug” and buried with all the other “nos“. Untouched. The bad thing is – there might have been an actual bug that might have interested the developers. With no information given this bug goes unnoticed.
  • ajdkl:
    In fact, this reply can vary, depending on where exactly the bug reporter smashed their fist into the keyboard. The message is the same. Developers cannot filter any useful information from this congregation of random characters. Again, there might have been an actual bug that might be important enough to be reported. Communicating a problem like this will go unnoticed because this variation will also be immediately marked as “not a bug” and buried with all the others.

A few beta testers do bother adding a comment but quite often their impression of the bug reporting system and what actually happens diverge. I cannot speak for all bug reporting systems but not always will there be a screenshot added to the description, nor is a video attached to the report. Just the text, with the information given by the bug reporter. No surprise that bug reports in the style of:

I clicked this and then that happened.

mean well but cannot be further processed. Both “this” and “that” are not clear to the developers. In fact, “this” and “that” could be anywhere and anything if the questions beforehand have not been answered.

Similarly ineffective are descriptions in the manner of:

This game needs improvement – a lot.

The bug reporter is most often 100% right. The game needs improvement. It is still being developed, graphics will be enhanced, items added, features implemented and the beta testers are a good source of receiving constructive comments on what can be improved. A bug report like this is, unfortunately, stating the obvious but not actually helping to improve the final product. Again, this is a bug that will be closed immediately, since there is no information included that could help improving the game.

So, are there bugs that will not be closed immediately?

Yes, there are. About 20-30% do hold enough information so that QA can work with them. Unfortunately, some of them do not hold enough information to assign them immediately to someone who can fix the bug. First, QA needs to hunt for further details, which are not given.

Trying to answer the following questions the best way possible will increase the chance of having the bug assigned to someone who can fix it in no time.

  • What?
    Something has happened. It is not enough to give a vague description of what has happened, make sure to be as precise as possible as to what exactly happened. Did the game freeze? Crash? Did something not load? Did it take too long? Descriptions like “weird” and “strange” are often not enough because they are too vague and less specific. Make sure to explain –>
  • Why?
    …something is strange. Because it should not happen but has happened. Maybe because the appearance is clearly flawed. Maybe because the 10x before it did something and now it does something else.
  • Where?
    Now that we know exactly what happened and why it is a bug we need to find out where it has happened so that we can go there and take a look ourselves, reproduce the bug and gladly assign it to someone who can fix it. Again, make sure to be as precise as possible. Give the name of the map, coordinates, if possible (and available), the area, mention landmarks, anything that can help locating the bug.
  • When?
    This one is tricky. It does not refer to the time the bug occurred (unless this is a crucial element of the game) but what has been done before the bug occurred. Single steps (babysteps) are most helpful. Developers do not play the game and do not know the basic steps/procedures of a game, so they need to know about the most simple things. “Start the game.” “Enter your name.” “Create character.” “Choose alignment.” Completely logical to the beta tester – not at all logical to the developer. Make sure to mention every click, every step that was made before something happened that seems to be an error and needs some fixing.

Reporting a bug is really not a big deal. If you know who is working with the information on the other end and what they can actually see and know, it is relatively easy to write a short report that will not be closed immediately but further processed and implemented.

Advertisements


3 Responses to “Beta Testers and Their Bug Reports”

  1. 1 Mark

    A good post, but I was truly struck by this:

    “Developers do not play the game and do not know the basic steps/procedures of a game, so they need to know about the most simple things.”

    If you factor in the amount of time lost trying to translate bugs back to developers (which is the bulk of your post), you could omit a great deal of that wasted time simply by having the developers play the game they’re making. Maybe not constantly — maybe not often — but why not simply stop production for a day now and then and have everyone play?

    It seems to me that the information gleaned in that work day might more than compensate for itself in time savings when compared with trying to interpret or divine the meaning of a beta-tester’s report. It would also have the additional benefit of reintroducing the developers to their own product, such that the team would all have the (relatively) same impression of the work, and of the stage of development.

  2. Theoretically, I fully agree. I always wondered why different departments have different approaches to the very same product they are creating. I’ve found out that the size of the company seems to be one major factor, how bugs are handled, how the processes are handled and how much time can be “wasted” (I don’t see it as a waste) for merely playing the game and getting a good overview over the current status.

    Since we are all freelancers, time is money (isn’t it always ;)). And time itself is scarce, since the company is relatively small and the deadlines tight. While this is in theory a good approach to bring everyone on the same level of knowledge it also means that production stands for a day. It would probably cost the company (this company) more money to spend a day on playing the game and getting to know it than letting the testers quickly add the necessary steps to let the developers know what is required to reproduce the bug. Quite often, if reproducing the bug is too time-consuming, they will approach QA and they will do it for them and provide them with the necessary information. All this works because the company is rather small and communication fairly easy.

    I agree, though, that bigger companies should actually look into this solution. Having every single developer on the same level of knowledge is essential to work on the game. It is essential to understand certain bugs (which are caused by certain mechanics) and essential to make a uniform game.

    How is this process handled at your company (assuming that you’re working in the industry)? I am curious about different approaches and if approaches differ due to company-size, management, etc.

  3. 3 jhorneman

    Ensemble is – sorry, was 😦 – famous for forcing everyone in the company to play their games once a week, from the financial director to the animators.

    They are the only company I know of that has a strict rule like that. However, it is often a good sign if the developers really like to play the game they are making. It is not always possible, since it’s hard enough to find decent developers and developers may not be the target audience.

    Still, I would expect most developers to be familiar with the basics of the game they are working on.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: