Beta Testers and Their Bug Reports
Finally, the beta is out, a first glimpse of the final product can be caught, being one of the first to play it and being able to actually be part of the developing process. How exciting!
The good part is – usually, everyone who manages to sign up until the deadline is in and becomes a beta tester. The bad part – everyone who manages to sign up until the deadline is in and becomes a beta tester.
No prior experience needed. No training required. Just a working machine, the ability to download and install the client. And that’s it, most of the time. This usually shows in about 70% of the bug reports that come in.
Depending on the report system, it is often required to answer a few basic questions (to isolate the area of drama and damage) which not only help QA finding and reproducing the bug but also the developers to get rid of it (which is why the bug is usually reported). Deciding not to answer these questions usually lowers the chances of the bug getting fixed. If QA is unable to isolate the bug, the developers cannot work on a solution to fix it.
Furthermore, the beta tester often has the opportunity to give a more detailed description of what happened when and where and before/after/during something went horribly wrong (or simply looks weird). This is usually the part that really helps QA and the developers locating and isolating an anomaly and getting rid of it as soon as possible.
Somehow, beta testers/bug reporters love to ignore both ways of communicating with the developers. Neither questions are answered that could help locate the bug, nor is a description given as to what happened and maybe how and where. Instead, questions are skipped and one of the two most popular replies is given:
You see, in the end, QA will receive a bug report that consists of exactly that:
Nothing else there to work with. Pretty much useless. This sort of bug report is immediately marked as “Not a bug” and buried with all the other “nos“. Untouched. The bad thing is – there might have been an actual bug that might have interested the developers. With no information given this bug goes unnoticed.
In fact, this reply can vary, depending on where exactly the bug reporter smashed their fist into the keyboard. The message is the same. Developers cannot filter any useful information from this congregation of random characters. Again, there might have been an actual bug that might be important enough to be reported. Communicating a problem like this will go unnoticed because this variation will also be immediately marked as “not a bug” and buried with all the others.
A few beta testers do bother adding a comment but quite often their impression of the bug reporting system and what actually happens diverge. I cannot speak for all bug reporting systems but not always will there be a screenshot added to the description, nor is a video attached to the report. Just the text, with the information given by the bug reporter. No surprise that bug reports in the style of:
“I clicked this and then that happened.“
mean well but cannot be further processed. Both “this” and “that” are not clear to the developers. In fact, “this” and “that” could be anywhere and anything if the questions beforehand have not been answered.
Similarly ineffective are descriptions in the manner of:
“This game needs improvement – a lot.“
The bug reporter is most often 100% right. The game needs improvement. It is still being developed, graphics will be enhanced, items added, features implemented and the beta testers are a good source of receiving constructive comments on what can be improved. A bug report like this is, unfortunately, stating the obvious but not actually helping to improve the final product. Again, this is a bug that will be closed immediately, since there is no information included that could help improving the game.
So, are there bugs that will not be closed immediately?
Yes, there are. About 20-30% do hold enough information so that QA can work with them. Unfortunately, some of them do not hold enough information to assign them immediately to someone who can fix the bug. First, QA needs to hunt for further details, which are not given.
Trying to answer the following questions the best way possible will increase the chance of having the bug assigned to someone who can fix it in no time.
Something has happened. It is not enough to give a vague description of what has happened, make sure to be as precise as possible as to what exactly happened. Did the game freeze? Crash? Did something not load? Did it take too long? Descriptions like “weird” and “strange” are often not enough because they are too vague and less specific. Make sure to explain –>
…something is strange. Because it should not happen but has happened. Maybe because the appearance is clearly flawed. Maybe because the 10x before it did something and now it does something else.
Now that we know exactly what happened and why it is a bug we need to find out where it has happened so that we can go there and take a look ourselves, reproduce the bug and gladly assign it to someone who can fix it. Again, make sure to be as precise as possible. Give the name of the map, coordinates, if possible (and available), the area, mention landmarks, anything that can help locating the bug.
This one is tricky. It does not refer to the time the bug occurred (unless this is a crucial element of the game) but what has been done before the bug occurred. Single steps (babysteps) are most helpful. Developers do not play the game and do not know the basic steps/procedures of a game, so they need to know about the most simple things. “Start the game.” “Enter your name.” “Create character.” “Choose alignment.” Completely logical to the beta tester – not at all logical to the developer. Make sure to mention every click, every step that was made before something happened that seems to be an error and needs some fixing.
Reporting a bug is really not a big deal. If you know who is working with the information on the other end and what they can actually see and know, it is relatively easy to write a short report that will not be closed immediately but further processed and implemented.
Filed under: game, Game Related, QA | 3 Comments
Tags: beta, bug, communication, developer, game, information, product, program, QA, report, reporter, screenshot, software, system, test person, tester