How to Report Your Bugs
UPDATE 05/06/13: A new field has been into the Mantis form: Version Number. We now require all future reports (and any existing reports which need to be updated) to have the associated GameMaker: Studio version number reported. This change has been made on Mantis and will follow on the web form shortly. Thank you.
Hi, I'm Ross Manthorp, one of the newer members of Core Tech QA and a name you may well have seen on the Mantis DB. I'm here to continue on from Dan's post about the changes to the bug submission process. My job with this post is to break down the process of submitting high-quality bug reports from the new web form, while also giving the same advice to those that still have access to Mantis, in order to create a consistently high quality of reports. Hopefully, as I fully explain each step of the process, it will also highlight again why we have made this change and the positive outcomes we are already seeing.
I am happy to report that everything has been going well and our eyes are already on those who have become Updaters and those who are still Reporters yet still producing excellent samples and helpful notes on Mantis – perhaps with a goal of being upgraded on Mantis - and we are paying close attention to this.
Submitting Bugs Using The New Web Form
Empty Web Form
- Your name: Text input - Enter your name. This can be also be a username if you wish, as it will be used as your username if you do not already have a existing Zendesk or Mantis account.
- Your email: Text input – As above, if you do not already have an account with this email address then one will be created for you.
- Summary: Text input - Concise, short bug title that highlights the problem and explains any important facts. If it's a specific function giving the problem, add the function name in here.
- Description: Text input - Try to provide as much information as possible while not losing the main issue amongst a wall of text. All details you can think to offer, including the scenario, how this affects your game, and any other strange behaviours surrounding the issue. Please also give a list of steps as to how to reproduce the error if you can reliably recreate the problem.
- Category: Drop-down - Select the most appropriate category from the drop-down. E.G. If a function does not work on any platform, then the category would be "Function". However, if a function does not work on one specific platform the category would be that platform's name.
- Platform: Drop-down - The target you chose in the drop-down inside GameMaker: Studio to compile for.
- OS: Drop-down - Select the operating system that the issue occurred on. If your bug appears on a phone, for example, add the phone's OS, not Windows.
- Reproducability: Drop-down - Select an appropriate reproducabilty value. Any suggestions are automatically "N/A", but you should attempt to reproduce bugs before submitting any report, so "N/A" is not valid for a bug.
Ideal Completed Web Form
Following The Bug And Interacting With The Helpdesk
The Helpdesk will try to understand the situation and in most cases will ask for a sample and perhaps some additional information, like crash logs, GameMaker's logs, or maybe a screenshot of the issue.
The interface you will encounter on the helpdesk
Once all the information has been collected and the helpdesk agent can understand the issue they will then pass it on to be added to the Mantis database. Now your bug is added to the public bugs.yoyogames.com to be investigated and assigned by our Core Tech QA team and then fixed by a developer.
This is an example of an email you will receive once your bug has been added to Mantis
This is an example of the automated email you will recieve from Mantis
Note that the Helpdesk ticket will now be marked as solved, as all further details and information regarding the report will be dealt with directly on the Mantis database.
Once The Report Has Been Added To Mantis
This is the report as it now appears in Mantis
An account is created for you using the email you provided, or if the email is already registered then that account is used. The same thing applies with your username.
The Category, View Status, Date, Updated, Reproducibility, Status, Resolution, Platform, and OS field values are all pulled in directly from your Helpdesk ticket or set automatically by Mantis. The summary is taken from Zendesk with the Category added to the start, and the description is copied along with any links to samples or any new information discovered when dealing with the Helpdesk. Additional Information also provides a link to the Zendesk ticket so we can ensure every detail in the ticket is available to the developer.
Priority and Severity are decided by myself and Dan when we receive the report into Mantis in order to keep our views on the classifications of bugs constant.
Also be aware that Mantis report tags have changed in the way that we now use them. No longer can Reporters and Updaters create tags themselves. We currently use tags for #GMKI (GameMaker Known Issues, which are things to be aware of until we fix them) and #URGENT (Important issues inside GM or someone's game that must be fixed for the next public GameMaker release). We will decide if these tags apply to your report.
Updaters: Submitting The Best Possible Reports Using Mantis
The 'Report Issue' form on Mantis
The same general guidelines for filling-in the form inside Mantis are just the same as for filling-in the web form, so if you haven't read the web form bug report description already, please go back up the page and do so now.
As an Updater you will also have the ability to edit the following fields (be aware we will change any incorrect fields):
- Severity: Text input
- Critical/Blocker: We want to keep these down to the smallest amount possible and only the most important issues in GameMaker. You must ensure that this issue is Critical/a Blocker to more than just yourself and your project - i.e., it will affect a lot of people. If it's a code issue a sample will be required.
- A - Crash/Hang: Crashes in the IDE, your game or the runner. Please ensure that this is not an issue with your code first! You are requested to submit a sample and possibly a crash log for this.
- B - Major: A large problem that someone will definitely notice with your game or GameMaker. Majoyly impacts the functionality of your game without crashing it, such as no audio, broken surfaces, no IAPS, etc.
- C - General: Something that effects the quality of GM or your game, but doesn't break it. Most issues are C class.
- D - Minor: Small issues with little-to-no meaningful impact on your usage of GameMaker or playing your game. Many users may not even notice this issue, but it still needs fixing at some point.
- Text/Typo: Typo in the UI or the documentation for GameMaker.
- Wish: A Wish is a task for a developer to implement once a Suggestion has been accepted. It is a low priority task, not a bug. (You should not submit Suggestions as Wishes. Thank you.)
- Suggestion: Suggestions are features or enhancements to GameMaker that you would like to see added to future releases.
- Priority: Text input
- Very High: This will addressed as quickly as possible, usually tied to As or Criticals.
- High: Highs will be looked at as soon as all of the Very Highs are resolved, or if the issue is urgent.
- Medium: These are as time allows. Each developer will look at the medium reports once they have dealt with their Highs and Very Highs.
- Low: Not a pressing concern at the moment, but we wish to keep a record of the issue.
- Very Low: No real impact, so not a big concern if this gets fixed or not, but good for records.
- None: Do not submit issues with 'None' priority. This is for reports that come in automatically from Zendesk before our Core Tech QA team assign them.
- Steps To Reproduce: Text input - A step-by-step guide of short and precise instructions on how to reproduce the issue. Start at a very basic level and explain all required steps: a For Dummies guide to recreating the bug, please.
- Additional Information: Text input - Any seemly unrelated information that may have an effect on the issue, such as other programs running and version numbers. Also, it is handy to note here if any screenshots and/or GMZ files have been attached by you.
- Upload File: Upload - We cannot stress enough how much we love example GMZ files. Internally, myself and Dan are creating a automatic bug testing system and when we receive outstanding examples we can add them into our testing rig for future use, which also means we can be sure your issue is fixed when the developer says it is resolved.
- If it's a Critical/A/B bug which affects code or your project (saving/importing, etc.) ensure you have a suitable .gmz ready before submitting the report, as one is very likely to be asked of you if you don't add it straight away.
- Either attach the sample when creating the report or host it on your own server and add a URL. Mantis allows attachments up to 10MB in size.
- A small sample which just shows the issue is always preferable to a full game, but if you must submit an entire game please give us clear reproduction steps and ideally any applicable cheats/save games so we can see the issue as quickly and easily as possible.
- Please comment your code well - we need to be able to understand the sample as quickly as possible.
- Screenshots are always welcome for highlighting graphical issues and for showing hard-to-reproduce issues, but please don't add lots of text to the screenshot instead of filling out the Mantis form.
- Feel free to highlight the line(s) of code you've identified as the problem, but pasting code blocks is rarely a decent substitute for a sample .gmz (and we'd still have to create one ourselves anyway).
- View Status: Drop-down - Public or Private – If your example or your report contains sensitive information then feel free to make it private, but ideally we'd prefer reports to be public. Another reason to use samples instead of submitting your full game.
This is the report as it should now appear in Mantis
I hope that reading this has led you to understand what we hope to achieve with Mantis and bug-tracking going forward and that any improvement in the quality of reporting has a great knock-on effect for our efficiency overall.
If you require any additional information/quidelines or have any questions regarding the topic above, feel free to contact us using the helpdesk.
Thank you for your time.