This section details our protocols for continuing our test plan as the size of the project increases and more code is added. This protocol will hopefully give us the ability to grow our testing along with our code at a manageable rate.
Developing New Tests
When developing new code or expanding on existing code, it is important that our testing expands as well in order to maintain optimal functioning of the program and prevent from having errors that spiral out of control and cause issues later on in the project. To this end, after each coding session, if any major changes or additions are made, the coders are responsible for creating a new test or modifying existing ones if the code update has outdated them, and running these test (see also regression tests) before committing any changes to SVN. All major functionality of the larger methods must be documented in these new tests and uploaded to SVN with the updated code itself.
Running Regression Tests:
Regression tests should be run, and any issues that arise must be fixed before submitting code to SVN.
Weekly Test Reports:
Along with the already existing weekly prototype/coder report, we plan on having a weekly testing report as well in order to update and document our progress in the test suite as well as the code itself. All of the changes to the code will have their associated tests, created as a result of our protocol for developing new tests (see above), documented in the weekly report. This will include the major elements being tested, the manner in which they are being tested, and current (expected) results, as well as their expected risk. Furthermore there will also be a section of the report that will be dedicated to suggested updates to User Testing - what we want testers to be aware of and what areas we want them to test most rigorously according to our risk analysis and code coverage.
Documentation and Tracking of Bugs:
When testing the code ourselves or with outside user testing, any applicable bugs if not able to be fixed immediately upon their appearance will be documented via the ticket system of the track as well as in the weekly report. There is already in place a system for tracking and assigning responsibility for fixing bugs on the team’s wiki - tickets labeled with the type defect will be classified as bugs and will have their own list available on the trac - each new bug that appears will be given a ticket, accordingly and organized with its risk and importance - critical through trivial. When fixing a bug, team members will “accept” a ticket and upon resolution close it, or update the ticket with any additional information or progress they have determined at the point that they need to stop.
Additionally, bugs that come up each week of testing, if not resolved promptly will be update in the weekly code/prototype with information about the exact circumstances and issues regarding the particular bug, as well as any progress that has been made by those working on it prior to the update.
With regards to user tests, we will have a standardized manner in which users will record any bugs or issues that they encounter while testing the game. The user will be asked to record his action or sequence of actions that led to the appearance of the bug, the state of the game at the point of breakdown, as well as any other mitigating factors that might be a factor (the machine that the game was running on at the time, discrepancies between normal testing and game play procedure or process). These reports will then be given to a team member who will attempt to fix the issue or update tickets as necessary.