source: design/testing/test_plans.tex @ 316

Last change on this file since 316 was 316, checked in by jelinson, 3 years ago

fixed dead link

File size: 31.3 KB
38\newcommand{\Wlog}{Without loss of generality }
39\newcommand{\rp}{$(\Rightarrow)$ }
40\newcommand{\lp}{$(\Leftarrow)$ }
79\HRule \\
80\textsc{Erin Coughlan, Julius Elinson, Michael Morton, Rebecca Thomas}\\[.1cm]
81\textsc{\Large{Test Log \& Plan}}\\[-.1cm] % Title
82\HRule \\
85\part*{Test Results}
86Here we keep a hyerlinked log of the tests ran and the bugs found.
87\section*{Test Log}
91\begin{longtable}{| l  | l | l | p{5.5cm} | p{4.5cm} | }
93ID & Date & Time & Action & Result \\
95JE & 3/22/12 & 11:00 PM & Created directory and suite for adding unit tests & - \\
97 JE & 3/22/12 & 11:30 PM & Added section in test plan for \hyperlink{spec:init}{\tc{blue}{Spec initialization}} and implemented it & Passed\\
98 \hline
99 JE & 3/23/12 & 2:15 PM & Wrote equality and print operators for Spec class to use in test & -\\
100 \hline
101 JE & 3/23/12 & 2:30 PM & Added test for \hypl{spec:get}{\tc{blue}{Spec getters}} & Passed\\
102 \hline
103 JE & 3/23/12 & 2:40 PM & Added test for \hypl{spec:set}{\tc{blue}{Spec setters}} & Passed\\
104 \hline
105  JE & 3/23/12 & 2:50 PM & Added test for \hypl{spec:add}{\tc{blue}{Spec addition}} & Passed\\
106 \hline
107  JE & 3/23/12 & 3:00 PM & Added test for \hypl{spec:sub}{\tc{blue}{Spec subtraction}} & Passed\\
108 \hline
109   JE & 3/23/12 & 3:15 PM & Added test for \hypl{state:build}{\tc{blue}{State building adding and deleting}} & Passed\\
110 \hline
111    JE & 3/23/12 & 3:30 PM & Added section in test plan for \hypl{state:research}{\tc{blue}{State research}} and implemented it & Failed, see the \hypl{bug:negres}{\tc{blue}{negative research bug}}\\
112 \hline
113     JE & 3/23/12 & 4:00 PM & Added section in test plan for \hypl{state:calc}{\tc{blue}{State turn change}} and implemented it & Passed\\
114 \hline
115 JE & 3/26/12 & 11:00 PM & Ran black box, system test for \hypl{main}{\tc{blue}{}} as if actual user, clicking everywhere to test GUI responses &  Passed with bugs, see the \hypl{bug:button}{\tc{blue}{disappearing button bug}}\\
116 \hline
120\section*{Known Bugs}
122\begin{longtable}{| l  | l | l | p{5cm} | p{5cm} | }
124ID & Date & Time & Description & Context \\
126\hypt{bug:negres}{JE} & 3/23/12 & 4:00 PM & State erroneously allows more research to be performed than the current money, resulting in negative money. Moreover, it allows for negative research to be performed, which would provide a way to generated money. & This bug was found on the alpha release. It was found by running the Python unit test $$\verb+test_research()+$$ in the class$$\verb+/src/tests/test_state+$$ The assertion failures catch and describe the bug with a specified message.\\
128\hypt{bug:button}{JE} & 3/26/12 & 11:00 PM & The right arrow button on the map display disappears & This bug was found on the March 26 prototype release. It was found by launching the game, then New Game -> Build -> Power -> Windmill -> Buy -> End Turn. Then click on the icon of the windmill on the map and the right button disappears. Notice, however, that it disappears only for that block (which you can tell by selecting the icon of any of the other blocks)\\
133\part*{Test Plans}
134\section*{Risk Analysis}
135One high risk area of our program is the State class because it handles the calculations for all of the game logic.  It has to determine which buildings can be built based on thresholds as well as calculate changes in the stats based on the Building that was built.  Other high risk areas include the Window and WindowEntry classes because everything that is displayed on the screen is part of a Window and each Window is made up of possibly several WindowEntry objects.  Since each WindowEntry is made up of a combination of images, rectangles, and text they are mostly unique and a bug that shows up in one might not show up in another.  Additionally, the WindowEntry class determines what response should be sent based on where a click occurs, which is a vital component of the user interface.  The Game class is high risk because it handles the parsing of responses and determines which methods need to be called to complete the desired task.  Furthermore, the Map and Block classes include a lot of hand-coded resizing and dynamic changes. Thus their implementation is risky and should be tested rigorously and together since they work intimately and exclusively together. Similarly, TurnMenu has high risk because it must dynamically compute what options should be available in a particular menu at run-time and then construct such a menu. Thus the reliance on this class to provide the necessary Window containing a requested menu exposes the program to the risk of the class's correctness in its computations.
137Note that the tests are listed in the following format:\vspace{-.5cm}
140 \item[method1(arg)]-- Type of Test\hfill\\
141 Description
142  \item[method2(arg)]-- Type of Test\hfill\\
143 Description
147 \item[\hypt{main}{main()}]-- Black Box, System Test via GUI\hfill\\
148 To test the system as a whole, we will continuously and relentlessly play our game. This will function to expose any errors that become apparent to a player over the course of extended use. We will explore new routes each time and test new features from the user standpoint as they are added. Essentially, we will test the entire software as a whole by approaching it like our players will and note the bugs that arise. Since every possible sequence of moves cannot be simulated, we will play extreme cases like only buying/researching in one category and technology, trying to do poorly in the game, and rigorously exploring new elements as they are added to the game.
152 \item[\hypt{state:build}{addBuilding(building), deleteBuilding(building)}] -- White Box, Integration, Regression Test \hfill\\
153 The purpose of this test is to ensure that State correctly updates as it interfaces with the Specs and Building classes upon adding a building. This is a key component to the game logic as a player progresses. Moreover, it is complicated because it involves changes to States due to choices made in the most recent turn and choices made throughout the entire game. Since this ultimately keeps track of the player's game, we want to rigorously test its calculations and its integration with the other classes used to retain data. This will be a white box test because we will observe the arithmetic used in updating values to try to find boundary cases, such as negative values, in order to test the correctness of our implementation. This will be a scripted test where we will create a number of different States and simulate both different changes to the game and the effect of multiple turns elapsing, while at each step checking the current State with the expected State. This will be a regression test that we will run as new States or building types are added, ensuring that the method works as it should under such changes and that the game logic is robust regardless of other implementation changes.
154  \item[\hypt{state:research}{research(category, value)}] -- White Box, Integration, Regression Test \hfill\\
155 The purpose of this test is to ensure that State correctly updates as it interfaces with the Specs and Building classes upon performing research. This test is very similar in spirit to the \verb+addBuilding+ test, but since it receives a raw value representing how much to invest, it also tests more boundary values. In particular, it checks if the research exceeds the amount of money available, as well as if the amount to research is negative, which would allow the player to generate money.
156 \item[\hypt{state:calc}{calcTurnChange()}] -- White Box, Integration, Regression Test \hfill \\
157 The purpose of this test is to ensure that State correct updates at the end of the turn based on the continuous Specs of its current Buildings. This reinforces the tests for Spec arithmetic, as well as ensures that the game logic will accurately compute changes to the player's progress at the end of each turn. The test will be crafted to the particular implementation to explore edge cases and hence will be a white box test. This will be a regression test that we will run as new States or building types are added, ensuring that the method works as it should under such changes and that the game logic is robust regardless of other implementation changes.
158  \item[displayStats()]-- Black Box, Component Test via GUI\hfill\\
159The purpose of this test is to ensure that the game data contained in Stats is properly displayed as a progress bar. Since this is all hand-coded using relative values to determine sizing of the rectangles representing the bars, the implementation is very risky and prone to bugs. Thus we will simulate multiple States and various changes to them and observe the results. Since the return value is simply a Window, the test must be done through the GUI and inspected for correctness.
165\item [\hypertarget{spec:init}{\_\_init()\_\_}] --\\
166White Box, Unit Test\hfill\\
167The purpose of this test is to ensure that the initial starting values for a game Spec are properly initialized to the constants defined in a specific external file.
168 \item[\hypt{spec:get}{getMoney(), getPower(), getMaterial(), getSpace(), getPollution()}] --\\ Black Box, Unit Test\hfill\\
169 The purpose of testing these similar methods is to ensure that the Stats returned have the correct values.  These stats are the main component of tracking progress in the game, and ultimately determining if the player wins or loses.  We will test this method by creating various Specs using boundary case values (like negative and floating point values) as stats and making sure that each of these are returned correctly.  This is a black box, unit test because we are simply testing that the methods work as they should, without designing the tests around their implementation. 
170  \item[\hypt{spec:set}{setMoney(newMoney), setPower(newPower), setMaterial(newMaterial)},]\hfill\\\textbf{setSpace(newSpace), setPollution(newPollution)} -- Black Box, Unit Test\hfill\\
171 The purpose of testing these similar methods is to ensure that we can change the stats in a given Spec correctly.  As before, these stats are the main means of tracking progress in the game.  We will test this method by creating various Specs using boundary case values for the Stats and then changing them.  We will also check to ensure that this method provides a deep copy and not a shallow one. Since the getStat() method would be tested before, we can call that method to make sure the Stats are correctly changed.  Therefore, this will be a black box unit test because we will write the test knowing the interface but not the implementation.
172 \item[\hypt{spec:add}{\_\_isub\_\_(otherSpec)}, \hypt{spec:sub}{\_\_iadd\_\_(otherSpec)}]-- Black Box, Unit Test\hfill \\
173 Note that these correspond to two similar, but separate tests. The purpose of testing these methods is to make sure that we can add and subtract Specs throughout the game.  This is integral of the game logic because at the end of every turn, these Specs are used to update the State of the game.  It is a data structure that needs to work correctly for the rest of the game to function properly.  We will test this by creating various Spec using boundary case values.  We will then add and subtract the Specs and check to ensure we get the expected result.  We also will look at adding or subtracting one Spec from itself to make sure that there are no problems in this regard.  This will be a black box unit test since we know that we can add and subtract, but do not need to know the implementation to design the test. \end{description}
176 \item[response(pos)]-- White Box, Component Test\hfill\\
177The purpose of testing this method is to ensure that the default response for the Window object is returned when appropriate. This will be a white box test because we will design the tests knowing the Window object first checks its components for a response and only returns the default response when there were no collisions. Since this function takes coordinates and returns a signal, we can write a component test that creates various Window objects and then computes responses for various input positions and then checks for the expected response.
178  \item[response(pos)]-- White Box, Integration, Regression Test\hfill\\
179The purpose of this second test for the same method is to ensure that Window and WindowEntry integrate as they should. The testing procedure will be similar to the first test of this method, but will be targeted at checking for correctly translating coordinates and correctly accessing data members of WindowEntry from in Window. As before, this will be a white box test since the tests will consider the order in which responses are determined. Lastly, this will be a regression test because modifications to these classes and elsewhere should not affect the process for determining responses, which is an essential element to the operation of the GUI. Thus we will want to this framework to consistently operate as it should.
180 \item[render(surface)]-- Black Box, Integration, Regression Test via GUI\hfill\\
181As with the corresponding test for WindowEntry, this test will ensure that WindowEntry objects are rendered properly on top of each on a single surface. As this returns a visual result, we will have to test it by creating Windows with different WindowEntry objects and then observing that they render correctly. Since this is ultimately a test of our understanding of how PyGame renders on a surface, it will be a black box test, since the tests will not be specific to any method implementation.
190 \item[getSpec()]-- Gray Box, Unit, Integration Test\hfill\\
191 The purpose of testing this method is to ensure that the continuous and initial Specs can be returned.  Thus, this is an integration test because it relies on the Spec class.  We will test this by creating Specs as in the Spec class tests and then making sure these same Specs are returned.  We will also test changing the Specs using the Spec class, and then trying to get the Spec.  This is a gray box unit test because we know that Specs are being used and that we can change these, but we do not know exactly how this method works.
192  \item[setSpec(newSpec)]-- Gray Box, Unit, Integration Test\hfill\\
193 The purpose of testing this method is to ensure that we can update Specs outside of the Spec class, by replacing the entire Spec.  We will test this method by creating various Specs as in the Spec class tests and then changing these values.  We will also test this by assigning one Spec to another and making sure each still updates and is displayed on their own.  Since the getSpec() method will be tested before this, we can use it to make sure the Specs are correctly changed as it would be used in the rest of the program.  Therefore, this is a gray box unit test as well as an integration test since we rely on the Spec class working correctly, and we do not know the exact implementation of the method.
198\subsection*{Block \& Map}
200 \item[addBuilding(name)]-- Black Box, Integration Test\hfill\\
201 The purpose of testing this method is to ensure that any given building can be added to the map.  This is important because this is one of the main means of feedback and entertainment for the player.  We will test this by adding buildings to the map, including the special cases when we add  buildings that already exist as well as when we add those same buildings in different map Blocks. This is black box integration test as it tests the interface between Block and Map, but only using Map's interface to design the test.
202  \item[endTurn()]-- Black Box, Integration Test\hfill\\
203 The purpose of testing this method is to ensure that at the end of a turn, the map is updated with all the correct buildings that were made during one turn.  We will test this by creating a list of buildings that will need to be updated in the map.  To test some boundary cases, we will add multiples of the same buildings as well as add no buildings during a turn.  Because we do not need to know how buildings are being added in these tests, this is a black box test.
204 \item[checkSpace()]-- White Box, Unit/Integration Test\hfill\\
205 The purpose of testing this method is to ensure that we can correctly detect when there is space to build a building.  We will test this by asking to add buildings one at a time, including when there is no more room on a given screen.  We will make sure that the buildings are only allowed to be added when there is space.  Since we know what the max number of buildings on a screen is, this is a white box test.  It is a unit test because the information depends only on this method and nothing else, while still being an integration test because it involves the interface between Map and Block.
209\subsection*{Deprecated Classes as of March 26 Prototype}
213 \item[endTurn()]-- White Box, Integration Test\hfill\\
214 The purpose of testing this method is to ensure that the game can be updated at the end of every turn.  This is extremely important because this method interfaces between several other classes in order to update every aspect of the game. Thus we will test this higher-level method correctly calls on the lower-level methods to end a turn in the correct order and that there are no unexpected side effects. We will test this method for turns that include multiple actions and no actions as boundary cases. Furthermore, one of the risks associated with this function is that it transfers data from one State to another. Since this changes critical information, we will want to ensure this function works properly and will design the test by examining how this operation is performed.
215  \item[createWindow(name)]--White Box, Component Test\hfill\\
216 The purpose of testing the createWindow method is to ensure that the correct Windows are created when prompted.  This is important because this information will be used by main to create a list of the current Windows, which are displayed in the UI class.  Since we know that the list is split on the underscore character, we will test this method by giving it a sequence that involves many underscores in a row as well as sequences that have no underscores.  This is a white box test because we know exactly how the information is being parsed.  It is a component test because the parsing is the base of many other classes as well as this function encompasses one of the main jobs of the Game class.
217 \item[build(name)]-- Black Box, Integration Test\hfill\\
218 The purpose of testing the build method is to ensure that the buildings the user decides to create are indeed being created, that there is space for the building to be created, and that the building actually can be created based on the current game progress.  Because of this task, the build method calls upon other classes, such as Map and Turn, and therefore this will be an integration test.  We will test that this method correctly calls the other classes' methods without side effects and that the overall response is that a new building is created, as desired.  We will try edge cases, such as creating buildings when there is have no space in the current block, insufficient stats in the State, and building something has not yet been unlocked based on the game progress.  This will be a black box test as we will examine Game's interface without looking at its implementation in order to test these situations.
223 \item[calculateBuildables(State)]-- Black box, Integration Test\hfill\\
224 The purpose of this test is to ensure that TurnMenu can properly interface with State, as well as Building, to correctly determine what can and cannot be built by the player at a given moment. This can be tested by passing the method a number of different States and compare the returned result with expected values of what is and isn't buildable. This is important because menu options change as the game proceeds and we want to ensure we will be able to correctly accommodate those changes. This will be a black box test because we will simply design the tests around the method interface, without examining the implementation.
225  \item[createBuildMenu(State)]-- White Box, Component Test via GUI\hfill\\
226 The build menu is dynamically created and thus this function constructs the appropriate Window to return for a given State. A major concern is whether the Window will have overflow if there are a lot of things that are buildable. This will be a white box test, as it will examine the for loop that is used to create the menu and try to cause an overflow. It will also test the ability to retrieve the necessary information and image for the menu items during run time as soon as the menu is opened. Since the return result is a Window, we will have to evaluate the results of the test with the GUI.
231 \item[isCollision(pos)]-- White Box, Unit, Regression Test\hfill\\
232The purpose of testing this method is to ensure the correctness of the technique to determine if a collision has occurred with a WindowEntry, which is essential for parsing graphic user interface input. However, because the parameters are just a position, we can test this method by constructing a WindowEntry with an assortment of different rectangles and images, and test the collision response for a number of coordinates. We will test boundary cases like negative coordinates, empty WindowEntry objects, etc. Thus it will be a simple unit test of this specific method and will be a white box because we will design the tests knowing the order in which the WindowEntry's individual components are tested for collisions. Furthermore, this is a regression test because we want to ensure this functionality remains intact as we develop elsewhere and possibly add to the contents of WindowEntry.
233\item[render(surface)]-- White Box, Component, Regession Test  via GUI\hfill\\
234The purpose of testing this method is to ensure that elements of a WindowEntry are rendered properly onto a surface and in the correct order. Since the return type is a surface, the test requires visual inspection to verify correctness through the GUI. We will construct WindowEntry objects with an assortment of overlapping components, added in different orders, and verify how they are rendered. This will be a white box test because we will design the tests cognizant of the order in which a WindowEntry's components are rendered to the surface. Furthermore, this is a regression test since this is how all visuals are rendered and we want to ensure this remains correct throughout development.
240 \item[eventCheck()]-- Black Box, System, Regression Test via GUI\hfill\\
241 The purpose of testing this method is to ensure that the user interface is working correctly and can recognize the user's mouse clicks.  We need to make sure that the clicks that are recognized have the same responses that the user expects.  If this method does not work correctly, the game will not function properly and the user will become frustrated, which will decrease the learning opportunities.  Knowing this, we plan on testing this method by clicking along the boundaries of buttons to make sure that no clicks are wrongly interpreted.   We will also make sure that each button has the proper response when clicked on.  Because this method is the primary means of interfacing with the user, this will be a regression test used throughout any GUI test.  Since the results will be visual, this is a black box test that will be done using GUI.
244As each individual test offers its own rationale, we will discuss the cumulative adequacy of the entire test plan based on the different types proposed. \\ \\
245We propose unit tests for WindowEntry, Spec, and Map \& Block. These are the primary low-level classes and retain actual game data. Thus by testing all of these with unit tests, we ensure the foundation of our program and provide a framework to extend beyond these classes in testing higher level classes, some of which must interface with these classes. Therefore these unit tests would allow us to proceed in checking more complicated classes and methods, knowing the simple components work as they should.\\ \\
246We propose component tests for WindowEntry, Window, TurnMenu, State and Game. All of these classes contain several methods that must be tested in tandem in order to fully test the class. Moreover, these classes all retain game data and component tests will ensure that the data is handled and manipulated properly through the various methods that they perform. Ensuring the accuracy of the internal game data is an essential requirement that will be addressed by these component tests.\\ \\
247We propose integration tests for Window, TurnMenu, State, Game and UI. These are the primary classes that have to work intimately with the other classes. In particular, all of these classes must interface with each other at some level. Thus we intentionally build in some redundancy in our testing by designing tests that target the interfacing performed in each of these classes. Moreover, the other classes not included here, like WindowEntry and Spec, do not access data from other classes, but instead are low-level. Thus integration tests for these classes that manage and interact with various other classes should be sufficient and thorough.\\ \\
248We propose system tests for UI and the entire program via A system test for our program would necessarily entail testing all the components by playing the game. We will use the game, paying attention both to the correctness of the UI responding to input and the general correctness of game logic and displays. By intentionally playing the game by looking for edge cases, focusing on new features and playing often, we will expose bugs that are apparent to the player, which are the types of bugs that must be infrequent as possible. Essentially, by experiencing the program as any user would, we will ensure that the game functions cohesively and as it should, testing the complete framework of the system.\\ \\
249Regression tests are proposed for WindowEntry, Window, State, and UI. The reason for testing these classes with regression tests is that these are the core classes that retain game data and dictate the program behavior. Thus as we add features and extend the program, these classes should retain entirely functional and unaffected. In other words, these are the classes that should work independently of changes elsewhere and should scale as the project and game does. Moreover, an error in these classes would cause an error in most other classes. Thus by using regression tests on these classes, we will isolate any bugs that do exist more effectively.\\ \\
250We propose both white and black box testing throughout. Any tests that require visual results to confirm the correctness of classes or methods was cast as a black box test since it will be handled through the GUI, which doesn't allow insight into the implementation. Moreover, simple methods that do not contain particularly complicated implementations can be tested via black box tests by only looking at the class interface. These tests simply ensure that the desired results are provided for particular methods and don't try to break a messy implementation. White box tests, however, address methods that are more complicated and contain more control statements. These tests were proposed for any methods with tricky implementations and for methods where the ordering of certain operations matters (like in what order things are rendered or checked for collisions). Moreover, white box tests were proposed for things that seem susceptible of working improperly (like TurnMenu that dynamically builds the menu with the chance for overflow).\\ \\
251We deem our test coverage thorough. We have proposed tests for all non-trivial classes in our program and for each nontrivial method. Moreover, various integration tests will test the same classes and methods multiple times. Thus each relatively risky class is addressed with tests both focusing on that class and via integration tests, as well as indirectly through system tests. This threefold approach to testing each essential to the game will ensure the coverage has both depth and breadth.\\ \\
252Some shortcomings of our test plan include a heavy reliance on GUI tests. These tests are cumbersome, require a user to walk through and do not provide clear, quantitative results. However, since this will be how the user experiences the program, it is important that we test from this perspective. Another shortcoming of our proposed tests is that Block and Map are tested together and thus a bug would require first determining in which class the problem is in. However, the reason for this coupling is that the classes work intimately together and the tests would be too contrived and unhelpful to test these classes in isolation.  Lastly, our test plans offered no system test that doesn't use the GUI. It was not clear how a test that did not require the GUI could be designed since in order to test the entire framework, since we rely on graphical user input for game events to occur. Nevertheless, it would be preferred if we could run an automated test that would check essential operations in the system as a baseline for the program's correctness.
Note: See TracBrowser for help on using the repository browser.