Changeset 688

Show
Ignore:
Timestamp:
05/01/2012 11:20:16 PM (2 years ago)
Author:
jelinson
Message:

updated result section of final report

Location:
releases/v1
Files:
3 modified

Legend:

Unmodified
Added
Removed
  • releases/v1/doc/final_report.tex

    r683 r688  
    120120Black box testing of the interface simulated the interaction a player would have with our game. These tests were used throughout to identify bugs in the GUI, as well as assess the game's performance. We performed both stress and endurance tests for the game's graphical elements. The stress tests attempted to expose bugs or mistakes in the design by using the game elements and windows against their intended purpose. Thus these tests looked for proper error handling for certain types of user input, as well as the general functionality of all the buttons and window elements. Endurance tests ensured that the program could withstand extending periods of use, by using the game for 10 to 20 minutes at a time. Lastly, we assessed the performance of the program by using Python's \verb+cProfile+ library to determine what parts of the program were inefficient and accounting for a lot of the processing time. This process enabled us to greatly reduce the running time of the program by optimizing the amount of rendering, image resizing, as well as internal window updating. Passing and performing well was deemed essential to verifying the quality of the game produced. 
    121121\subsection*{User Tests} 
    122 User tests were conducted on three occasions. The first was done on an initial user interface design that reflected the intended model of our game. The second and third instances were done using the Alpha release and a pre-Beta release. The purpose of these tests were to gather feedback on how intuitive the design was, how the clear the goals were and solicit any other feedback an outside user would offer for the game. The UI testing was conducted with middle school students and thus gave us an indication of what was clear and unclear about the layout of the window and the general gameplay. The subsequent tests were effective at determining what was confusing or vague in the overall game structure, as well as identifying areas for improvement, along with bugs. While we did not have the opportunity to test the final result with students and thus accurately assess whether we accommodated the feedback suggested throughout, all the feedback was indeed evaluated and handled in a manner intended to satisfy the most number of users. The layout and gameplay underwent many revisions as a result of this feedback. 
     122User tests were conducted on three occasions. The first was done on an initial user interface design that reflected the intended model of our game. The second and third instances were done using the Alpha release and a pre-Beta release. The purpose of these tests were to gather feedback on how intuitive the design was, how the clear the goals were and solicit any other feedback an outside user would offer for the game. The UI testing was conducted with middle school students and thus gave us an indication of what was clear and unclear about the layout of the window and the general gameplay. The subsequent tests were effective at determining what was confusing or vague in the overall game structure, as well as identifying areas for improvement, along with bugs. Users were given high-level instructions and had to navigate the interface to determine how to accomplish each task. While we did not have the opportunity to test the final result with students and thus accurately assess whether we accommodated the feedback suggested throughout, all the feedback was indeed evaluated and handled in a manner intended to satisfy the most number of users. The layout and gameplay underwent many revisions as a result of this feedback. 
    123123\subsection*{Testing Methodology} 
    124124Regression tests were performed on a regular basis with each new feature. These were run and inspected before the feature was committed as final. Black box testing was done both informally before any change commit as well as formally each week. The formal instances were done to assess that week's work and to provide a rigorous assessment of the game's current progress. As stated, user testing was done three times, each at every different stages of development. This timing was effective in providing useful feedback for each major milestone in the project.\\ \\ 
    125125As bugs were identified, a cursory inspection of the relevant code was performed. If an immediate fix was visible, the fix was made and then committed. If not, a ticket was filed and an entry was added in the Test Report, documenting the context of the bug. As bugs were addressed, both the ticket was updated as well as the entry in the Test Report. 
    126126\section*{Results} 
    127 There were many lessons that we learned throughout this process, ranging from user interfaces, to large software design to facts about technology. Regarding user interfaces, we learned about the importance of user testing. It is easy to become too familiar with your own work and sample users provide an abundance of insightful observations that would be otherwise overlooked. Moreover, user interfaces should aim for simplicity and elegance. Realizing the importance of these qualities significantly impacted the design of our layout and was some of the invaluable feedback received from users.\\ \\ 
    128 One of the most consequential lessons learned through the process was about extensible program design. After designing our alpha prototype, we nearly entirely scrapped out framework and started over. This opportunity allowed us to fully assess and critique what we had been using and design a new framework that would resolve and address those issues. In particular, our previous model was not conducive to adding new features. Each added element need to be properly linked and integrated nearly all other components. By better separating game logic and graphical elements, as well as using design principles about inheritance, we created a much more effective and efficient architecture that once in place supported many additions and modifications to the actual gameplay.\\ \\ 
    129 Lastly, we ourselves learned much about technology. In particular, certain technologies logically depend on others while some are independent. Moreover, we had to research into the different effects technology actually has on aspects of communities and the environment. Upon doing so, in addition to learning about it directly, we also learned that many players are quick to understand this in the context of a game, but perhaps might not think about it as much in the real world. Thus by instill this information and our message in the realm of a computer game, we provide an effective means of reaching the player and conveying the learning objective.\\ \\ 
    130 As described during their enumeration, the final game achieves a vast majority of the established requirements and al the deliverables have been produced in some form. We are very pleased with the outcome of our project. To reiterate, one area that would benefit from further improvement are making the art style more homogenous and perhaps even more professional quality. This could be easily achieved by designing all artwork from scratch using a single graphics editor, and sufficient artistic talent. Another area would be further optimizing the relative values of the building specs (perhaps through a genetic algorithm) for the most realistic and simultaneously fair gameplay. While we've ensured that the given values do permit the game to be won, we have not developed a sufficient metric for determining the feasibility of the current values used and thus we feel these offer areas of improvement. Lastly, while we did our best to accommodate all user feedback and make the game as intuitive as possible, because we were unable to conduct a final user test, we cannot say with certainty if our final result indeed is intuitive and easy to play, as was promised. However, our framework is designed to be readily adaptable should certain aspects be unclear and thus we feel we have produced not only the best game given the extent of feedback we've received, but also a highly flexible game that could be further improved, given more information. 
     127\subsection*{Lessons} 
     128There were many lessons that we learned throughout this process, ranging from insights on user interfaces, to large software design to facts about technology. In addition to simply learning the importance of user testing, such an endeavor demonstrated how easy it is for the developers and people familiar with the game to perceive the game much differently, and perhaps understand it more, than most users would. It is easy for a designer to become too familiar with their own work and sample users provide an abundance of insightful observations that would be otherwise overlooked. Thus we learned to appreciate the value of the outsider perspective to offer fresh ideas and feedback. Moreover, we learned that user interfaces should aim for simplicity and elegance. Clarity is essential and something our solution did not emphasize at first. Realizing the importance of these qualities in interface design significantly impacted our entire visual layout and was some of the invaluable feedback received from users.\\ \\ 
     129One of the most influential lessons learned through the process was about extensible program design. After designing our alpha prototype, we resorted to almost entirely scrapped out framework and starting over. This opportunity allowed us to fully assess and critique the architecture we had been using and design an entirely new framework that would resolve and address those issues. In particular, our previous model was not conducive to adding new features. Each added element need to be properly linked and integrated in nearly all other components. By better encapsulating and separating the game logic and graphical elements, as well as demonstrating design principles through techniques like inheritance, we created a much more effective and efficient architecture that once in place supported many additions and modifications to the actual gameplay. Moreover, by continually abstracting out information through refactoring and relying on data files to provide much of the game content, adding features became a simple task, which allowed for streamlined development.\\ \\ 
     130Lastly, we ourselves learned much about technology. In particular, certain technologies logically depend on others while some are independent. Moreover, we had to cursory research in the different effects technology has on aspects of communities and the environment in reality. Having done so, we also learned that many players are quick to understand these dynamics between technology and society in the context of a game, but perhaps might not think about it as much in the real world. Thus by instilling this information and our message in the realm of a computer game, we provide an effective means of reaching the player and conveying the learning objective. This demonstrated to us that a game can significantly affect the way the its content and message is perceived because of the context it is presented in and based off of the effectiveness of the game itself. 
     131\subsection*{Final Deliverables} 
     132The final submission includes the game directory along with a user guide that features installation instructions, game instructions, a chart demonstrating the features of the buildable items, as well as a troubleshooting and known bugs section. These are indeed all the tangible deliverables promised in the original proposal.\\ \\ 
     133While we believe all the functional requirements enumerated at the onset of the project (and included above in the Requirements and Analysis section), we believe certain requirements could be better met. In particular, we believe we have not fully optimized the actual values of item costs, heuristic triggers and resource allocation. As a result, while it is possible for a player to make ``bad'' decisions throughout the game and still win (which was the explicitly stated requirement) we feel it is nevertheless important that these values be both realistic and not such that the game becomes too challenging. Although considerably effort was made to test the difficulty of the game and the effectiveness of the values chosen, we believe this could have been performed in a more rigorous way than trial and error. In particular, future work would have entailed designed a framework to test these values (perhaps using a genetic algorithm) to find an optimal values for the various stats. The importance of doing so is that these values ultimately reflect the message we communicate to the users regarding the difficult technologies and thus they should be well-founded and meaningful. However, we consider the solution offered in our game sufficient for its intended use. Moreover, as this was prioritized as only the 10\textsuperscript{th} Functional Requirement, we consider such incompleteness acceptable. \\ \\ 
     134Regarding our non-functional requirements, we feel two areas would have benefitted from improvement, though, again, they were achieved in some form in our final result. The first of which is the consistency of the game artwork. While we continuously updated our artwork and general game aesthetics throughout development, we feel the game would still benefit from further improvement. In particular, we relied primarily on vector graphics libraries which were useful but sometimes do not mesh well across different libraries. Thus the ideal for future work would be to develop artwork from scratch that would appear homogeneously and mesh well together. Moreover, this would resolve some issues of image perspectives which are not always consistent, for instance, with the buildings on the landscapes.\\ \\ 
     135Lastly, while we believe we have achieved it, we have not had a reliable means of testing the ease of playing in our final result. We were unable to conduct user testings on our final result and thus cannot accurately gauge how effective our development was in addressing previous concerns raised. Thus, while we made every effort to ensure the game was intuitive and easy to learn, we are unable to conclude such simply by our own observation. Rather, future work should pursue conducting thorough user testings to evaluate whether this requirement was indeed achieved---something we ourselves inherently cannot determine. 
     136\subsection*{Known Bugs} 
     137Despite thorough testing and revisions, there still persist several known bugs. One of which is improperly wrapped text boxes when the game undergoes certain resizing. This was difficult to resolve because of Pygame's seemingly arbitrary measurements for fonts, which inhibited us from achieving the desire precision in a text wrapper. One means of improving this would be to develop our own font rendering library, where would be able to control the amount of precision in the letter sizing. Another known bug is in the main menu button on the game screen. Because it is simply an image, it responds to any collisions in its rectangular area, but since the image is a hexagon, this bounding rectangle includes area not included in the image. As a result, during collision testing, the button responds to clicks that ought to not trigger it. One remedy for this would be to expand the existing collision check to test for transparency in the pixel at the specified position, instead of using a bounding rectangle. 
     138\begin{comment} 
     139As described during their enumeration, the final game achieves a vast majority of the established requirements and al the deliverables have been produced in some form. We are very pleased with the outcome of our project. To reiterate, one area that would benefit from further improvement are making the art style more homogenous and perhaps even more professional quality. This could be easily achieved by designing all artwork from scratch using a single graphics editor, and sufficient artistic talent. Another area would be further optimizing the relative values of the building specs (perhaps through a genetic algorithm) for the most realistic and simultaneously fair gameplay. While we've ensured that the given values do permit the game to be won, we have not developed a sufficient metric for determining the feasibility of the current values used and thus we feel these offer areas of improvement. Lastly, while we did our best to accommodate all user feedback and make the game as intuitive as possible, because we were unable to conduct a final user test, we cannot say with certainty if our final result indeed is intuitive and easy to play, as was promised. However, our framework is designed to be readily adaptable should certain aspects be unclear and thus we feel we have produced not only the best game given the extent of feedback we've received, but also a highly flexible game that could be further improved, given more information.  
     140\end{comment} 
    131141\begin{thebibliography}{99} 
    132142\singlespacing