Overview of Testing Strategy for Filesync Version 1.1 of 06/18/96 Testing Goals The goals of the filesync test effort are: (1) to provide a fully automated, exhaustive, functional validation test suite that can be run by anyone, whether they are familiar with filesync or not. (2) to provide a permanent repository for the large and growing set of test cases that are required to fully exercise the complex capabilities and subtleties of the filesync program. (3) to provide an easy repository for new regression test cases that become appropriate as bugs are found. (4) to provide a stress scenario that is broad enough and deep enough to give us confidence that filesync can be trusted with mission critical data. (5) to provide a fully automated tool that can be run on all platforms for unit testing. (6) to provide a fully automated tool that can be run on all platforms for pre-integration testing. (7) to provide a fully automated tool that can be run on all platforms for system testing. Testing Methodology Fortunately the filesync program is entirely data driven and has (virtually) no user interaction. This makes it a very good candidate for fully automated testing. The basic methodology for testing filesync functionality is: identify a feature to be tested create a set of rules, a pair of directory trees (and potentially a state file) whose relationship will stress the feature to be tested. run filesync to analyze and or reconcile the situation, capturing its verbose commentary. examine the output to ensure that filesync has properly recognized and characterized the situation, and that it has identified the proper course of action. where appropriate, run additional programs on the post- reconciled directory hierarchies in order to determine that the appropriate reconciliation actions were, in fact, properly carried out. Ultimately the automated testing will work by comparing the obtained output (from filesync and the post-reconciliation examinations) with golden output. The golden output will be obtained by running the test cases, and carefully examining the output to determine that the programs have behaved exactly as expected. Doing this will surely find many bugs, and final golden output will not be available until all of these bugs have been identified and fixed. A few special utilities will be needed in order to implement this testing methodology: a utility to automatically construct directory trees according to a concise set of (test case specific) requirements. a utility to obtain/validate information about files that is not readily available from standard unix commands high resolution access and modification times the type of the file system on which a file resides a test harness to carry out the steps of the above described testing methodology an overarching test harness to gather all of the test cases together into a single suite, run all of them, and produce overall pass/fail reports. Defining Test Cases How will we identify specifically what test cases need to be included in the test suite? Test cases can be broadly broken down into four classes: does the program properly deal with error conditions? mis-specified parameters missing files files that are not adequately accessible files with unexpected types files with unexpected contents does the program properly analyze directory trees? does it properly enumerate all trees does it properly interpret all analysis rules does it properly recognize all types of files does it properly note all relevant file attributes does the program properly identify and characterize changes does it note all types of differences does it appropriately disregard unimportant differences does it properly summarize the important differences does the program properly reconcile differences what does it decide to do in every possible case does it properly implement its decisions Error conditions relating to parameters and parameter files can be enumerated by reviewing the specification for all possible parameters, and then writing test cases around all imaginable ways in which any parameter could be mis-specified. Similarly, error cases for file contents can be generated by examining the required syntax, and trying to violate all rules in all possible ways. Error conditions in analysis and characterization can be enumerated in two ways. In an initial black-box pass, we will consider all relevant types of files, and all conceivably interesting things that could distinguish them. In the second (white-box) pass, we will examine the code to see what things the program looks for in various files, and what would trigger various code branches. The result of these two passes should be a fairly comprehensive list of "interesting cases". A good list of interesting reconciliation cases can be obtained in a similar two pass scheme. In the first pass, we look at the list of analysis/characterization cases, and ensure that all of them are represented in reconciliation test cases. In the second pass, we look at the reconciliation code and attempt to ensure that all code paths are exercised. This should give us a very good start, although we will almost surely miss a few important cases. As bugs are discovered through actual use, we can (a) add new regression test cases and (b) ask whether or not these bugs are evidence of a whole class of missing test cases.