Writing Tests Using the CS 70 TestingLogger Library

In CS 70, we provide a testing library to make it easy to write tests for our classes and functions, run those tests, and see the results. It is based on a TestingLogger class and three global functions:

affirm_expected(expr, expected_result)

Checks that an expression evaluates to an expected result (the most commonly used testing macro).

affirm(condition)

Checks that a condition is true (for when there isn't a specific value to test against).

affirm_throws(expr, exception_value)

Confirms that an expression would cause a specific exception to be thrown.

If you've seen assert in C, these functions are similar, although they do not stop the program if the test fails; they just log the failure, recording where in the program the failure occurred, including the function and line number.

An Example

Here's some code for testsuite.cpp that shows how this logging framework works:

#include <iostream>
#include <string>
#include <vector>
#include <sstream>
#include <stdexcept>
#include <cs70/testinglogger.hpp>

// This example test will pass because 1 + 1 = 2
bool test_addition() {
    TestingLogger log("Addition test");

    affirm_expected(1 + 1, 2);

    return log.summarize();
}

// This example test will fail
bool test_multiple() {
    TestingLogger log("Multiple tries of multiplication");

    for (int i = 0; i < 3; ++i) {
        int val = i * i;  // imagine we were *supposed* to have computed i + i
        affirm_expected(val, 2*i);
    }

    return log.summarize();
}

// This example test will pass
bool test_lessThan() {
    TestingLogger log("Less than challenge");

    constexpr int MAX = 4;
    int result = 5;   // imagine this result came from some more complex code
    affirm(result < MAX);

    return log.summarize();
}

// This example tests printed output
bool test_printedOutput() {
    TestingLogger log("Printed output check");
    std::stringstream ss;

    ss << "Hello World! " << 42;
    affirm_expected(ss.str(), "Hello World! 42");

    return log.summarize();
}

// This test tests exception handling
bool test_exceptions() {
    TestingLogger log("Exception handling gauntlet");

    // Exactly five elements
    std::vector<int> v{1,2,3,4,5};

    // But we can't access at index 5, that should throw an exception
    affirm_throws(v.at(5), std::out_of_range("*"));
    return log.summarize();
}

int main() {
    TestingLogger log("All tests");
    test_addition();
    test_multiple();
    test_lessThan();
    test_printedOutput();
    test_exceptions();

    return log.summarize(true) ? 0 : 1;  // Return 0 if all tests passed, else 1
}

You can compile this code in on the CS 70 server with

clang++ -g -std=c++20 -Wall -Wextra -pedantic -c testsuite.cpp
clang++ -g -std=c++20 -o testsuite testsuite.cpp -ltestinglogger

Notice that we need to link against the testinglogger library.

Running this example produces the following output:

Addition test passed!
FAILURE (after 1 passes): testsuite.cpp:23:     val, expecting 2*i
        actual:   1
        expected: 2

Summary of affirmation failures for Multiple tries of multiplication
----
Fails   / Total Issue
1       / 3     testsuite.cpp:23:       val, expecting 2*i

FAILURE (after 0 passes): testsuite.cpp:35:     result < MAX

Summary of affirmation failures for Less than challenge
----
Fails   / Total Issue
1       / 1     testsuite.cpp:35:       result < MAX

Printed output check passed!
Exception handling gauntlet passed!

Summary of affirmation failures for All tests
----
Fails   / Total Issue
0       / 1     [Addition test]
1       / 3     [Multiple tries of multiplication]
1       / 1     [Less than challenge]
0       / 1     [Printed output check]
0       / 1     [Exception handling gauntlet]

affirm_expected

The first two examples (test_addition and test_multiple) use affirm_expected, which takes two arguments: an expression and an expected result. The test passes if the expression is equal to the expected result. If they aren't equal, the test fails, and the message shows you both what the expression actually evaluated to, and what the expected result was.

In test_addition, the expression 1 + 1 is equal to the expected result, 2, so the test passes. Notice that when tests pass affirm_expected doesn't print anything and log.summarize() just prints a short message saying the test passed.

In test_multiple, the expression i * i evaluates to 1, but the expected result was 2. The message shows you the actual value and the expected value:

FAILURE (after 1 passes): testsuite.cpp:23:     val, expecting 2*i
        actual:   1
        expected: 2

To avoid cluttering your screen with lots of output when a test fails multiple times, the message is only printed the first time a test fails. The summary at the end of the test shows you how many times the test was run, and how many times it failed.

Summary of affirmation failures for Multiple tries of multiplication
----
Fails   / Total Issue
1       / 3     testsuite.cpp:23:       val, expecting 2*i

(If the types being tested do not support printing (via operator<<), affirm_expected still works, but the actual/expected values are not reported.)

affirm

The next example (test_lessThan) uses affirm, which takes a boolean argument. The test passes if the boolean is true, and fails if it is false.

In test_lessThan the failed affirm produces the message

FAILURE (after 4 passes): testsuite.cpp:23:       i < MAX

You should use affirm_expected when you can, because it gives you more information when a test fails. But sometimes you just want to check that something is true, and there isn't a specific value to test against. In that case, use affirm.

Testing Output with affirm_expected

So far, our examples of affirm_expected have just tested the value of an expression. Sometimes, however, we want to print something and see if it looks the way it is supposed to. We can do that with C++'s std::stringstream class.

Normally, C++'s I/O streams read or write text to or from files or to our terminal, but a std::stringstream instead targets an internal std::string object. When we write to a std::stringstream with <<, it adds our printed output characters to that string. std::stringstream's str() function returns the contents of this string object.

In test_printedOutput, we print a string and a number to ss (which is a std::stringstream) . We then use ss.str() in affirm_expected to check that the string is what we expect it to be.

affirm_throws

Sometimes functions are supposed to throw exceptions under certain conditions. The affirm_throws macro lets you test that your code throws the right exception at the right time. It's probably the testing function you'll use the least often, but that makes it all the more important for us to describe how to use it properly.

Basic Usage

affirm_throws takes two arguments:

1. An expression that should throw an exception.
2. An exception object describing what you expect to be thrown.

Here's a reminder of the code from test_exceptions above

bool test_exceptions() {
    TestingLogger log("Exception handling gauntlet");

    // Exactly five elements
    std::vector<int> v{1,2,3,4,5};

    // But we can't access at index 5, that should throw an exception
    affirm_throws(v.at(5), std::out_of_range("*"));
    return log.summarize();
}

The "*" in the exception means “any message is fine”—we just care that it threw the right type of exception.

Checking Specific Messages

If the assignment specifies an exact error message, you can test for it:

// Suppose the assignment says:
// "withdraw() must throw std::runtime_error with message
//  'Insufficient funds' if the withdrawal exceeds balance"

bool testBankAccount() {
    TestingLogger log("bank account");

    BankAccount account(50.00);

    affirm_throws(
        account.withdraw(100.00),
        std::runtime_error("Insufficient funds")
    );

    return log.summarize();
}

Wildcards and Base Classes

There are two ways to accept “any message”:

1. Use "*" for the specific exception type:

   affirm_throws(myMap.at("missing"), std::out_of_range("*"));
   

2. Use std::exception() to accept any exception type at all:

   affirm_throws(dangerousOperation(), std::exception());
   

The second form is useful when the assignment says something “should throw an exception” but doesn't specify which kind.

Respecting the Specification

Here's an example of what NOT to do:

// BAD TEST - if the assignment says negative indices are undefined behavior
bool badTest() {
    TestingLogger log("bad test");
    MyList list;

    // DON'T DO THIS - we're invoking undefined behavior!
    affirm_throws(list.get(-1), std::out_of_range("*"));

    return log.summarize();
}

Instead, only test cases where the specification guarantees an exception:

// GOOD TEST - the spec says out of bounds throws
bool goodTest() {
    TestingLogger log("good test");
    MyList list;
    list.add(42);  // list now has 1 element at index 0

    // Index 1 is out of bounds, spec says this throws
    affirm_throws(list.get(1), std::out_of_range("*"));

    return log.summarize();
}

Wrapping Up with log.summarize()

Every testing function ends with a call to the summarize() member function of our TestingLogger instance. This function returns true if all the tests passed, and false if any failed. It also prints a summary of the test. The summary is more detailed if the test failed.

Typically, our testing functions end with return log.summarize(), so that any code that wants to easily check if all the tests in a group passed can just look at the return value of the function.

The summarize() member function has an optional bool parameter (that defaults to false). Passing true causes it to produce a detailed summary that lists the passed tests as well as the failed ones. Normally that's not what we want, we just want to know what failed.

The main function is a bit of an exception. By convention, main returns 0 if there were no errors and a non-zero value if there were errors. So main returns 0 if log.summarize(true) returns true, and 1 otherwise. Also, in main, we want to end with a nice summary of everything, including all the passed tests, so we pass true to summarize(). So main usually ends with

return log.summarize(true) ? 0 : 1;

Note that if your function doesn't end with log.summarize(), the testing framework will detect that you failed to do so and complain about preventing test results from being lost. Keep it happy by always ending your testing functions with return log.summarize();.

Abandoning a Set of Tests

Sometimes, it might not be worthwhile to proceed with a set of tests. For example, you might know that right now in the development process, that whole set will all fail, or perhaps during testing you find that a test of key fundamental operations has failed and so all further tests are pointless.

One option is to just not call the testing function for that set of tests, but then the failed set won't appear in the final summary, which might make it look like everything is fine when it really isn't.

Instead, you can use log.skip() to abandon the rest of the tests in that set. This function will cause the summary to show that all the tests in that set were skipped. It's used like this:

bool test_complexFunction(bool skip) {
    TestingLogger log("complexFunction tests");

    if (skip) {
        log.skip();
        return false;  // indicate that we failed because we gave up
    }

    // ... actual tests here ...

    return log.summarize();
}

Now you can call test_complexFunction(true) to skip all the tests in that set, and the summary will show that they were skipped.

TestingLogger

The TestingLogger class is the class that actually tracks all the passed and failed affirmations and knows the name of the set of tests being worked on.

If you create a new TestingLogger when one already exists in an outer function, its overall performance (total tests and total fails) will be logged into the outer logger. That's why main produced the output

Summary of affirmation failures for All tests
----
Fails   / Total Issue
0       / 1     [Addition test]
1       / 3     [Multiple tries of multiplication]
1       / 1     [Less than challenge]
0       / 1     [Printed output check]
0       / 1     [Exception handling gauntlet]

Running without any TestingLogger

You can use affirm and affirm_expected even without a testing logger; but if there is no logger, they just abort the program with a message if a test fails. This behavior means that you can safely use affirm and affirm_expected in any CS 70 implementation code (you wouldn't be calling affirm_throws in implementation code, since that is only for tests, but technically you can use it the same way).

Conclusion

The TestingLogger framework is specific to CS 70. There are other testing frameworks for C++, but most of them have so many features that there is a lot to learn before you can use them, and we want you to focus on writing good test cases, not worry about how to use a complicated testing framework. With TestingLogger, most of the time affirm_expected will be all you need.

On this page, we've focused on how to use the testing framework. For advice on how to write good tests, see the Writing Tests help page.

(When logged in, completion status appears here.)