It’s just as important for test code to be readable as it is for nontest code. Other coders will often look at the test code as unofficial documentation of how the real code works and should be used. So if the tests are easy to read, users will better understand how the real code behaves.

When test code is big and scary, here’s what happens:

Instead, you want to encourage users of your code (especially you!) to be comfortable with the test code. They should be able to diagnose why a new change is breaking an existing test and feel like adding new tests is easy.

In our codebase, we had a function to sort and filter a list of scored search results. Here’s the function declaration:

// Sort 'docs' by score (highest first) and remove negative-scored documents.
void SortAndFilterDocs(vector<ScoredDocument>* docs);

The test for this function originally looked something like:

void Test1() {
    vector<ScoredDocument> docs;
    docs.resize(5);
    docs[0].url = "http://example.com";
    docs[0].score = -5.0;
    docs[1].url = "http://example.com";
    docs[1].score = 1;
    docs[2].url = "http://example.com";
    docs[2].score = 4;
    docs[3].url = "http://example.com";
    docs[3].score = -99998.7;
    docs[4].url = "http://example.com";
    docs[4].score = 3.0;

    SortAndFilterDocs(&docs);

    assert(docs.size() == 3);
    assert(docs[0].score == 4);
    assert(docs[1].score == 3.0);
    assert(docs[2].score == 1);
}

There are at least eight different problems with this test code. By the end of the chapter, you’ll be able to identify and fix all of them.

As a general design principle, you should hide less important details from the user, so that more important details are most prominent.

The test code from the previous section clearly violates this principle. Every detail of the test is front and center, like the unimportant minutiae of setting up a vector<ScoredDocument>. Most of the example code involves url, score, and docs[], which are just details about how the underlying C++ objects are set up, not about what this test is doing at a high level.

As a first step in cleaning this up, we could create a helper function like:

void MakeScoredDoc(ScoredDocument* sd, double score, string url) {
    sd->score = score;
    sd->url = url;
}

Using this function, our test code becomes slightly more compact:

void Test1() {
    vector<ScoredDocument> docs;
    docs.resize(5);
    MakeScoredDoc(&docs[0], -5.0, "http://example.com");
    MakeScoredDoc(&docs[1], 1, "http://example.com");
    MakeScoredDoc(&docs[2], 4, "http://example.com");
    MakeScoredDoc(&docs[3], -99998.7, "http://example.com");
    ...
}

But this isn’t good enough—there are still unimportant details in our face. For instance, the parameter "http://example.com" is just an eyesore. It’s always the same, and the exact URL doesn’t even matter—it’s just needed to fill out a valid ScoredDocument.

Another unimportant detail we’re forced to see is docs.resize(5) and &docs[0], &docs[1], and so on. Let’s change our helper function to do more work for us and call it AddScoredDoc():

void AddScoredDoc(vector<ScoredDocument>& docs, double score) {
    ScoredDocument sd;
    sd.score = score;
    sd.url = "http://example.com";
    docs.push_back(sd);
}

Using this function, our test code is even more compact:

void Test1() {
    vector<ScoredDocument> docs;
    AddScoredDoc(docs, -5.0);
    AddScoredDoc(docs, 1);
    AddScoredDoc(docs, 4);
    AddScoredDoc(docs, -99998.7);
    ...
}

This code is better, but still doesn’t pass the “highly readable and writable” test. If you wanted to add another test with a new set of scored docs, it would require a lot of copying and pasting. So how do we go about improving it further?

CheckScoresBeforeAfter("-5, 1, 4, -99998.7, 3",  "4, 3, 1");

CheckScoresBeforeAfter({-5, 1, 4, -99998.7, 3}, {4, 3, 1});

void CheckScoresBeforeAfter(string input, string expected_output) {
    vector<ScoredDocument> docs = ScoredDocsFromString(input);
    SortAndFilterDocs(&docs);
    string output = ScoredDocsToString(docs);
    assert(output == expected_output);
}

vector<ScoredDocument> ScoredDocsFromString(string scores) {
    vector<ScoredDocument> docs;

    replace(scores.begin(), scores.end(), ',', ' ');

    // Populate 'docs' from a string of space-separated scores.
    istringstream stream(scores);
    double score;
    while (stream >> score) {
        AddScoredDoc(docs, score);
    }

    return docs;
}

string ScoredDocsToString(vector<ScoredDocument> docs) {
    ostringstream stream;
    for (int i = 0; i < docs.size(); i++) {
        if (i > 0) stream << ", ";
        stream << docs[i].score;
    }

    return stream.str();
}

The preceding code was nice, but what happens when that assert(output == expected_output) line fails? It produces an error message like this:

Assertion failed: (output == expected_output),
    function CheckScoresBeforeAfter, file test.cc, line 37.

Obviously, if you ever saw this error, you’d wonder, What were the values of output and expected_output?

assert(output == expected_output);

BOOST_REQUIRE_EQUAL(output, expected_output)

test.cc(37): fatal error in "CheckScoresBeforeAfter": critical check
    output == expected_output failed ["1, 3, 4" != "4, 3, 1"]

File "file.py", line X, in <module>
    assert a == b
AssertionError

import unittest

class MyTestCase(unittest.TestCase):
    def testFunction(self):
        a = 1
        b = 2
        self.assertEqual(a, b)

if __name__ == '__main__':
    unittest.main()

File "MyTestCase.py", line 7, in testFunction
        self.assertEqual(a, b)
AssertionError: 1 != 2

output == expected_output failed ["1, 3, 4" != "4, 3, 1"]

CheckScoresBeforeAfter() failed,
  Input:           "-5, 1, 4, -99998.7, 3"
  Expected Output: "4, 3, 1"
  Actual Output:   "1, 3, 4"

void CheckScoresBeforeAfter(...) {
    ...

    if (output != expected_output) {
        cerr << "CheckScoresBeforeAfter() failed," << endl;
        cerr << "Input:           \"" << input << "\"" << endl;
        cerr << "Expected Output: \"" << expected_output << "\"" << endl;
        cerr << "Actual Output:   \"" << output << "\"" << endl;
        abort();
    }

There’s an art to choosing good input values for your tests. The ones we have right now seem a bit haphazard:

CheckScoresBeforeAfter("-5, 1, 4, -99998.7, 3",  "4, 3, 1");

How do we choose good input values? Good inputs should thoroughly test the code. But they should also be simple so that they’re easy to read.

For example, suppose we had just written:

CheckScoresBeforeAfter("1, 2, 3", "3, 2, 1");

Although this test is simple, it doesn’t test the “filter negative scores” behavior of SortAndFilterDocs(). If there were a bug in that part of the code, this input wouldn’t trigger it.

On the other extreme, suppose we wrote our test like this:

CheckScoresBeforeAfter("123014, -1082342, 823423, 234205, -235235",
                       "823423, 234205, 123014");

These values are needlessly complex. (And they don’t even test the code thoroughly.)

Simplifying the Input Values

So what can we do to improve these input values?

CheckScoresBeforeAfter("-5, 1, 4, -99998.7, 3",  "4, 3, 1");

Well, the first thing you probably noticed is the very “loud” value -99998.7. That value was just meant to be “any negative number,” so a simpler value is just -1. (If -99998.7 was meant to be “a very negative number,” a better value would have been something crisp like -1e100.)

Key Idea

Prefer clean and simple test values that still get the job done.

The other values in our test aren’t too bad, but while we’re here, we can reduce them to the simplest integers possible. Also, only one negative value is needed to test that negative values are removed. Here’s a new version of our test:

CheckScoresBeforeAfter("1, 2, -1, 3", "3, 2, 1");

We’ve simplified the test values without making them any less effective.

Large “Smasher” Tests

There is definitely value in testing your code against large, crazy inputs. For instance, you might be tempted to include a test like:

CheckScoresBeforeAfter("100, 38, 19, -25, 4, 84, [lots of values] ...",
                       "100, 99, 98, 97, 96, 95, 94, 93, ...");

Large inputs like these do a good job of exposing bugs such as buffer overruns or others you might not expect.

But code like this is big and scary to look at and not completely effective in stress-testing the code. Instead, it’s more effective to construct large inputs programmatically, constructing a large input of (say) 100,000 values.

CheckScoresBeforeAfter("2, 1, 3", "3, 2, 1");    // Basic sorting
CheckScoresBeforeAfter("0, -0.1, -10", "0");     // All values < 0 removed
CheckScoresBeforeAfter("1, -2, 1, -2", "1, 1");  // Duplicates not a problem
CheckScoresBeforeAfter("", "");                  // Empty input OK

Test code is typically organized into functions—one for each method and/or situation you’re testing. For instance, the code testing SortAndFilterDocs() was inside a function named Test1():

void Test1() {
    ...
}

Picking a good name for a test function can seem tedious and irrelevant, but don’t resort to meaningless names like Test1(), Test2(), and the like.

Instead, you should use the name to describe details about the test. In particular, it’s handy if the person reading the test code can quickly figure out:

A simple approach to construct a good test function name is to just concatenate that information together, possibly with a “Test_” prefix.

For instance, instead of naming it Test1(), we can use the Test_<FunctionName>() format:

void Test_SortAndFilterDocs() {
    ...
}

Depending on how sophisticated this test is, you might consider a separate test function for each situation being tested. You could use the Test_<FunctionName>_<Situation>() format:

void Test_SortAndFilterDocs_BasicSorting() {
    ...
}

void Test_SortAndFilterDocs_NegativeValues() {
    ...
}

...

Don’t be afraid of having a long or clunky name here. This isn’t a function that will be called throughout your codebase, so the reasons for avoiding long function names don’t apply. The test function name is effectively acting like a comment. Also, if that test fails, most testing frameworks will print out the name of the function where the assertion failed, so a descriptive name is especially helpful.

Note that if you’re using a testing framework, there might already be rules or conventions on how methods are named. For instance, the Python unittest module expects test method names to start with “test.”

When it comes to naming helper functions in your test code, it’s useful to highlight whether the function does any assertions itself or is just an ordinary “test-unaware” helper. For instance, in this chapter, any helper function that calls assert() is named Check...(). But the function AddScoredDoc() was named just like an ordinary helper function.

At the beginning of the chapter, we claimed there were at least eight things wrong with this test:

void Test1() {
    vector<ScoredDocument> docs;
    docs.resize(5);
    docs[0].url = "http://example.com";
    docs[0].score = -5.0;
    docs[1].url = "http://example.com";
    docs[1].score = 1;
    docs[2].url = "http://example.com";
    docs[2].score = 4;
    docs[3].url = "http://example.com";
    docs[3].score = -99998.7;
    docs[4].url = "http://example.com";
    docs[4].score = 3.0;

    SortAndFilterDocs(&docs);

    assert(docs.size() == 3);
    assert(docs[0].score == 4);
    assert(docs[1].score == 3.0);
    assert(docs[2].score == 1);
}

Now that we’ve learned some techniques for writing better tests, let’s identify them:

Some code is easier to test than other code. Ideal code to test has a well-defined interface, doesn’t have much state or other “setup,” and doesn’t have much hidden data to inspect.

If you write your code knowing you’ll be writing a test for it later, a funny thing happens: you start designing your code so that it’s easy to test! Fortunately, coding this way also means that you create better code in general. Test-friendly designs often lead naturally to well-organized code, with separate parts to do separate things.

Of all the ways to break up a program into classes and methods, the most decoupled ones are usually the easiest to test. On the other hand, let’s say your program is very interconnected, with many method calls between your classes and lots of parameters for all the methods. Not only would that program have hard-to-understand code, but the test code would be just as ugly, and hard to read and write.

Having lots of “external” components (global variables that need to be initialized, libraries or config files that need to be loaded, etc.) also makes it more annoying to write tests.

Generally, if you’re designing your code and realize, Hmm, this is going to be a nightmare to test, that’s a good reason to stop and rethink the design. Table 14-1 shows some typical testing and design problems.

On the other hand, if you have a design that’s easy to write tests for, that’s a good sign. Table 14-2 lists some beneficial testing and design characteristics.

It’s also possible to focus too much on testing. Here are some examples:

In test code, readability is still very important. If your tests are very readable, they will in turn be very writable, so people will add more of them. Also, if you design your real code to be easy to test, your code will have a better design overall.

Here are specific points on how to improve your tests:

And above all, make it easy to modify and add new tests.