A Roadmap for Testing on an Agile Project

Brian Marick (marick@exampler.com)
(First published in Agile Times #5. Slightly revised June 28, 2004.)

If I were starting up an agile project, here is how I'd plan to do testing. (But this plan is a starting point, not the final answer.)

I assume the programmers will do test-driven design. That's well explained elsewhere (see the Further Reading), so I won't describe it here (much).

Test-driven programmers usually create their tests in what I call "technology-facing" language. That is, their tests talk about programmatic objects, not business concepts. They learn about those business concepts and business needs through conversation with a business expert.

Nothing will replace that conversation, but it's hard for programmers to learn everything they need through conversation, even the frequent conversations that a collocated business expert allows. Too often, the business expert is surprised by the result of a programming task - the programmer left out something that's "obvious". There's no way to eliminate surprises entirely - and agile projects are tailored to make it easy to correct mistakes - but it's sand in the gears of the project if, too often, a programmer's happy "I'm done with the order-taking task!" results in quick disappointment.

It's better if conversations can be conversations about something, about concrete examples. When those concrete examples are executable, we call them "tests" - specifically, I call them "business-facing" tests.

A business-facing test has to be something a business expert can talk about. Most business experts - not all - will find tests written in Java or C# too painful. A safe and satisfactory choice is to use Ward Cunningham's Fit. In it, tests are written as a variety of HTML tables that look not too dissimilar from spreadsheet tables. Fit is ideal for tests that are data-centric, where each test does the same kind of thing to different kinds of data.

Some tests are processing-centric, where each test is composed of a different set of processing steps. Tables are more awkward for that. I would augment the official version of Fit with my own StepFixture or Rick Mugridge's forthcoming DoFixture. Such fixtures make processing-centric tasks more compact.

More important than the format of the tests is how they're created: collaboratively. The conversation begins with the business expert describing a new feature. This is often done in general terms, so it's important to train the team to say, "Can you give me an example of that?" whenever there's a hint of vagueness. Those concrete examples will help the programmers understand, and they may well also make the business expert suddenly call to mind previously overlooked business rules.

Those examples turn into business-facing tests, but I think it's important that they not start that way. I don't want to see people huddled around a screen, editing tables. It's too easy to get distracted by the tools and by making things tidy. I want to see people in front of a white board, scribbling examples there. Those examples can later be put into files.

What's the role of the tester in all this? One part, probably, is clerical. Guess who gets to turn scribbling into tables? But the more important parts are as translator and idea generator.

Experts are characteristically bad at explaining why they do what they do. Their knowledge is tacit, and it's hard for them to make it explicit. It's the tester's responsibility to draw them out. Fortunately, many testers are quick studies of a domain - they've had to be. It's also the testers' responsibility to think of important ideas the business experts and programmers might overlook. For example, both business experts and programmers tend to be focused on achieving return on investment, not on loss. So they concentrate more on what wonderful things a new feature could do, less on what it shouldn't do if people make mistakes (error handling) or intentionally misuse it (security). The tester should fill that gap, make sure the tests describe enough of the whole range of possible uses of the feature.

Quite likely, most of the tests will come after the programmer's started writing the feature. I'd want the initial set of tests to be enough for the programmer to estimate accurately enough and get started quickly. The tester can then produce additional tests in parallel. Always, any doubtful cases - "what should the program do here?" - will be reviewed by the business expert. As time goes on and the whole team learns the domain, fewer and fewer cases will be doubtful. The team will get better, in all ways, at making choices that make sense for the business.

The programmer will use the tests in something like the standard test-first way. When working on technology-facing tests, the programmer watches a new test fail, makes a small change to the code, watches the test now pass (and earlier ones continue to pass), cleans up if necessary, and repeats. The same thing will be done with business-facing tests. When I use Fit, I display its results in the browser. My cycle starts by looking at a table element that isn't right (isn't green, in the case of checks; or white, in the case of actions). I flip back to my programming environment and do what's required to make that element right. If the job is simple, I can do it directly. If it's too big a job to bite off at once, I use technology-facing tests to break it into smaller steps. When I think I've got it right, three keystrokes run the tests, take me back to the browser, and refresh the page so I can see my progress. (It's a measure of the importance of rapid feedback that those three keystrokes feel like an annoying slowdown.)

All this is an important shift. These tests are not, to the programmer, mainly about finding bugs. They're mainly about guiding and pacing the act of programming. The programmer evolves the code toward a form that satisfies the business expert. The tests make the evolution smooth and pleasant. Now, testers are not traditionally in the business of making programmer's lives pleasant, but the good tester on an agile team will strive to present the programmers with just the right sequence of tests to make programming smooth. It's too common for testers to overwhelm the programmers with a barrage of ideas - too many to keep track of.

I'm edging here into the most controversial thing about testing on an agile project: testing is much more explicitly a service role. The tester serves the business expert and, especially, the programmers. Many testers are, to say the least, uncomfortable with this role. They are used to seeing themselves as independent judges. That will effect hiring: with exceptions, I'm more interested in a helpful attitude, conversational skills, and a novelty-seeking personality than testing skills. It's easier to grow the latter than the former.

By that token I am (with an important exception) content with having no testers on the project. If the programmers and business expert can and will develop testing skills, there may be no need for a person wearing a hat that says "Tester" on it. But, in addition to seeing programmers grow toward testers, I'd be happy to see testers grow toward programmers. I'd be ecstatic if someone I hired as a tester began to contribute more and more code to the code base. Agile is about team capability: as long as the team gets the work done, I don't care who does it.

Now for various exceptions.

Automated business-facing tests are the engine of my agile project, but I'm leery of automating everything. In particular, I will absolutely resist implementing those tests through the GUI. They are written in business terms, not GUI terms. They are about business value, not about buttons. It makes no sense to translate business terms into button presses, then have the GUI translate button presses into calls into the business logic. Instead, the tests should access the business logic directly, bypassing the GUI. The prevalence of automated GUI tests is a historical accident: testers used to have to test through the GUI because the programmers were not motivated to give them any other way to test. But agile programmers are, so our tests don't have to go through the traditional nonsense of trying (often fruitlessly) to make our tests immune to GUI changes while not making them immune to finding bugs.

But what about the GUI? How is it tested? First, realize that testing below the GUI will keep business logic out of the GUI. If there's less code there, less can go wrong. Moreover, a thin GUI offers less opportunity for "ripple effects" than does code deeply buried in the business logic. Selected technology-facing tests (for example, using jsunit for javascript input checking) plus manual tests of both the functionality and usability of GUI changes should suffice. We shouldn't need full automation.

I would expect arguments about this. I'd also expect to win them, at least at the start. But if my simple practice really did let too many bugs past, I'd relent.

Another manual testing practice is derived from my passion for concreteness. Few people buy cars without test driving them. That's because actually using something is different than reading about it or talking about it. Both of those acts are at a level of remove that loses information and narrows perception. An automated test is at the same level. So I would introduce a form of semi-structured manual exploratory testing into the project.

I'd do that by piggybacking on my fondness for end-of-iteration rituals. Some agile projects have a ritual of demonstrating an iteration's results to anyone they can drag into the project room. After that, it'd be natural for people to pair off (team members with team members, team members with observers) to try the product out in ways interesting to the business expert. One pair might look at what the web site feels like over dialup lines; another might emulate a rushed and harried customer-service person who's making mistakes right and left. They're looking not just for bugs but also especially for new ideas about the product - changes that can be made in later iterations that would make it better even though it's not wrong now. They're throwing up potential work for the business expert to decide about.

I would make a final exception for types of testing that are both highly specialized and don't lend themselves to a test-first style. Some of these types of testing are security testing, performance testing, stress/load testing, configuration testing, and usability testing. I don't now see any reason for these to be done differently on agile projects.

Further reading

For test-driven design, I'd start with Kent Beck's Test-Driven Development: By Example and also read either David Astels' Test-Driven Development: A Practical Guide or Hunt and Thomas's Pragmatic Unit Testing. See also testdriven.com.

The best web site for exploratory testing is James Bach's. Kaner, Bach, and Pettichord's Lessons Learned in Software Testing is the closest thing to a book-length treatment. It's not very close, but it has a wealth of complementary material.

Fit is explained in Mugridge and Cunningham's forthcoming Getting Fit for Software.