|
Testing Foundations
|
Reviews Organized by Task
This page contains a list of task names and brief descriptions. Links under each task name take you to relevant reviews. The index that follows is arranged roughly according to larger project activities. Beware that not every task falls neatly into one category.
Code coverage measures how thoroughly tests exercise code. It is typically used either as an estimate of how good testing is, or to point to undertested areas of the product.
The Craft of Software Testing, by Brian Marick, describes the use and dangers of different types of code coverage.
The architectural design describes the underlying structure of the entire software product. It contains those facts and interfaces individual developers must know so that their code works correctly within the product. It does not describe the user interface. It may or may not describe interfaces to other products, hardware or software.
Microsoft Secrets, by Cusumano and Selby, describes an architecture whose goal is to allow features to be added or modified independently of each other.
Architectural design means creating interfaces. Writing Solid Code, by Steve Maguire, has a good discussion of designing interfaces that are less likely to be misused.
Crossing the Chasm and Inside the Tornado, by Geoffrey Moore, discuss how the focus of the product changes over its life cycle. Ideally, the architectural design would support those changes. Note that the books are mainly about product marketing, so the information about design is scattered thinly throughout them.
Rapid Development, by Steve McConnell, has a brief and conventional chapter on Designing for Change.
Automated tests execute the product and report on failures without human intervention. They are most often used to look for new bugs after code changes.
Testing Computer Software, by Kaner, Falk, and Nguyen covers the subject from an appropriately skeptical viewpoint. The emphasis is on deciding whether automated testing is a good idea.
The Craft of Software Testing, by Brian Marick, is not as skeptical as Kaner et al, but shares their concern with the maintainability of automated tests. Marick avoids external test scaffolding; instead, he advocates embedding test support code in the product itself. That reduces maintenance cost and increases the chance maintenance will be done.
Writing Solid Code, by Steve Maguire, has a better discussion of internal test support code than Marick does. Ironically, it's never described as test support code, but rather as debugging and development support.
This is the task of assembling software components into a version of the product. "Software components" may be only executable code, or it may be everything shipped on the CD (online documentation, help files, and so on).
Microsoft Secrets, by Cusumano and Selby, describes Microsoft's "synch and stabilize" approach. It uses daily builds to avoid serious problems when many people change the source base at once.
Rapid Development, by Steve McConnell, has a good description of using daily builds to reduce schedule risk. The technique is nearly the same as that described in Microsoft Secrets, but neither discussion is a superset of the other.
Writing Solid Code, by Steve Maguire, has a good discussion of designing interfaces that are less likely to be misused. It also advocates adding debugging support code to the product. The discussion covers both assertions and more elaborate code.
Configuration testing is the process of varying a product's environment in a way that efficiently and effectively discovers bugs dependent on that environment.
Testing Computer Software, by Kaner, Falk, and Nguyen has a detailed (though dated) chapter on testing printers. Read between the lines, and you'll find a treatise on the process of configuration testing.
Writing Solid Code, by Steve Maguire, says code that fails should do so quickly and obviously. Code that does so is easier to debug.
Problem tracking systems record what bugs have been found and who is doing what about them.
Testing Computer Software, by Kaner, Falk, and Nguyen gives a good description of what a problem tracking system should do for you, what data it should keep, and how to use it.
Feature tests explore a particular feature (such as printing, or text justification, or image manipulation) in careful detail.
The Craft of Software Testing, by Marick, gives several feature testing techniques.
Task-based tests attempt to emulate a typical (or otherwise interesting) user task, in order to encounter the sorts of bugs a user might encounter. They often find usability problems and interaction problems among features.
Crossing the Chasm, by Geoffrey Moore, describes a process of target-customer characterization that can be used as a starting point for task-based tests.
"Practical Priorities in System Testing", by Nathan Petschenik, suggests using bug reports to help design task-based tests.
Detailed design captures the thinking an individual developer would do just before (or while) creating a body of code. The scope of detailed design varied widely among companies. There's usually a notion that an architectural design describes relationships between subsystems, and that the detailed design concerns itself with the internal structure of a subsystem.
Writing Solid Code, by Steve Maguire, has a good discussion of designing interfaces that are less likely to be misused. The discussion applies to both architectural and detailed design.
In practice, developers almost always do some testing of their code, but it's usually unplanning, haphazard, and considered part of coding or debugging.
The Craft of Software Testing, by Brian Marick, was written for both developers and independent testers, but the techniques it describes are best applied by developers.
Writing Solid Code, by Steve Maguire, has a weak discussion of test design. The discussion of test implementation is strong (though much of it is disguised as a discussion of debugging).
"Testing Made Palatable", by Marc Rettig, has modest content on the "how" of developer testing, but it's real strength is "why".
Requirements are descriptions of what customers desire. This task is one of discovering those desires, especially making implicit desires explicit, and clarifying the ambiguity that invariably surrounds them.
Exploring Requirements, by Gause and Weinberg, is the best book on this task.
Microsoft Secrets, by Cusumano and Selby, describes activity-based planning, in which common customer activities are used to weed out unimportant requirements and to ensure that key requirements are not overlooked.
Crossing the Chasm, by Geoffrey Moore, describes a process of target-customer characterization similar in results to the activity-based planning sketched in Microsoft Secrets, but oriented more toward products that either don't exist or are searching for an application niche.
Rapid Development, by Steve McConnell, has a brief discussion of techniques for eliciting requirements. There's a good chapter on removing requirements (requirements scrubbing, feature-set reduction), something poorly covered in other books.
This is the task of planning how information about the product should be presented to the user. As with all planning, a major constraint is time and other resources.
Crossing the Chasm and Inside the Tornado, by Geoffrey Moore, discuss how the psychological profiles of users change during a product's life cycle. Because the different types of users prefer different types of documentation, the makeup of the documentation set should change over the life cycle.
Rapid Development, by Steve McConnell, is mostly targeted toward software development, but the chapters on risk management, classic mistakes, motivation, teamwork, and team structure are general purpose. The discussion of lifecycle models is something documentation leads need to know for their planning.
Everyone knows how to have meetings. This topic is "how to have meetings that don't make people realize once again how much they hate meetings."
How to Make Meetings Work, by Doyle and Straus, is an essential book.
Debugging the Development Process, by Steve Maguire, recommends having fewer meetings by looking for less disruptive ways to accomplish the same ends. When meetings must be had, Maguire provides some useful tips for scheduling them and running them, as well as one central strategy: meetings should have explicit goals and lead to concrete decisions.
Exploring Requirements, by Gause and Weinberg, has two fairly conventional chapters on meetings and facilitation. It has a useful chapter on brainstorming.
These are techniques for examining "work products" to find flaws. "Review" is typically taken to refer to a variety of informal techniques, whereas "inspection" means a rigorous, well-defined process.
Software Inspection, by Gilb and Graham, gives a detailed treatment of inspections. It's best for people with a well-controlled, step-by-step software development process. It's good on using inspections in process improvement. The authors have eccentricities that may put some people off.
Exploring Requirements, by Gause and Weinberg, contains a brief summary of the technical review.
Rapid Development, by Steve McConnell, gives a brief description of Gilb-style formal inspections.
The Craft of Software Testing, by Brian Marick, describes the creation of documents called "test requirement checklists". It concentrates on their use in test design, but they can also be used for code or design inspections. The book also includes a separate code inspection checklist.
Testing Computer Software, by Kaner, Falk, and Nguyen has a checklist of localization pitfalls to check.
Localization means changing a product to make it useful in a particular country. In addition to translating the language, you must also be concerned with issues like numeric and currency formats. Localization testing searches for flaws in localization.
Testing Computer Software, by Kaner, Falk, and Nguyen has a checklist of localization pitfalls to check.
At some point, someone has to decide whether the product is ready to ship. These works describe the information used to make that decision.
Microsoft Secrets, by Cusumano and Selby, describes how Microsoft decides a product is ready to ship.
Rapid Development, by Steve McConnell, describes different lifecycle models. Following the right model can make the ship decision less traumatic.
This is the task of dealing with changes made late enough in the project that they jeopardize the product's stability. Examples: how do you cope with new features added late in the project? When and how do you remove features? How do you decide which bugs should be fixed?
Rapid Development, by Steve McConnell, describes lifecycle models that soften the impact of feature set changes. It also has a chapter on feature set reduction (as a way of saving development time).
Microsoft Secrets, by Cusumano and Selby, describes Microsoft's implementation of a variant of one of McConnell's lifecycle models (evolutionary delivery).
Debugging the Development Process, by Steve Maguire, defines the development lead's job as "ruthlessly eliminating any obstacles that keep the developers from the truly important work: improving the product". This is an essential book.
Microsoft Secrets, by Cusumano and Selby, describes how Microsoft emphasizes small, tight teams who make their own schedules.
Writing Solid Code, by Steve Maguire, is a good discussion of how developers can prevent or detect bugs before they escape to be found by testers or customers. Managers who can read C code and have developers reporting directly to them will find it a good read. Managers of managers will find it too detailed.
"Testing Made Palatable", by Marc Rettig, will help you in one of your more difficult jobs: convincing developers that they really do need to test their code, even though they almost certainly don't want to. It won't help you with the second part of that job, which is teaching them how to test.
Rapid Development, by Steve McConnell, has chapters on motivation and teamwork, but they are rather abstract, almost anthropological. It's the sort of discussion that makes people without a feel for the human touch worse managers, rather than better, but there are some good tidbits. The discussion of different team structures is both interesting and useful, if again a bit schematic.
Testing Computer Software, by Kaner, Falk, and Nguyen has a chapter on managing a testing group.
"Negotiating Testing Resources: A Collaborative Approach", by Cem Kaner, describes how to decide what to test when you don't have nearly enough time and people.
Microsoft Secrets, by Cusumano and Selby, describes how Microsoft emphasizes testing features in parallel with their development, often pairing testers with developers. In that case, developers give private releases of their code to "testing buddies" so that bugs can be found before they contaminate the master source. The book gives brief descriptions of the type of testing done.
"Practical Priorities in System Testing", by Nathan Petschenik, gives a fairly conventional approach (but one that is followed less often than it should be).
This means determining the project schedule: matching tasks to people, resources, and time. In practice, it often means determining the development schedule. Testing and documentation are expected to tag along. See also documentation planning and test planning.
Rapid Development, by Steve McConnell, has "must read" chapters on classic mistakes, risk management, and project lifecycle models. The section on estimation is primarily top-down, using the predicted size of the software to create person-month and then calendar time estimates. If you have estimation problems (and who doesn't?), the background discussion and some of the tidbits will help, but there's no magic solution here. The major benefit of the estimation chapters is that they provide some common sense (and data) about what's not possible.
Debugging the Development Process, by Steve Maguire, recommends milestone-based scheduling, in which the project is broken into small, coherent subprojects, each a couple of months long. In addition to surfacing cumulative schedule slips quickly, small milestones keep project enthusiasm high because a significant goal is always in sight.
Microsoft Secrets, by Cusumano and Selby, describes how Microsoft divides projects into roughly three month chunks, and strives to make each chunk an "almost shippable" product. Microsoft is also noteworthy for building explicit buffer time into schedules to handle the unexpected.
Project recovery is the task of dealing with a project in crisis, one that appears as if it will never finish.
Rapid Development, by Steve McConnell, has a valuable chapter on rescuing a project that appears as if it will never finish, much less finish on time. McConnell advocates a modest (but psychologically hard) approach of dropping a few features, increasing productivity as much as you can, and slipping the schedule as needed.
It's worth noting that McConnell describes Microsoft's NT and Visual C++ projects as "spending all their time doing ... project recovery" (based on his reading of Jim McCarthy's Dynamics of Software Development and Zachary's Showstopper). This chapter, then, provides an interesting commentary on those books, as well as on Microsoft Secrets, by Cusumano and Selby.
Testing Computer Software, by Kaner, Falk, and Nguyen has the best explanation of "how to report a bug so that it gets fixed".
See also Bob Binder's bibliography (at one point to be found at www.rbsc.com).
The Craft of Software Testing, by Brian Marick, describes one approach to testing object-oriented software.
There are two related tasks: using documentation to help test the product, and testing the documentation itself for accuracy and usefulness.
Testing Computer Software, by Kaner, Falk, and Nguyen. A chapter covers both using the documentation in product testing and checking (reviewing) the documentation. A good description of the production cycle for a manual in the shrinkwrap software industry.
Observe how real users use the software, and you may learn something.
Microsoft Secrets, by Cusumano and Selby, describes a conventional approach briefly.
Jakob Neilsen's useit.com is a good online resource. See especially his Alertbox column.
The user interface design describes what the user sees and how he or she interacts with the product. For products driven through the user interface, I consider the specification (the description of what the product does) to be part of the user interface design.
Microsoft Secrets, by Cusumano and Selby, describes Microsoft's process for designing user interfaces and what particular features do. It's an iterative, collaborative design between developers and program managers (who are the interface between developers and marketing).
Crossing the Chasm and Inside the Tornado, by Geoffrey Moore, discuss the how the psychological profiles of users change during a product's life cycle. Different types of users have different requirements of the user interface, so the design is likely to evolve over time. Note that the books are mainly about product marketing, so the information about design is scattered thinly throughout them.