Testing Computer Software

by Cem Kaner, Jack Falk, and Hung Quoc Nguyen
(Van Nostrand Reinhold, 1993, ISBN 0-442-01361-2)
reviewed by Brian Marick on August 23, 1996
Buy this book from fatbrain.com technical bookstore

From the preface:
"Testing Computer software provides a realistic, pragmatic introduction to testing consumer and business software under normal business conditions. We've tested software and managed testers for well known, fast-paced Silicon Valley software publishers. We wrote this book as a training and survival guide for our staffs."

This is, simply, the best book for the product tester or the testing lead. The reason for that is exemplified by chapter 5, "Reporting and Analyzing Bugs". I don't know of any other testing book that devotes a full chapter to this essential topic. Problem reports are the main way testers have an effect on the product, yet - save for this book - you'd get the impression that the sum total of human knowledge on this topic is captured by the single sentence "Make sure bug reports are repeatable by someone else." In contrast, this chapter opens with "The point of writing Problem Reports is to get bugs fixed." The authors of course emphasize reproducibility. But they also warn you how easy it is to antagonize programmers to the point that bugs don't get fixed. They describe the the information programmers need to fix bugs. And they help you present problem reports so that important ones don't get shuffled into the "fix it if we end up with extra time" pile.

There are people in the world who have technical knowledge but are somehow nevertheless ineffective at their jobs. I don't know how to characterize what they lack. Some of it is people knowledge. Some of it is a sense for priorities. Some of it is a feel for how things fit together. Whatever "it" is, this book was written by people who have it, and any tester who wants it could benefit by reading Testing Computer Software.

And then there are people who are ineffective at their jobs because they lack technical knowledge, which in this context mainly means the ability to design good tests. Testing books traditionally discuss test design from the wrong perspective. They treat it as a systematic exploration of certain aspects of the product, somewhat assisted by a nebulous process called "error guessing". In fact, error guessing - the use of past experience and an understanding of the frailties of human developers - makes the difference between a systematic process that finds tons of bugs and one that systematically accomplishes little.

Testing Computer Software is still somewhat weighed down by the traditional approach. The chapter on test design has the conventional text about equivalence classes, boundaries, and the like. There are nuggets on error guessing scattered throughout the book, most notably in an appendix containing some 400 errors, but there is no systematic treatment. I may be asking for too much here - I'm certainly asking for more than I provided in my own book. Nevertheless, I've heard too many people dismiss chapter 8, Testing Printers (and Other Devices), as "just about testing printers on PCs". It's actually a fascinating case study that, upon reflection, illustrates a number of general points, including how error guessing can be used effectively. But the authors don't make this evident.

Bottom line: in 1981, my first post-college boss handed me a copy of Glenford Myers's The Art of Software Testing and told me I had three months to test the HFPM (don't ask). Myers's book helped. But if I could go back in time, I'd give myself Testing Computer Software and tell myself to read it at least three times at yearly intervals.

This book will help you with the following tasks

Click on the task name to see other works that address the same task.

Automating tests

Covered from an appropriately skeptical viewpoint. The emphasis is on deciding whether automated testing is a good idea.

Configuration testing

Read between the lines of the chapter on testing printers and you'll find a good chapter on configuration testing.

Designing feature tests

A conventional treatment of the subject.

Designing a problem tracking system

A good description of what a problem tracking system should do for you, what data it should keep, and how to use it.

Localization testing

The chapter on this topic contains a checklist of localization pitfalls to check.

Reporting bugs

The best explanation of the topic.

Testing user documentation

A chapter covers both using user documentation to help you test the product and also how the tester can help improve the documentation. A good description of the production cycle of a commercial (shrinkwrap) software manual.

Planning a testing project

Takes the view that "a test plan is a valuable tool to the extent that it helps you manage your testing project and find bugs. Beyond that, it is a diversion of resources." A good description of what should be in a test plan and how to evolve it over time.

Notes on using this book

"Negotiating Testing Resources: A Collaborative Approach", by Cem Kaner, addresses the universal situation of having more testing tasks than time to do them in. It describes how to agree on a schedule. It gives more detail than the corresponding sections of the book.

This book's table of contents

Section 1 - Fundamentals

1. An example test series
2. The objectives and limits of testing
3. Test types and their place in the software development process
4. Software errors
5. Reporting and analyzing bugs

Section 2 - Specific Testing Skills

6. The problem tracking system
7. Test case design
8. Testing printers (and other devices)
9. Localization testing
10. Testing user manuals
11. Testing tools
12. Test planning and test documentation

Section 3 - Managing Testing Projects and Groups

13. Tying it together
14. Legal consequences of defective software
15. Managing a testing group
Appendix: common software errors