Classic Testing Mistakes - Index and Discussion

Testing Foundations
Consulting in Software Testing
Brian Marick

An Index of Some Classic Testing Mistakes, With Discussion

For a checklist that's more suitable for printing, see either the PDF paper or the HTML version.

The role of testing

Thinking the testing team is responsible for assuring quality.	Discussion
Thinking that the purpose of testing is to find bugs.
Not finding the important bugs.
Not reporting usability problems.	Discussion
No focus on an estimate of quality (and on the quality of that estimate).
Reporting bug data without putting it into context.
Starting testing too late (bug detection, not bug reduction)	Discussion

Planning the complete testing effort

A testing effort biased toward functional testing.	Discussion
Underemphasizing configuration testing.
Putting stress and load testing off to the last minute.
Not testing the documentation
Not testing installation procedures.
An overreliance on beta testing.
Finishing one testing task before moving on to the next.
Failing to correctly identify risky areas.	Discussion
Sticking stubbornly to the test plan.

Personnel issues

Using testing as a transitional job for new programmers.
Recruiting testers from the ranks of failed programmers.
Testers are not domain experts.
Not seeking candidates from the customer service staff or technical writing staff.
Insisting that testers be able to program.
A testing team that lacks diversity.
A physical separation between developers and testers.
Believing that programmers can’t test their own code.
Programmers are neither trained nor motivated to test.

The tester at work

Paying more attention to running tests than to designing them.
Unreviewed test designs.
Being too specific about test inputs and procedures.	Discussion
Not noticing and exploring "irrelevant" oddities.
Checking that the product does what it’s supposed to do, but not that it doesn’t do what it isn’t supposed to do.
Test suites that are understandable only by their owners.
Testing only through the user-visible interface.	Discussion
Poor bug reporting.	Discussion
Adding only regression tests when bugs are found.
Failing to take notes for the next testing effort.

Test automation

Attempting to automate all tests.
Expecting to rerun manual tests.
Using GUI capture/replay tools to reduce test creation cost.
Expecting regression tests to find a high proportion of new bugs.

Code coverage

Embracing code coverage with the devotion that only simple numbers can inspire.
Removing tests from a regression test suite just because they don’t add coverage.
Using coverage as a performance goal for testers.
Abandoning coverage entirely.

[an error occurred while processing this directive]