Consulting in Software Testing
by Michael A. Cusumano and Richard W. Selby
(The Free Press, 1995, ISBN 0-02-874048-3)
reviewed by Brian Marick on October 29, 1996
this book from fatbrain.com technical bookstore
From the introduction:
"This book provides a detailed portrait of how Microsoft
organizes, competes, develops new products, and tries to learn
and improve as an organization. We have focused on describing 'best
practices' from the most successful groups within the company.
Open access to Microsoft made it possible for us to write this
story. Although we are not the first authors to study Bill Gates
and Microsoft, which are in the news almost every day, most of
the many articles, academic case studies, and popular books have
focused on the history of the company and the personality of the
founder. What remains largely a mystery are Microsoft's crown
jewels: the key concepts that it has used to create technology,
shape the markets in which it competes, and manage the creative
energies of thousands of highly skilled technical people."
In Inside the Tornado, Geoffrey
Moore describes how pragmatic customers usually adopt a new
technology all at once, in the process anointing a market leader.
The market leader may not have the best technical solution, but
it has the one that's safest, most available, and easiest to
justify. (I am emphatically not sneering at the
pragmatists here - when competing products are all acceptably
competent, other issues must dominate.) Moore calls this sudden
surge of adoption "the tornado". It quickly sweeps away
the old and replaces it with the new, after which things settle
down once again and people consolidate their gains.
We're seeing such a tornado today in the marketplace of ideas.
For a long time, the thinking about how to develop software
was dominated by contract software, especially systems built for
the military. In that model, predictability and completeness were
highly valued. The contract was signed at the beginning of the
development process; therefore, it was important that the thing
being built be specified early and well. If not, how could you
bid? And how could you successfully sue for breach of contract?
Things changed. More and more often, there was no contract.
You built something, put it on store shelves, and people either
bought it or didn't. And the market grew more intense. No matter
how consistent and complete your planning, outside events
required reaction, sometimes wrenching change. Your careful
planning might be worse than useless if it made change harder.
Flexibility was more important than consistency.
But the contract development model hung on, as paradigms do.
Oh, there were changes around the margins, as people experimented
with and wrote about things like the spiral model or evolutionary
development. But I would argue that the mainstream of software
development thought was still centered on the underlying
assumptions of contract development. For example, development and
maintenance were treated as separate worlds - with most of the
writing being about the world of new product development, despite
the fact that most people were working on release 2.0 or later.
In contractual software, it makes sense to think of development
and maintenance as separate tasks, possibly done by separate
organizations. In commercial software, development is
But now, in large part because of Microsoft's phenomenal
success and their relatively recent decision to publicize the way
they develop, the floodgates have opened. The contractual model
is being displaced by a shrinkwrap model, as witnessed by the
succession of books like Maguire's Debugging the Development
Process (1994), McCarthy's Dynamics of Software
Development (1995), and (in parts) McConnell's Rapid Development (1996),
all written by people associated with Microsoft. These are not
just books by Microsoft people, they're books with a distinctly
different, non-contractual view of the world. For example,
Maguire says, "The typical lead [member of his target
audience] is usually embarking on the development of version 4.21
of an application...". Development is maintenance.
So where does Microsoft Secrets fit in? It's not
really appropriate for the general reader interested in an
overview of how Microsoft does things. It's too long and goes
into too much detail about many topics. It lacks narrative flow
and the sense of Microsoft that you get by talking with people
who work there. (And, it should go without saying, the book doesn't
actually reveal any secrets.)
The specialist reader wants detailed information to help with
particular tasks. An excellent book provides both details and
underlying principles. Details, because the reader doesn't have
time to invent them; principles, because there will always be
details missing, and the reader needs to create them correctly.
It should also warn of what problems are likely to occur. By this
criterion, Microsoft Secrets ranges from good to
inadequate, depending on the task.
In order to help the specialist, I'll do the following in the
- For different product development tasks, I'll sketch
Microsoft's approach as explained in the book.
- I'll say whether the explanation is adequate.
- I'll suggest other works that better explain the approach
or similar approaches. You can, as always, click on the
task name to find all the reviewed works that address it
(perhaps with completely different approaches).
When using the book to research a particular task, be aware
that some tasks are discussed in several places. Use the index.
Most likely, you will want to read all of both chapters 4 and 5 (at
least) if you read any of either of them. That will give you some
sense of the interconnections between tasks.
Click on the task name to see other works that address the
- The most important architectural idea is the division of
products into two layers: a flexible underlying engine
and features which are, ideally, independent. Because a
new release is composed mostly of a set of new or changed
features, feature independence allows more people to work
without tripping over each other. It also promotes the
development of strong teams. There is also a strong
emphasis on APIs (defined function call interfaces, in
this case). Other than API documentation, there is little
architectural documentation. People learn the
architecture by reading the code and talking with other
- The discussion of "engine architectures" lacks
detail, but I don't know of a better source. Writing Solid Code,
by Steve Maguire, has some discussion of designing APIs
that are less likely to be misused.
- Build management
- Different people in a project work on distinct copies of
the source code at the same time, so it is easy to create
incompatible changes. The longer it takes to discover
problems and reconcile the changes, the greater the
expense. Microsoft solves the problem by requiring daily
builds. The process, in short, is for a developer to make
changes to a private copy, test the changes, incorporate
changes other developers made to the master copy since
the private copy was made (typically no more than a few
days ago), rebuild a private executable, test it (looking
for interactions with recent updates to the master copy),
then add the changes to the master copy (again resolving
differences made by other developers since the previous
merge). A designated developer rebuilds the executable
from the master copy every day and runs a series of
automated tests against it. This process is a good
example of trading off short-term pain for long-term gain.
It's also a good example of the reinforcing feedback
loops described in Debugging
the Development Process, by Steve Maguire.
- The level of detail is good.
- In any project, a major problem is what features (or
requirements) to leave out. The book discusses a process
called "activity based planning". In it, users
are studied to discover what their major activities are (such
as "preparing a budget using information from
department heads") and what steps they take when
performing each activity. Proposed features are judged by
how well they support key activities. The concentration
on understanding complete activities also helps reduce
the chance one will be "90% supported", which
may be effectively no support at all.
- The amount of detail is inadequate, but I don't know of a
better source. Crossing
the Chasm, by Geoffrey Moore, discusses a
similar process of user characterisation with about the
same level of detail, but adds a useful example.
- Making the
- Microsoft uses the following information when deciding
whether the product is ready to ship: a checklist of
feature completion; trends in number of bugs found,
active, fixed, and verified; number of unfixed bugs and
their severity; status of testing against plan; changes
in bug severity distribution; amount of regression
testing done; amount of recent change to the source code;
and "gut feel".
- The amount of detail is good.
- Developers make their own schedule estimates (which often
produces an aggressive schedule that they're committed to).
Tasks are broken down into fine granularity subtasks (half
days to a few days) to encourage people to think through
the problem before committing to an estimate. An overall
goal is to break large projects into small autonomous
teams, because tight teams are more efficient.
- The amount of detail is inadequate. Debugging the
Development Process, by Steve Maguire, has a
good discussion of teams, but even less about estimating.
by Steve McConnell, has a chapter on estimation and one
- Project planning
- Large projects that produce new versions of desktop
applications (e.g., Excel) are divided into multiple
phases, each lasting a few months, each producing a
"nearly shippable" product, each adding a wave
of new features. Each phase has some buffer time to
accommodate the unexpected. Features that don't make the
milestone may be cut or pushed into the next milestone.
The frequent milestones force early visibility of
problems, as does the emphasis on completing features
before moving on (bugs are fixed continuously, not put
off to the end of the project). The entire project has a
stabilization phase at the end. There is no separate
product maintenance. Development is preceded by a
planning phase that produces a "vision document"
for the release and also outlines of specification
documents (descriptions of new features). Resources (calendar
time and people) are fixed at the beginning of the
project to ensure effort is focused on key features
rather than frills.
- The amount of detail is good, but the discussion does not
hang together well.
the testing project
- Microsoft emphasizes testing features in parallel with
their development, often pairing testers with developers.
In that case, developers give private releases of their
code to "testing buddies" so that bugs can be
found before they contaminate the master source.
Microsoft claims a 1-1 ratio of developers to testers.
The book gives brief descriptions of the type of testing
- The amount of detail is inadequate, but I don't know of a
- Usability testing
- Microsoft's version is fairly conventional. It's briefly
described. They also use data from phone support in an
unusually thorough way. The description of that is also
brief, but I know of no other source.
- User interface
- Developers and program managers (who are links between
development and marketing) cooperate in the development
of the user interfaces of features. Program managers
often do mockups or prototypes in, for example, Visual
Basic. The specification of what a feature does is given
in outline form to accommodate change and invention.
Usability testing is used to refine the interface.
- The amount of detail is inadequate, but I don't know of a
[an error occurred while processing this directive]
- Organizing and Managing the Company
Find "Smart" People Who Know the Technology
and the Business
- Managing Creative People and Technical Skills
Organize Small Teams of Overlapping Functional
- Competing with Products and Standards
Pioneer and Orchestrate Evolving Mass Markets
- Defining Products and Development Processes
Focus Creativity by Evolving Features and "Fixing"
- Developing and Shipping Products
Do Everything in Parallel with Frequent
- Building a Learning Organization
Improve Through Continuous Self-Critiquing, Feedback,
- Attack the Future!