Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Mon, 28 Apr 2003

Imre Lakatos and persuasion

I was talking to Mark Miller last week. He's interested in capability security, I'm interested in context-driven testing and agile methods, but we're both interested in persuading people to change their minds. Mark quoted a common summary of Kuhn: "Science advances one dead scientist at a time." The old guard dies off, and the new guard takes their place.

Quite true, in the main, but some in the old guard have to change their minds for the new guard to get started. Some people have to switch from being Newtonians to being relativists, or from being Ptolemaists to being Copernicans. Why and how does that happen?

The philosopher Imre Lakatos's methodology of scientific research programmes talks about why and how scientists should change their minds. He extracts rules for rationality from a somewhat idealized history of great scientific advances. I'm going to do violence to his intentions by not caring (here) about rationality. Instead, I'm going to build off my reaction to Lakatos, which is "Those sure are nice rules. They capture a lot of what I mean when I say, 'Those people who call themselves the Fooists are really going places, and I want to go along." If other software people respond to the same cues as I do, then we who want people to go along with us might benefit in two ways from careful study of Lakatos's rules:

  • We'll explain what we're doing in a more appealing way.
  • We'll think about what we're doing in a way tuned for success. We'll make better choices.

But note well that I'm not claiming a complete solution to the problem of persuasion.

So here goes. For skimmability, I'll indent the explanation of Lakatos differently from some of my speculations about various software movements.

At the heart of any research programme is a "hard core of two, three, four or maximum five postulates. Consider Newton's theory: its hard core is made up of three laws of dynamics plus his law of gravitation." (Motterlini 1999, p. 103)

Agile software development has four postulates. I might be right in saying that capability security's hard core is also concise. It seems to be a single diagram, the Granovetter Diagram.

Software postulates probably can't be as crisp as F=ma, but I think the Agile Manifesto postulates are actually quite good as a guide to behavior. If you have a problem in a project, you would first try to solve it by thinking about the people involved and their interactions, only secondarily thinking about people-independent processes and tools. If getting working software soon is in conflict with comprehensive documentation, you go with the working software.

I've been critical (as an insider) of the context-driven testing principles, but they actually stack up reasonably well. My objection has been that principles like "good software testing is a challenging intellectual process" or "people, working together, are the most important part of any project's context" are too easy to agree to. I still think that's a problem, but they are also - when taken literally - a good guide to behavior. The first says that if you're trying to solve a problem by reducing the amount of thinking testers have to do, you're solving it wrong. The second has much the same import as the Agile Manifesto's "individuals and interactions over processes and tools".

I'd recommend the context-driven principles be restated in more of an active voice. Make them less statements of truth and more explicit heuristics for behavior. And maybe knock off a few. (Though I'm skeptical of Lakatos's "at most five", I don't know if I'll ever be able to remember all seven.)

A set of postulates starts out inadequate. The common (Popperian) wisdom about science is that scientists posit hypotheses, specify refutable predictions that follow, and replace the hypotheses when those predictions fail. That's not in fact what happens. Counterexamples are shelved to be dealt with later. For example, Newton's theory did not correctly predict the observed movement of the moon, but he did not discard it. When Le Verrier discovered the motion of Mercury's perihelion was faster than predicted by Newton, people shrugged and waited for Einstein to explain it.

Research programmes can proceed despite their obvious falsity. Rutherford's model of the atom (mostly empty space, electrons orbiting a nucleus) violated Maxwell's equations, which were believed to be rock solid. They were certainly much more compelling than the new evidence Rutherford's model was intended to explain. But Rutherford's programme essentially said, "We'll figure out how to reconcile with Maxwell later." (The solution was quantized orbits - the "Bohr atom".)

How can we use this? I made a little splash in 1999 by attacking the popular V model for software development. I suggested requirements for a better model. I now think of those requirements as groping toward agile methods, particularly Scrum's management style and XP release planning. As is typical, my presentations were well received by those predisposed to agree with me, and thought scandalous by those not. Defenders of the V model tended to list problems that projects using my alternative might have. Rather than argue with them, I should have changed the subject (as Rutherford did): "Of course an early version of a model or methodology will have problems. So? Do you expect progress with no backward steps? Or can we talk about something useful? "

Similarly, people attempt to discount XP by saying "XP is crucially dependent on the myth of the on-site acceptance-test-writing customer". That's a valid criticism in the sense of being a statement of a limitation that XP should want to tackle someday. But it should have no more innovation-stopping power than saying, early last century, "Well, relativity has problems too". (Lakatos said of Einstein: "Einstein's theory inherits 99% of [the anomalies found with Newtonian mechanics], eliminates only 1%... As a matter of fact, Einstein's theory increases the number of anomalies; the sea of anomalies does not diminish, it only shifts." (ibid, pp. 100-101))

The preface to XP Explained does the right thing: "XP is designed to work with projects that can be built by teams of two to ten programmers, that aren't sharply constrained by the existing computing environment, and where a reasonable job of executing tests can be done in a fraction of a day." (p. xviii) Specifically eschew universal claims, and don't let people tempt you into claiming more than that you have a set of principles that work in some cases and that you're going to work hard to make them work in more.

A problem that context-driven testing has, I think, is that it's not sharply enough delineated. It clearly comes from the shrink-wrap product world, but it's hard to draw boundaries around it. We "contexters" are always making nods toward highly-regulated environments, talking about how we'd write oodles of documents if the context demanded it. I think that weakens our impact by lashing together the contexts where we provide novelty and innovation and ferment with the contexts where we don't say much (yet).

But surely factual evidence counts for something. Lakatos says it does, in two ways.

First: while "theories grow in a sea of anomalies, and counterexamples are merrily ignored (Motterlini, p. 99)," the same is not true of dramatic ("novel") confirmations. What convinced scientists of Newton's theory of gravitation? According to Lakatos, it was Edmund Halley's successful prediction (to within a minute) of the return date of the comet that now bears his name. What "tipped" scientific opinion toward Einstein's theory of general relativity? The famous experiment in which the bending of light was observed during a solar eclipse. (Interestingly, according to Overbye's Einstein in Love, the observational evidence was actually shaky.)

Sometimes critics of agility will concede that it could be appropriate for "completely chaotic, changing environments". The implication is that a properly managed project, with customers who knew their own minds and markets, could do better. But if you're hopeless anyway, perhaps agility can salvage something.

The temptation is to argue that agility is not so narrowly applicable, but I think that takes us off into unpersuasive meta-arguments about what methodology would be best in the abstract. Instead, rely on the fact that chaos exists, that people on projects view it as a problem, and that solving this problem serves as a novel confirmation of a surprising prediction.

Seek out what's unlikely and demonstrate success at it. Much credibility flows from that. That's part of the rhetorical power of origin myth projects.

Something that many project members believe unlikely is job satisfaction. Job satisfaction is thus both a novel and testable prediction. For that reason, I think it's good that agile advocates predict that agile methods lead to it. (See for example, Lisa Crispin's writing and how Josh Kerievsky makes Enjoyment one of the five values of Industrial XP.)

The second way that factual evidence counts is in the way proponents respond to it. Lakatos pictures the core postulates as being surrounded by a protective belt of auxiliary hypotheses that are used to handle telling counterexamples.

Newton provides an important example of the right kind of protective belt. He was informed that observations of the moon refuted his theory. To protect it, he invented a new theory of refraction that, together with his laws, did predict the moon's movement. (Sort of - it was never right, because the moon's center of mass isn't at the center of the sphere, which throws off the calculations.)

His theory of optics not only corrects refuting observations to make them match the theory. It is also a theory of its own, makes new predictions, and had some of the new predictions confirmed. Strictly, the observations that refuted Newton's theory of gravitation served to surprisingly confirm his theory of optics. He knew that there were refuting observations, but he didn't have the values, so he couldn't "work backwards" from them to a theory of optics that served no purpose other than to protect his hard core.

It's that "working backwards" - fitting the protection to the specifics - that distinguishes what Lakatos calls "ad hoc" protective hypotheses from the good kind. "[Some proposed counterexample] was never discussed before, but now you can account for this case too by introducing an ad hoc auxiliary hypothesis. Everything can be explained in this way, but there is never any prediction of a corroborated novel fact." (ibid, p. 101)

The existence of ad hoc protective belts shows why my attempts to take down the V model were pointless. For any criticism of the use of the V model in a specific project, one can always invent an ad hoc explanation. It needn't ever be the fault of the methodology. It can always be the fault of inadequate training, improperly motivated staff, a lack of management support, upheaval due to an acquisition... None of these explanations provide prescriptions about what to do and also make novel predictions.

In contrast, I don't think Scrum's approach to management problems is ad hoc. It makes a specific prediction: that if project management sees its role as facing outward from the team (clearing roadblocks from their path) rather than inward (telling them what to do) the team will perform perfectly fine. Moreover, Scrum addresses the criticism that "agile doesn't scale" with a specific mechanism (the " scrum of scrums"), and predicts that this mechanism alone suffices for teams of hundreds.

To be convincing, we must scrupulously avoid ad hoc, after-the-fact explanations of why something didn't work. Instead, we must either forthrightly admit that we don't know how to deal with problem X, or we need to make predictions much more specific than "get proper training".

And forget about flailing after things like the V model. While such attacks do serve to cement the solidarity of those already on your side, they are one of the less effective ways of convincing the unconvinced.

Research programmes, even ones as successful as Newton's, eventually degenerate. A programme "is degenerating if ... (1) it does not lead to stunning new predictions (at least occasionally...); (2) if all its bold predictions are falsified; and (3) if it does not grow in steps which follow the spirit of the programme." (ibid, p. 106) (This last seems to mean avoiding the ad hoc additions mentioned above. But I think there's also a less pin-down-able sense. It would not be in the spirit of an agile method to extend itself by adding more and more examples of written process documentation.)

I take this to mean that research programmes, like sharks, have to keep moving or die. It would not be enough for agile methods to confine themselves to small projects where there's rapid change, else they would drift out of the software development attention space (see commandment 4). The agile methods have to keep trying to push the boundaries of applicability. They need to report success in new types of projects, with a reasonable argument that the success is due to new practices or extensions of old practices.

P.S. The citations are from Lakatos's "Lectures on Scientific Method", in For and Against Method, Matteo Motterlini (ed.). I should be citing Lakatos's Philosophical Papers, Vol. 1: The Methodology of Scientific Research Programmes. But when I wrote a long email on this same topic, some years ago, I'd just happened to pick up Motterlini.

## Posted at 08:10 in category /misc [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo