Exploration Through Example

This is a retitled version of my SPA 2007 talk, “Monasticism for the Married”. It’s an encouragement to think of things as bundles of actions, framed by some alarm about the state of Agile.

Clearly reason was the goal here, and with Mark and Grace calmly looking on, it struck me just how good men are at agreeing exactly what “reason” is, how it should be pursued, and at what cost achieved.

—Thomas H. Cook, The Cloud of Unknowing

I’m intensely aware that at this time tomorrow, you’ll be hearing from Sir Tony Hoare, one of my heroes when I first got into this field. When I noticed him on the programme, I reread his 1980 Turing Award lecture, which contains this famous quote:

I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.

When I was young I wanted nothing more than to pursue that first way, but looking back I see that I haven’t. In such a case, the only honorable thing to do is to blame someone else:

I first blame my wife. As a surprising number of you know, she’s a professor of Farm Animal Internal Medicine, where she makes deficient bodies better. What you may not know is that your body is a gross kludge. It’s not so good at encapsulation, and the notion of an organ “doing one thing and doing it well” is a sick joke. The body is the very epitome of something so complicated that there are no obvious deficiencies.

Nevertheless, people like her intervene in that complicated mess, and they do manage to do it well enough. She gave me hope that we could deal with complexity.

The second person is Richard Stallman. Many years ago, I was the maintainer of the GNU Emacs editor on an obscure brand of computer. Doing that, I came to know the file unexec.c. The way Emacs was built was to first compile the C core, then have it suck up a whole bunch of compiled Lisp files, then have it dump its running image into an executable file. That way, starting Emacs wouldn’t make you wait for those Lisp files to be loaded. unexec.c was the file that made the executable. It had to be complicated because it had to work on different versions of Unix (and VMS, I think), many of which had different executable file formats and even different ways of getting to the process image. What was fascinating was that unexec.c kept getting better and better, easier and easier to work with. It was the first example I’d seen of coders holding back entropy. It could be done.

Nowadays I’m one of “those Agile people.” While I’ve fallen short of my earlier intentions, I like to think I haven’t chosen Hoare’s second approach. What I want to do now is create the software and then improve the deficiencies as they come to our attention. (There are cases, I note, where it’s not safe to have software with deficiencies; Agile isn’t for everything.)

Now I’ll shift course for a bit. People who want to pursue a life to its utmost—to exercise, as the ancient Greeks wanted, “vital powers in a life affording them scope”—have always had problems. Other people don’t care about what they care about, so they’re constantly tempted to pursue another life. A traditional response to this temptation is to retreat into a community of like-minded individuals—moving away from temptation.

I believe that Agile projects have some of that character. That’s why the title of the talk is “monasticism for the married”. Having everyone together in the same room reinforces the pursuit of whatever it is that Agile projects pursue (more on that later).

There’s a problem with the monastic approach, though. Sometimes people don’t leave you alone. Monasteries, for example, often had money or things of value. Bad people would come and take them away. As a consequence, monasteries were often surrounded with defensive structures. They also provided services to the outside world, such as prayers for the souls of departed relatives, a place for the nobility to stash inconveniently undeparted excess relatives, and trade goods such as excellent beer.

Agile projects are like that, too. They’re not so much defensively structured (though the chattiness, odd behavior, and clear group dynamics in the team room can seem like a wall), but they do have a semi-explicit deal: “Leave us alone to build software our way. In return, we’ll give you frequent releases of business value.”

What is “our way”? What is the pursuit? I believe an important one is to create what Richard P. Gabriel refers to as habitable software in his book Patterns of Software. (See also Stewart Brand’s How Buildings Learn.) Since I changed this speech several times after I left home for here, I don’t know Gabriel’s definition of habitability, but here’s what I consider habitable about a typical house:

• You can find things. Generally, things you look for often are really easy to find, but things you use less often are still findable.
• More generally, activities you perform often are easy (cooking, getting ready in the morning, etc.).
• They reward tinkering. You can do small things that make your life just a bit better.
• Expansion is something you’ll do willingly. Not many people are wild about remodeling their house, but nevertheless it’s easy enough to do that it’s affordable. And we desire it enough that we’ll pay for it to be done.

I think those points are also true of habitable software. Notice, by the way, that the actions in those sentences are more important than the things being acted on: “find,” “perform,” “tinkering,” and “expand” versus, well, a word as vague as “things”. More on that later, too.

There are two related temptations that distract us from habitability: the urge to abstract, and a quest for simple foundations. Those two things are seen as more virtuous than is the more detailed superstructure built on top of the abstract foundations. I want to claim that belief is a habit. It’s not something that we thought through for software: we just assume it. (Now, I don’t deny that abstraction and simplicity and such are often useful tools; I’m just saying that we’re too apt to make them a fetish.)

I blame all this on Plato. In a stunning example of the power of metaphor, he’s controlled a lot of our thought for two thousand years. His metaphor is one of a cave. We—all of us humans—are like prisoners sitting, backs to a wall, in a cave. Behind us, a light source projects shadows on the wall of the cave. Because we can’t see behind us, we see only the shadows. And so most of us believe that the shadows are reality, but the truly wise person—the philosopher-king—knows that the real reality is somewhere else, somewhere purer, and bends his intellect toward its discovery and appreciation.

A distant descendant of that Platonic myth was my firm statement, around 1985, that it’s the design that’s the real thing. The code is, properly, merely a consequence. So people ought to stop concentrating on code so much, I said, and think more about design.

I think that myth is reinforced in us because software development started as the handmaiden of physics and mathematics. We want to grow up to do what they do, and what they do is simplify and abstract. For example, the bewildering number of subatomic particles discovered especially by postwar physics was tamed by the quark theory. What had been a confusing zoo of particles turned out to be the consequence of simple arrangements of a much smaller number of particles. That was a great achievement.

But just because Murray Gell-Mann and others succeeded with quarks doesn’t necessarily mean that the same approach must succeed in other fields. For example, the American Pragmatists (Pierce, William James, Dewey, Rorty, and others) argue that it’s failed in the very field Plato more-or-less started: philosophy. Rorty says, in the introduction to his Consequences of Pragmatism:

People have, oddly enough, found something interesting to say about the essence of Force and the definition of “number.” They might have found something interesting to say about the essence of Truth. But in fact they haven’t.

So is software more like physics or more like philosophy? I’ve no firm answer, but one of my favorite stories might give a hint. It’s Ward Cunningham’s story of Advancers. <Here, I’ll refer you to my previous telling of that story, noting that my interpretation has changed somewhat.>

How do Advancers fit into the Platonic universe? Not so well. First of all, there was no design of advancers: they arose out of a mechanical translation of code that had only the most indirect connection to the world outside the code. The “motive force” behind them was not an abstract Idea; it was Ward and others poking at the code. The story is more one of people intervening in code than one of ideas.

Worse yet for Plato, it seems that such a powerful idea—something that seems so tuned to the world of taxes—ought to correspond to something in the world. But we’ve no evidence that it does. Even if tax experts do have an idea we could map onto Advancers, that idea played no role in Ward’s story.

If you want to connect Advancers to the outside world, the connection is action-to-action, not thing-to-thing. People intervene in the world of financial instruments; that provokes interventions by programmers in the world of code; the effort to make that code habitable leads to a nice little class.

If it makes you more comfortable, this kind of notion isn’t unknown in respectable science. One question in the philosophy of science is when we’re justified in thinking of something as real. If I remember correctly, some people justified the Copernican theory by saying it was just a calculational convenience: you could use it to get to the desired accuracy with fewer epicycles than the Ptolemaic theory required. But have no fear: the Copernican system isn’t real. The earth is really in the center.

The philosopher Ian Hacking was once talking to some people looking for free quarks (quarks not bound together with other quarks within a particle) using a variant of Millikin’s oil drop experiment for measuring the charge of the electron. The experiment involved suspending a niobium sphere in an electric field. Hacking asked how they moved it. They answered that if they needed the sphere to be more positive, they sprayed it with positrons. If they needed it to be more negative, they sprayed it with electrons.

Hacking had an epiphany: if you can spray them, then they are real. That is, if you can consistently use some property of electrons (their charge and mobility, in this case) to cause some useful effect in an experiment that otherwise has nothing at all to do with electrons, you might as well consider them real. Hacking puts it this way in his Representing and Intervening:

The best kinds of evidence for the reality of a postulated or inferred entity is that […] we can set out, from scratch, to build machines that will work fairly reliably, taking advantage of [it].

I want to apply that “interventionist” way of thinking to software, like so:

We intervene in existing code to support others
intervening in the world.

Classes, designs, words, etc. have meaning
if and only if
they have consistent-enough causal properties
that we use when intervening
in existing code
and coordinating our actions.

So what would that mean when thinking about our jobs? Consider Eric Evans’s Domain-Driven Design. He talks about a ubiquitous language that’s used by both the people working on the code and by people working on the business. It’s easy to slip into the habit of thinking that those words mean the same thing to both parties. But they needn’t. In Image and Logic, Peter Galison studied (among other things) how experimentalists and theorists in particle physics coordinated their actions using words and images. He likened it to speaking in a pidgin: an artificial language, used in trading zones between cultures, to coordinate specific activities. As long as the words work, it doesn’t matter who means what by them.

As an example of an odd use of language, consider the diagnostic language my wife uses. <See my earlier writeup on tacit knowledge; here I am again taking a different slant on the same material.> The words “bright” and “dull” coordinate actions. For example, a clinician who sees “bright” in a case report will evaluate the report differently than if she saw “dull.” But is there a real thing called “brightness”? I suppose Hacking would say yes: it seems to have consistent causal properties. But does it really matter? It’s not so much that “bright” represents something that’s true of some cows and not of others. It’s really just a token that appears in conversations that make use of its properties.

We have our own such words. Consider “duplication”, a code smell. If we were less hung up on reality, we’d spend less time writing computer programs that can detect duplication according to some definition and more time apprenticing to those people who’ve an uncanny ability to detect, declare, and use subtle duplication.

Both “Advancer” and “duplication” can be thought of as nothing more than mnemonic devices to help programmers remember where and how to intervene in the code.

It’s also reasonable for things other than words to coordinate activity. Once I was pairing with Jeremy Stell-Smith. He was driving, and I was navigating, when I suddenly got a strong feeling that I didn’t have intellectual control over the code, at least not nearly the degree of control I used to require. (I used to say that I would think and think and think about the design until I couldn’t stand it any more, then think about it again. Finally I’d be ready, and the code would just pour out of me.) I also wasn’t sure if Jeremy had intellectual control either, but that didn’t matter because I believed that the combination of Jeremy, me, and the tests did have intellectual control. Or, more precisely, that we collectively could move forward safely.

In his Cognition in the Wild, Edwin Hutchins writes about this kind of distributed cognition. He studied navigation, including the navigation of a naval vessel. He decided that it made sense to think of the tools sailors use as “stored thought.” For example, certain maps are devices by which calculations can be converted into drawing straight lines. He also decided that the sensible unit of analysis was the combination of the different people and their tools. Treating people in isolation wasn’t as useful.

So I see software development as a mish-mosh of people, code, teams, and tools, all made habitable so that certain interventions in the code are easy and pleasant. Lots of people will find the messiness distasteful, but I think a lot of people won’t. Agile teams, in particular, are nicely tuned to resonate on this frequency. If we took these kinds of things more seriously, we’d have a glorious time.

Except that I don’t think we’ll be allowed.

You see, monasteries got attacked all the time. A quick little Google search found some area where three different monasteries were raided in eight years. One of them was burned to the ground, thirty years after the last time it had been burned. This was in Ireland, a thousand years after the introduction of Christianity. I bet that the raiders were Christians, but they apparently had no problem putting their immortal souls in danger for… what? In at least one case, apparently nothing more than provisions.

Here’s a surprise for you: people often don’t really believe what they profess. It may be time for more of us to come to explicit grips with the clear fact that a lot of the people who fund Agile projects don’t much care about business value, ROI, and all that. As everyone but a University of Chicago economist must surely know, primate status games distort the pretty picture of profit maximization and shareholder value. I’ve seen too many Agile projects who’ve kept their side of the bargain—that do a fine job of delivering value—eaten alive by the business. The image is of a gazelle being eaten by a lion.

The problem, I’m afraid, is a disparity of power. Agile projects are isolated: they’re gazelles cut off from the herd. Except that it’s worse. There is no herd. Teams in good health don’t protect the vulnerable ones.

I think they should. I’ve given some thought to how they might, but I don’t think I’ve come up with anything useful. Let’s have a birds-of-a-feather session tonight. Perhaps you’ll think of something.

Thank you.