Exploration Through Example

Update: some might want to skip the introduction and go straight to the guiding values, which are the point of the talk.

I’m going to begin with a revisionist interpretation of the Agile Manifesto.

The Manifesto stirred up quite a fuss when it was published. Programmers and other “worker bees” identified with it, felt that it had been written for them. True enough. But I’m going to point out that it was also a marketing document, a bargain proposed by a new style of team to the business world.

Here’s the shape of the proposed deal. The teams promised the business two things. First, they promised to respond to changes from the business. That’s a fancy way of saying they’d stop whining when the business changed requirements. It’s not that they were admitting that all those years of whining were some sort of unjustified passive-aggressive trick — as many in the business world more than suspected. It’s that the teams claimed they now had the skills and the technology to let them make software soft enough that they could quickly and correctly implement requirements without caring whether they’d been created last week or last year.

Next, the teams promised to care deeply about delivering working software. Again, the meaning of that is a little unclear. The teams were admitting that they knew the business expected the following trajectory from every last project they paid for:

In January, the team says it’ll be done in November. Then they, more or less, disappear from view. They pop up in September to alarm the business by admitting, in a noncommittal way, that there’s “some schedule risk”. In October — one month before the scheduled release date — they pop up again to admit the schedule will have to slip until, oh, just before Christmas. Since nobody in the U.S. gets anything done between Thanksgiving and Christmas, that December date proves unrealistic. Finally in January, system testing can start. According to the PERT chart, it’ll finish by the end of the month…

Except testing… finds bugs! Bad bugs. Show-stopper bugs. Fixing them has pushed the release into February. Well, March, actually. That’s when the product finally limps into release.

That slow-motion train wreck is what the business has come to expect. A long wait, with disappointment at the end. So promises of working software don’t mean a lot to the business. They don’t trust their team.

But the Agile Manifesto gives a different meaning to the phrase “working software”. What Agilists mean is that if the project starts in January, the business can — indeed, must — expect them to pop back up with working software no later than February. (I’m going to assume a one-month iteration, because it makes for prettier slides, although two weeks is much more common these days.)

Working software means software that the business could — if it wanted — put into actual use. The software does at least one thing — one thing chosen by the business — that has value.

Or, if the business doesn’t want to put the software out for use, they can still try it out — hands on, unscripted — to see if it is salable and convince themselves that the team is making progress. The team’s no longer asking for trust, they’re providing evidence. (Which of course is an excellent way to build trust.)

This process, the Manifesto promises, could proceed month after month after month, stopping when the business decided it had enough value or at any arbitrary date it chose.

That progression of steady increments of value seems pretty implausible. But — the Manifesto claims — it can now be done because teams now have the skills and technology to do it.

That’s one half of the bargain. To make the whole deal work, the business has to do some things in return. First, they have to talk to the development team. And “talk” actually means spoken conversation, preferably face to face, and surprisingly often. Actually, let me highlight that. The amount of communication required will surprise the business even after they think they’re prepared for it.

The second of the business’s obligations is implicit in some of the lines in the Manifesto. The unspoken message is don’t “help” us by telling us how to develop software. We — the team — realize that you — the business — have been scared because our work was so invisible. So you put in checkpoints (like requirements reviews and architectural design reviews) along the way to get early warning that things were going off track. But that didn’t work. In fact, in a perverse way, it made things worse.

The dirty little secret is that the process, the documents, always hindered the team’s ability to respond to changing requirements. Trying — and failing — to reduce one problem increased another. However, we now have the skills and technologies to deliver frequent releases of working software — which is better evidence of progress than documents that talk about what the software will be when we finally get around to producing it. So that was the deal. At first, not too many businesses took it. But now lots have, as witness the existence of this conference.

So the story of the manifesto is over, really. The time for marketing is past. Now what teams have to do is execute. Here, the news is not so good. A lot of teams execute poorly. Helping you avoid their fate is what this talk’s about.

First, though, you’ll notice my emphasis on teams. Teams are what I know. The kind of consulting I do involves embedding myself into teams and helping them out. While there’s lots to be said about making the enterprise safe for Agile, or for making the enterprise agile, or for doing other things involving the word “enterprise”, I’m not the person to say those things. The person to say them is David Anderson, who — fortunately for you — will be giving a keynote on Thursday, making our talks nice bookends for the conference.

Now, given what I’ve said before, you might expect that I’d be touting particular skills and particular technologies. I’m not going to do that, but I hope you sought them out in the tutorials, will keep seeking them out the next two days, and realize that your learning can’t stop with this conference. Skills and technologies are essential. But people know that. So, instead, what I’m going to talk about are guiding values.

Why am I going to spend your time on that topic? It’s because many of the problems I see are due to teams giving into temptation. Values are what keep us on the straight and narrow path in the face of temptation. Teams that have strong internalized values will stick to good Agile practices — and get good Agile results — while teams without guiding values will drift into the ditch.

I’m using “temptation” in a broad sense. For example, I include allowing yourself to be bullied as, most often, a giving into temptation: the temptation to be weak, to play it safe. For that reason, I’m going to name the first value courage. I’ll illustrate courage with a story. I heard it from Ken Schwaber as a description of the kind of person a Scrum Master should be. Once, there was a software team working in cubicles. They didn’t like working in cubicles, an entirely sensible attitude. They wanted an open workspace. That makes a lot of sense, given the amount of communication that goes on between the business and the team and among the team members. However, the furniture police forbade it. Company policy was cubicles, and company policy could not be changed for a single team.

The team went through the usual lengthy arguments, and — as usual — company policy prevailed. Until, one weekend, the Scrum Master disassembled the cubicles and created an open workspace. Come Monday, she announced that if the cubicles were restored, she would immediately resign. That’s courage. That’s a Scrum Master. The job of the Scrum Master is to move immovable objects for the good of the team.

Courage as a value was not present in the Agile Manifesto because that value wasn’t relevant to the sales job. The next value, working software, was present because it’s so important. As a guiding value, it says that if you don’t know what to do, if you think you’re going astray, always err on the side of getting some software doing something that someone can look at. I don’t quite have a story to illustrate that, but I do have a guideline from Kent Beck. As you probably know, people — and especially programmer types — are prone to long, long design discussions about what to do next, about what the appropriate approach should be. A great many of those discussions are wasted time. The fewer facts people have, the harder they argue, it seems. So Beck made up a guideline. Here it is: No design discussion should last more than 15 minutes without someone turning to a computer to do an experiment.

The assumption underlying this guideline is something like this: Humans with their experience augmented by watching — or writing — running code, are smarter than humans alone. Or, perhaps, humans + running code are smarter than humans + time. I’m convinced that books could be written about this worldview. (Actually, they have been — they just don’t mention software.) For now, though, I’ll stop with the guiding value: whenever you’re doing something that’s starting to the immediate making or observing of running code, find a way to irritate you, and that something does not involve add working software to the mix.

Now, there’s a word I just used that I want to go back to, “irritation”, because it’s a bridge to the next guiding value, ease.

Let me start my explanation with a dead philosopher. This [picture] is Martin Heidegger, a truly odious man, but someone who coined a useful distinction. Consider the act of hammering. There are two ways you could experience the hammer. It could be ready-to-hand. In that case, you don’t think about the hammer — you think about hammering. You’re not really conscious of the hammer at all. You’re conscious of the goal, the object of the task — in this case, moving the nail. On the other hand, the hammer could be present-to-hand, in which case you’re constantly aware of the hammer, typically because there’s something wrong with it, putting the goal at risk. Maybe the head’s loose, so it feels like it’ll fly off if you swing the hammer too hard. Because of that, you constantly have to attend to something that’s normally automatic (or ready-to-hand). You’ll always be feeling the hammer present in your hand.

Here’s another example: surgery. I’m told by a surgeon that, if you have a good surgical nurse, you just reach out, maybe vocalizing what you’re thinking, and the right instrument just appears in your hand. It’s ready-to-hand. You focus on the operation, not the tool.

Suppose, in contrast, the surgeon had to search for the right instrument in something like our junk drawer. Now the surgery is made less easy by the present-to-hand-ness of some peripheral activity.

Here’s an example of peripheral activities in software.

I needed to fix a particular bug. To reproduce it, I had to find a Windows Vista machine, but they were all allocated to other people, so I had to upgrade from Windows XP. To do that, I needed more memory, so I had to buy memory, but that meant I needed to get approval from my manager Frieda, but she was angry with me because of a little incident that was still fresh in her memory, so I had to find some way to, uh, even out her opinion of me by delighting her, perhaps with a present. I remembered she was knitting an old-style Dr. Who scarf using rare and exotic animal hair, which is how I ended up at the zoo, shaving a yak.

The phrase “Yak-shaving” has entered the software jargon. One online dictionary defines it as:

Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you’re working on.

This kind of thing happens in software all the time, and it kills productivity.

A characteristic of high-performing Agile teams seems to be how much they value ease of work. So, for example, someone working in some code that’s… awkward… will spend a little time making it easier for the next person. If the build keeps breaking, someone will do a little error-proofing to make it harder for people to break the build. If exploratory testing requires too much tedious manual setup, a programmer might put a shortcut in the app for the testers. And so on. All this is a good example of how a guiding value works. It’s always a temptation to skip doing something in the short term that pays off in the long term. But if you believe, down to your core, that you deserve a pleasant work experience — that your focus on your task should not be diverted by the present-to-hand-ness of everything you want to bring to bear on that task — then your software will get progressively easier to work with.

That attitude counters a problem I sometimes see in new Agile projects. They start out with a burst of productivity. They’re actually delivering software! Each and every iteration! The business is ecstatic. But as time goes on, the trend line is toward less and less value per iteration. The software is becoming rigid, just like all the proponents of Big Design Up Front told us it would.

I believe that if you attend to the ease of your moment-to-moment work, your whole project will be pulled toward the the malleability or softness that’s essential to delivering on the Manifesto’s promise.

This kind of malleability I think both allows and is built upon being reactive. That’s odd, since it’s not respectable to be reactive. We’re supposed to be proactive in all things. So let me tell another story.

I needed help with a persistent problem. I’d have planned out a certain path toward a coding goal, and I’d have coded some way along that path when I realized things weren’t working out — I was getting irritated, unsure of myself. Now — having been taught something by the working code I was writing — I saw a new, better path that I should have taken. But to take it now, I’d have to undo some of what I’d done, which is fairly error-prone. Of course, version control systems let you take a snapshot so that you can automatically and flawlessly jump back to a previous state of the code. The problem was, I never made snapshots at the right places, so I didn’t have the right places to jump back to. I asked Zhon Johanson, a better programmer than I, how he decided when to take snapshots to back up to, and he surprised me by saying “I never back up. I just bend the code toward the better solution.”

I hear this metaphor of bending a lot. Rather than spending too much time predicting the future, you take a stab at it and react to what you learn from writing and running code, always flexibly moving toward the goal.

The Agile belief is that — on average — this being reactive in an uncertain world where you must constantly respond to change, wins over priding yourself at being proactive. This is sometimes referred to as waiting until the last responsible moment to make a decision. Of course, knowing when the last responsible moment is, is a matter of skill. So is knowing how to write software that’s soft enough that you can keep pushing the last responsible moment later and later, closer to release. I want to show you an example of such a skill, because it illustrates another guiding value, fast feedback — getting those little twinges that tell you you’re on the wrong track, that it’s time to react.

In school, we were taught to first build infrastructure. Once you got that done, you add the features — pop pop pop — which ought to be pretty easy now, because you took the time to get the infrastructure right. Hmm… “Took the time” seems like a different thing than “fast feedback”. Let’s look at what “taking the time” means.

You might spend three weeks of a month-long iteration doing infrastructure. Only after that do you implement the first feature. What might that implementation tell you? It might tell you that you got the infrastructure wrong. But now you’ve wasted a lot of time. You’re so far off track that bending toward the solution is too hard. You have to backtrack.

Or, maybe worse, it’s only when you finish the first feature that you get to ask the business “is this what you meant?” From their answer, you might discover you have a fundamental misunderstanding of what they were getting at. But now it’s kinda too late — the end of the iteration is upon you. No time to finish what you promised you’d do for them in exchange for your princely salary.

Infrastructure-first development is a recipe for sudden big surprises that make you… the business… your family and friends all doubt whether you really know what you’re doing.

For that reason, Agile teams usually develop differently. I’m going to show you an example, but first a note: Real Agile development is like the drumming you heard at the beginning — it consists of different threads and themes, each moving at its own cadence, but all working together and synchronizing periodically. One of the problems outsiders often have with Agile is that it looks so disorderly, but from the inside you can see how everything meshes.

To avoid confusion, I’m going to show the new-style development in a simplified, serial form. The team starts with one feature. They finish it. They rush off to the business representative — the one who’s talking to them so frequently — and they get the feedback they crave.

Now they add a second feature. But before they go rushing off to show that to the business, another value takes hold: ease. They’ve probably made the code less easy to work with, in no small part because there’s too much of it. They’ll find that the two features have some things in common, and can share some code.

Now the code is easier to work with. There’s been a slight delay before showing the second feature to the business representative — a slight delay in feedback — but it’s worth the cost. (There are always tradeoffs between values.)

This process continues until the end of the iteration.

What’s interesting is that — if you have the skill to develop software in vertical slices and to generalize by removing duplication — you can end up with an app with an infrastructure. In fact, an infrastructure that looks like it was designed by a fantastically good designer who built exactly and only what was needed by the feature set, with no wasted motion at all.

That’s a nice result. You get to the same place via fast feedback route, but with less danger along the way.

But still, we’re talking about feedback cycles perhaps days in length. To a fast feedback addict, that kind of feedback is scarily slow. What’s more comfortable is feedback every few minutes, as you get with test-driven design. If you have this skill, you describe what you want to accomplish in, say, the next five minutes, then you write the code to accomplish it, then you ask the computer to check whether you did what you said you’d do. This kind of fast feedback, measured in minutes, is what Agile people value, and they will constantly be pushing for faster feedback. I believe that, in doing so, they’re often showing how much they value naiveté, a sort of foolish innocence that willfully denies the idea that software has to be rigid.

I realized the power of naivete back when test-driven design became popular. I’d had extensive experience teaching programmers to unit test in the early-to-mid 90’s and also experience with other kinds of test automation. Based on that experience, I decreed that test-driven design would fail. In particular, I predicted that test suites would not be maintained, that almost all of them would be abandoned after no more than two years. And so would test-driven design — it would be abandoned — just as the last fad for programmer testing had been.

The reason, I said, was that changing test suites to match feature and code changes would be too much work. Well, I was wrong. Way wrong, as it turns out. The interesting question is why. What happened that I hadn’t imagined would happen?

What happened was that the programmers looked at what made unit testing irritating, difficult, and unmaintainable. It was the structure of the code (which I’d always assumed was just part of the unchangeable fabric of the universe), so they changed it. They changed the way they programmed because they demanded ease and fast feedback. It was as if they believed that software — including tests — wants to be soft, wants to be maintainable, and all that’s required is to discover the right skills and technologies to unleash that latent potential, to let that malleability out.

This sort of naive utopian attitude works out amazingly often — when you value malleability enough to make the effort, and when you’re naive enough to try what experience — and the experienced — tell you can’t possibly work.

This naivete extends to more than just software. As an example of it, I’ll use another value, visibility. In some sense, visibility is a consequence of fast feedback — there’s no feedback without new information, and it doesn’t even make sense to talk about new information that you can’t see. But Agile teams tend to make visibility spill over into outright exhibitionism. One reason is that it’s often the case that the mere exposure of a problem is enough to see it solved.

For example, there was a team who believed in pair programming — having programmers work in pairs to write code. What they noticed was that certain people tended to work together over and over. Intellectually, the team agreed that’s a bad habit because it inhibits the diffusion of knowledge, something pair programming is notably good at. But knowing something intellectually and acting on it are two different things. They could have solved the problem prescriptively, by making a schedule that explicitly mixed people up. Instead, they just made the problem visible by putting up this chart. All people had to do was tick off the right cell on the chart whenever they worked with someone. The gaps in the chart made their bad habits constantly obvious. It was what Alistair Cockburn calls an information radiator. And that alone was enough to increase the diversity of their pairing. No mandates required.

There’s an implicit assumption here, one about humans, not software. It is that humans, not just software, can be soft, malleable. I think that’s important. You see, the dominant way of thinking about change is that people are so set in their ways that major change requires a crisis — and certainly many organizations switch to agile because of a crisis.

However, the agile attitude seems to deny the necessity of the crisis, the big fuss, and all the techniques needed to shepherd people through the crisis to improvement. Just make bad habits visible enough. The pressure of constant visibility will make you lay those habits aside, and — over time — what you do instead will themselves become habits, but this time good ones. And, over time, this collection of changes can grow great team members and great teams, exactly in the way that the product grows smoothly from something barely functional to something to be really proud of.

I like this emphasis on steady growth over crisis-driven change, no matter how naive and utopian it seems, because it’s getting toward the last of the agile values I want to talk about, one so out of place, one so shameful, that I barely dare utter its name:

joy

Here’s the thing. If you talk to the people who wrote the Agile Manifesto, eventually most of them will make reference to that one great project, the project that made them think “this is the way software ought to be done.” And the fact that the way it ought to be done was hardly ever done is what made them into evangelists for Agile Software Development.

It was such a pleasure, back in my early days of Agile consulting, to go on a gig and hear both programmers and business people say “this is the greatest project I’ve ever worked on”.

And that’s what worries me. As Agile is becoming widespread, as it’s taking hold in larger and more mainstream companies, I’m starting to hear a different reaction: “At least my job doesn’t suck as much as it used to.”

[Imagine a more impassioned voice.]

Ladies, gentlemen, and others, this is not a lofty goal. I think you deserve better. I think that if you have the courage to frankly bend your work in a joyful direction when you have that choice — while still always, always producing working software at frequent intervals — you can do better than a job that doesn’t suck.

However, I’m envisioning your boss asking you what you got out of the conference, and you replying “Joy!” Joy’s too embarrassing to mention in a business setting. So, which of the other guiding values would I like you to concentrate on?

It’s ease. As I travel around different projects, I see them spending their time shaving yaks, solving — but just for today — problems that are keeping them away from solving their real problems. I see them using tools — or skills — or technologies — or techniques — that are always, always going to be awkward, but not being naive enough to try to fix them. And I see them thinking that their environment — the context of their work — 300,000 lines of legacy code — those damn DBAs — the furniture police — this company’s implementation of Sarbanes-Oxley — are all something that they have to put up with, something they’ll be driven by, rather than something they can and should change.

I know high-performing Agile teams would not put up with that. I know they would immediately start adding ease into their work. Do I know that they’re high-performing teams because they add ease to their work? No. I believe it, but I can’t prove it. You can’t either, but you can try leaning toward ease, bending toward ease, and watch what happens.

If what you see — and make visible — are signs that the combination of the team and the code are getting steadily better at producing value, I think the business has no cause to complain that your work seems too smooth — that you’re not struggling — that you seem to be having a good time.

Remember, it’s part of the Manifesto’s original deal that the business doesn’t tell you how to make software, just as you don’t tell them that the features they asked for are wrong and so you’re going to give them different ones.

I feel strongly enough about ease that I’ve made up some posters and brought some here to give you. They’re down front here. If you don’t get one, but want one, send me me mail and I’ll send you a poster — free — until I either run out or get tired of going to the post office.

With that, I thank you, but we’re not going to have quite the usual question period now. Instead, I want to get together a little panel of people who know more about Agile than I do, so that they can tell me what values I missed, or what values I included that don’t belong, or tell stirring stories of teams putting values into use.

Jeff Patton, come on down!
Rachel Davies, come on down!
Linda Rising, come on down!
J.B. Rainsberger, come on down!
Antony Marcano, come on down!
David Hussman, come on down!