Test Maintenance; Or, The Third Era of Programmer Testing
Here’s an app that’s used to reserve teaching animals at the University of Illinois vet school:
A user selects procedures to perform by clicking in the leftmost window; animals to perform them on in the rightmost window. Clicking moves the procedure or animal to the corresponding center window and can also narrow down the possible choices. For example, some procedures can only be performed on horses. If you pick one of them, all the cows disappear from the rightmost window.
The first user interface didn’t look like this. It used three windows instead of four, and drag-n-drop instead of clicking. It was heavily tested using a mock-centric style like that of Growing Object-Oriented Software. The switch to a new interface broke all those tests. Bummer!
However, I discovered an interesting thing while dealing with the broken tests. None of them applied directly to the new user interface. However, each of them had some original purpose; each captured an issue that had been important. The exact issue was no longer important, but usually one very like it applied to the new UI. So the broken tests became something of a checklist and spur to the imagination. Rather than just fixing them, or throwing them away, I used them to ask questions: “What user intention or user safeguard does this test represent? How would that intention be exercised in the new UI? [And what test should be written?] Is there a similar danger in the new UI? [And what test would show that the user is protected from it?]”
It was something of a revelation. Allofasudden, maintaining the tests had actual value. Yes, mock-heavy tests do require more maintenance. No, that doesn’t bother me.
Being prone to grandiose statements, that’s led me to proclaim the dawning of the Third Age of Programmer Testing.
The First Age
In the first age, the vast majority of programmers thought testing was boring, pointless, beneath them, and a big waste of time. A much smaller number of programmers thought it was worth doing, although virtually no one was happy with it. I think we all realized, at least unconsciously, that something was wrong with our practice.
The Second Age
Around the beginning of this century, the practice of programmer testing took a big jump forward. Things like one-test-then-just-enough-code are, in part, mental tricks: avoiding one big chunk of testing (at either the beginning or, more usually, at the end) fools our lizard brains into thinking that we’re programming the whole time, so it knows not to be bored. We know much better how to write code that’s easy to test, avoiding tedious and time-consuming test-writing (especially setup). And so on.
Many, many programmers are still stuck in the First Age, but I don’t particularly care. I don’t have to work with them. There are plenty of Second Age folk around.
However, an interesting thing about Second Agers: the vast majority of them think test maintenance is boring, pointless, beneath them, and a big waste of time. A tiny minority (maybe just me?) embraces it.
The Third Age
Just as test-writing took a big leap forward in the past 12 or so years, I predict test-maintaining will do the same over this decade. We’ll develop mental tricks, coding and testing strategies, and tools that will make maintenance as much a thinking-about-design activity as test writing is today.
June 20th, 2011 at 7:02 pm
Hi Brian,
I am a Java EE developer with quite some years on my back that has been reluctant to go from the First Age to the Second Age exactly because of the test maintenance issue. I have seen a few projects where enthusiastic young developers used TDD from the start of the project. As soon as version 1 of the software was out it, and it now came to maintaining the unit tests, these enthusiastic young developers became much less enthusiastic, and after seeing the unit tests fail with each change they made soon went to commenting out failed unit tests until finally they turned off unit testing in their CI server completely. Worse case was even if new people came in that were supposed to work and maintain other people’s unit tests. Nobody did that. As soon as somebody left the project his unit tests were pretty much switched off.
I agree with you that the mental trick of seeing the unit test as coding is something I will try out (the project I just started is very agile with everybody on the TDD train). What I see is that changing and maintaining unit tests to match changes in your software is akin to the old Java EJB 2.1 development cycle: In order to change one thing in the code you actually had to touch up to 7 files (implementation, local and remote interfaces, descriptors, etc.). With TDD we have the situation that for one change in the code worst case I have to change several unit tests and the mocks and stubs related to unit tests. What do you think about that?
BTW: You are lucky that I am very interested in your story, otherwise I would not have bothered to register to post on your blog. I found that quite a high barrier to participate in a discussion.
June 27th, 2011 at 4:58 pm
Marc: I am not very familiar with the Java ecosystem after around 1998. I do have one good rule of thumb, which is that there should be no words in a test that are unrelated to the purpose of the test. Typical example: if the test is not about logging in, it should say no more than “given logged-in user Jane…”.
That’s part of my message. In “the third age”, we’ll need to write tests differently and also have infrastructure that doesn’t penalize change (as so much of Java does, it seems to me as I visit client sites and pair with Java programmers). The issue is not “… we have the situation that …” but: how do you change your situation? [Ideally, that could be done gradually, a la Feathers’ /Working Effectively with Legacy Code/, but that may not be possible.]