Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick

191.8

⇒

167.2

⇒

186.2

183.6

184.0

183.2

184.6

Fri, 05 Dec 2003

Testing language

Martin Fowler writes (quoted in its entirety):

I'm currently sitting in a session at XP day where Owen Rogers and Rob Styles are talking about the differences between XP's unit and acceptance tests. This triggered a thought in my mind - what should a language for writing acceptance tests be?

Commercial tools for UI testing tend to have their own proprietary language. Brett Pettichord, amongst others, question this; preferring a common scripting language such as Ruby or Python.

But I wonder if a language designed for programming is really the right language for writing tests. The point about tests is that they operate by example. They don't try to cover how to handle any value, instead they describe specific scenarios and responses. I wonder if this implies a different kind of programming language is required. Perhaps this is the truly startling innovation in FIT.

Acceptance tests are things to point at while having a conversation. The speakers will be testers, programmers, and business experts. The neatest surprise with FIT, to my mind, is the way you can turn a web page of tests into a narrative. I've mislaid my good example of this, but here's a snippet of page about an example Ward often uses: a bond trading rule that all months should be treated as having 30 days. (Note: I don't know if the following is correct - I was just playing around with FIT, not writing a real app.)

The end of the month is another special case. The last day always counts as day 30, even though that means a jump between the next-to-last day and it. fit.Thirty360 from to actual() thirty360() Feb 27, 2003 Mar 1, 2003 2 4 Feb 28, 2003 Mar 1, 2003 1 1 Feb 28, 1996 March 1, 1996 2 3 Feb 29, 1996 March 1, 1996 1 1 Days past day 30 in a month just count as 30. fit.Thirty360 from to actual() thirty360() Jan 29, 1950 Feb 1, 1950 3 2 Jan 30, 1950 Feb 1, 1950 2 1 Jan 31, 1950 Feb 1, 1950 1 1

The page is organized as set of special cases, each with a brief description followed by some checked examples. I really like this style: it's an easy-to-write mini-tutorial for the programmer and reader.

However, when it comes to tests that are sequences (action fixtures), I find that the tabular format grates. The table, instead of being a frame around what I read, is something that intrudes on my reading. It may be that I'm too used to code.

Another difficulty is variables: such useful beasts. Think of constructing a circular data structure. Yes, you can write a parseable language to do that (Common Lisp had one, as I recall), but to my mind it's simpler just to create substructures, give them names, and use the names to lash things together.

Or consider stashing results in a variable and then doing a set of checks. Or comparing two results to each other, where the issue isn't exact values but the relationships between values.

You could invent variables in the FIT language, but you're starting to get into notational problems that scripting languages have already solved. That way lies madness (after inventing loops and subroutines and...)

And yet, there is the niceness of HTML format. I've been toying with the idea of a free-form action-ish fixture that looked like this:

fit.ScriptFixture

start 'stqe' stop check records.size, 1 forget 1 check records.size, 0

Running the test might yield this:

fit.ScriptFixture
start 'stqe' stop check records.size, 1 forget 1 check records.size, 0	0 expected 1actual stack trace log

Here, I'm following RoleModel's lead in using Ruby as a somewhat customer-readable language. (<rant>And I also fixed the backwards order of xUnit assert arguments. Few English speakers say, "check that nice is the weather" instead of "check that the weather is nice". When I'm talking to people as I write an assert, it's really awkward that the required typing order isn't the natural order of speaking or thinking - first you think about what you're going to check, then you think about what the value should be. It's too late to fix this in xUnit, but we can do better for our customers.</rant>)

This format also allows for a mixture of scripting and tabular styles. I have some tests that look like this:

# A simple start->stop start mail # record 1 assert_states [mail], [] assert_new_record 1, mail, mail_start stop mail # record 1 stops assert_states [], [] assert_stopped_with(1, 1.minute)

You'll notice that I indent the asserts. That's because the tests (especially the longer ones) do a lot of starting, stopping, and pausing of various interacting jobs. It's too hard to see what's going on if the the asserts aren't visually distinguished from the actions.

Nevertheless, it's still pretty ugly. I showed it to a customer-type (my wife, who knows no programming languages). She understood it fine after I explained it, but it didn't thrill her. This might be better:

timeclock.fit.StateActions
step	running job	paused jobs	extra checks
start mail	mail	none	check records.last.number, 1 check records.last.job, mail check records.last.start_time, mail_start
stop mail	none	none	check records.last.number 1 check records.last.elapsed_time, 1.minute

The fixture synthesizes all the cells into one Ruby method. Is this better? (I'm not a particularly gifted visual designer, as anyone who's looked at my main website can tell.)

A final thought. I've read a couple of papers on intentional programming. (Can't find any good links.) It didn't click for me. I think it was the examples. One example had a giant embedded in some C code. Stacking and n and C are simultaneously variables in the surrounding function and mathematical variables in the equation. The idea of intentional programming is that there's one underlying representation (our friend, the syntax tree). Different bits of it get converted into the most suitable representation. So the mathematical bit gets turned into an equation¹.

The thing is, I found the mixture of C and mathematical notation jarring. Is the equation really clearer - to a programmer immersed in the app - than
Stacking = sum(0, n, C) / det(C)? The thing is, here we have one person reading two styles. I'm not sure that the required switching between styles isn't more cognitive trouble than it's worth.

But when we're talking about tests, we have two people who have very different cognitive styles, goals, and backgrounds reading one text. I can envision a common underlying test representation that can be switched between two modes. One is the "discuss the test with the customer" mode. The other is the "see the test as executable code" mode.

Perhaps Intentional Software will take up Martin's challenge. (If so, my consulting fees are quite modest...)

¹ The equation shown isn't the one used in the paper, not unless there's been an amazing coincidence.

## Posted at 16:41 in category /testing [permalink] [top]