Thu, 08 Dec 2005
Working your way out of the automated GUI testing tarpit (part 1)
In this series, I'll present two ideas that have been percolating
in my head for a while. Last week, I began thinking they might be
appropriate for a client. We ended up taking a different approach,
but not until after I'd spent an evening building a
prototype. Yesterday, I was so sick of replying to mail, chipping
away at a task backlog that's metastasized during recent travel, and
slogging through other things I really ought to be doing that I
rebelled and decided to rewrite the prototype. It was fun.
The general idea here is (1) to gradually work your way toward
declarative tests that generate their own page navigation and (2)
to use caching to speed up tests and maybe
improve program structure.
I've never tried these ideas for real. They might be impractical
in the wild.
Here are three GUI-oriented tests, in increasing order of
goodness. The scenario has something to do with a veterinary clinic
(of course). In each test, a case record is created, an animal visit
is recorded, and an audit record is appended. (All the steps are
necessary, because you can't record a visit until there's a case, and
you can't create an audit record until there's a visit.) Normally,
there can be multiple audits attached to a case. But if the first
audit is marked as "nominal", it's the only one that can ever be
created. If so, there should be no "Add Audit" button on the Case
Management page. That's what the test checks. (It also uses the title
of the page to make sure the assertion is checking the right page.)
The first test is like one you might get from a straightforward use
of Watir
or jWebUnit.
def test_cannot_append_to_a_nominal_audit
go('http://app.com/app')
enter(:login, 'unimportant')
enter(:password, 'unimportant')
press('Login')
press('New Case')
enter(:client, 'unimportant')
enter(:clinic_id, '213')
press('Record Case')
press('Add Visit')
enter(:diagnosis, 'unimportant')
enter(:charges, '100')
press('Record Visit')
press('Add Audit')
enter(:auditor, 'unimportant')
enter(:variance, 'nominal')
press('Record Audit')
assert_page_title('Case Management')
assert_page_has_no_button_labeled('Add Audit')
end
|
What are the problems with this test?
In all this code, what's important? Only two lines, which I've
highlighted so you can find them easily. In real life, the
important lines aren't in bold blue font, so such tests
are hard to read.
The test is fragile in the face of change. Change the
name of a field, introduce another field that has to be filled
in, split a page in two: all of these will break this test and
many, many others besides. Now you get to fix them all. Because
they're hard to read, it's easy to fix them badly. (There are a
lot of tests out there that inadvertently no longer test what
they're supposed to test.)
The test is likely to be slow, because it drives a
browser. Programmers who are used to a
fast test-code-refactor
cycle won't put up with that. So the tests will be run
infrequently, and they'll provide information well after it'd be
most valuable.
To solve the problem of fragility, some people put a library between
the tests and the browser. Here's what such a test would look like:
def test_cannot_append_to_a_nominal_audit
go('http://app.com/app')
login('unimportant', 'unimportant')
new_case('unimportant', '213')
new_visit('unimportant', '100', nil)
new_audit('unimportant', 'nominal')
assert_page_title('Case Management')
assert_page_has_no_button_labeled('Add Audit')
end
|
The test is easier to read, but it has some problems. The fact
that an audit record exists is essential to the test, whereas
the existence of a visit is incidental. Yet they're given equal
prominence. The use of the "unimportant" token makes the use of
"nominal" stand out - that particular value must be important to this
test. But what about "213" and "100"? They're not important, but
there's no convenient "ignore this value" token for numbers.
It is more resistant to change than the previous test. If there are
changes within a page, you might only have to change one
library method.
But other changes can still break a bunch of tests. In the next
iteration, suppose
an FDA contact record has to be added before an audit can
happen. That means
every test that goes directly from adding a visit to adding an audit
record will become broken. Either you fix all the tests or you
change new_visit to silently add an FDA contact record
- which I guarantee will make for some frustrating debugging down
the road.
It's just as slow as the previous version.
I believe such a test is still not good enough. It's
still procedural - it's still of the form "do this... now
this... now this... finally you can check what you care about." Here's a
better test:
def test_cannot_append_to_a_nominal_audit
@browser.as_our_story_begins {
we_have_an_audit_record_with(:variance => 'nominal')
we_are_at(:case_display_page)
}
assert_page_has_no_button_labeled('Add Audit')
end
|
-
This test is declarative. It says that there must be a case
with an audit record, but it doesn't say how that record's
created. Moreover, it strives to be minimal, to use no word
unless it's clearly related to the intention of the test.
It says nothing about any of the fields that the previous
tests described as "unimportant". It's even silent on the existence
of case records and visits, simply assuming that whatever's required
for there to be an audit record has happened. (Presumably,
requirements like "you can't add an audit record unless there's been
a visit" have been tested elsewhere.) All of this makes the
test still easier to read.
-
The test is even more resistant to change. Because there's no
sequence of steps in the test - no workflow - changes to the
workflow will require localized changes in the support code, not to
the tests themselves.
-
However, the test is still just as slow as the other ones,
so there's room yet for improvement.
In the next installment, I'll show what the code behind the scenes
looks like. Right now, I want to emphasize that all three tests do
the same thing. Here's an execution log for the third test:
$ ~/src/procedural2declarative 601 $ ruby declarative-test.rb
Loaded suite declarative-test
Started
Go to <http://app.com/app>
Enter "unimportant" into field :login
Enter "unimportant" into field :password
Press "Login"
Press "New Case"
Enter "unimportant" into field :client
Enter "213" into field :clinic_id
Press "Record Case"
Press "Add Visit"
Enter "unimportant" into field :diagnosis
Enter "100" into field :charges
Press "Record Visit"
Press "Add Audit"
Enter "nominal" into field :variance
Enter "unimportant" into field :auditor
Press "Record Audit"
.
Finished in 0.005032 seconds.
1 tests, 1 assertions, 0 failures, 0 errors
|
## Posted at 08:02 in category /testing
[permalink]
[top]
|
|
