I have this urge to write a book…

It would begin like this:

On a bright Illinois morning on the sixth of November, 2010, Teresa Williams dragged a picture of her twins in their Halloween costumes into a mail message. She clicked a button to send it to their grandparents. Three minutes later, in Göteborg Sweden, Teresa’s mother’s phone buzzed discretely. The mail had arrived.

This book is about everything that happened between the click and the buzz.

It would be a popularization, hitting a technical level somewhere around Singh’s The Code Book—that is, technical enough that someone who really knows the details wouldn’t consider it incorrect, but accessible to the layperson.

What do you think?

Old programmers

A correspondent writes:

How does one continue to build a career in software development, when there are younger, hungrier people (i.e.people who can, and will work 16-hour days and can learn things at a ridiculous pace) joining the field?

I’m at the ripe, old age of 33 and am already feeling like it’s a challenge to keep up with the 23-, 24-, 25-year-olds. :/

Also — and I know this is partly a function of the field’s explosive growth over the years — but I just don’t see that many software devs in their 40’s and up, so I don’t have much in the way of precedent to observe, in terms of a career path, other than going into management

Since I’ve pretty much decided to devote this next decade to programming, part of it for hire, this is an important topic to me. (I’m 50.) I don’t know that I have much useful to say, though. Nevertheless…

  • In a team, some people serve as catalysts for other people’s abilities. For example, ever since my 20’s, I’ve been hung up on friction-free work. So I was more likely than other people to make the build work better, write and share emacs functions to automate semi-frequent tasks, or to work on testing. Those are not glory tasks—”I’m a rock star build-fixer!”—but they help the team. As the team’s codger, I might emphasize that bent even more, freeing the team’s whippersnappers to concentrate on the most prodigious feats of coding.

    A typical way in which an older programmer can catalyze is by paying attention to human relationships. If you can avoid the damnable tendency old people have to pontificate at people or tell only marginally relevant stories about how they did it 20 years ago, you can be a person who “jiggles” teams into a better configuration. (Term is due to Weinberg’s Secrets of Consulting, I think.) The ideal role is that of player coach (but one gradually recognized by the team, rather than appointed for it.)

  • Another reason for the patriarch to work in a tight-knit team is that a young programmer’s advantages are not uniform. For example, what makes me feel most like a doddering oldster is the sheer amount of stuff kids these days know about: tool upon tool upon tool, gem upon gem upon gem, the 83 new things in the newest point release of Rails, etc. But if you have one of those people on the team, the advantage accrues to everyone, and the centenarian’s loss of a pack-rat mind is not such a disadvantage.

    When what matters is the team’s capability, balance is more important than each individual’s uniform excellence. So when fogy-dom looms, focus on being complementary rather than unique.

  • A traditional way for the older programmer to cope is by being one of the dwindling number of experts in a has-been technology (Cobol being an example, Smalltalk being another). That technology doesn’t necessarily have to be boring. Sometimes, as has sort of happened with Smalltalk, Lisp, and maybe the purer functional languages, the has-been becomes hot again.

    A perhaps-related route is to become an expert in a very specialized and difficult technology like, say, virtual machines or security–something that’s difficult to pick up quickly and requires continuous learning.

  • Now that we’ve learned that legacy code doesn’t have to suck, perhaps the graybeard should angle to attach himself to a particular large and long-lived code base. There could be a lot of pleasure in watching your well-tended garden improve year after year.

  • It’s also useful not to act old. For example, I have to fight the urge to be sort of smug about not knowing CSS. In my case, of course, I don’t because, well,… CSS. But it’s easily interpreted as my saying “Oh, another passing fad, *Yawn*, give me RPG any day”. Similarly, I should be careful of saying things to Clojure programmers like, “Well the way we did that in my Lisp days was…” As a final example: this website looks like I’ve learned nothing about web technologies since 1994.

    People are sensitive to old people acting old. The flip side is that it’s easy to subvert expectations. I think it’s a good strategy to be able to talk in modest depth about two or three technologies that are sort of new or even faddish. So, for example, I’m pleased that I can talk about Clojure, Cappuccino, and, oh, Sinatra. You want to both present the appearance of being able to–and actually be able to–synthesize the past and the present.

  • Finally: if programming is indeed one of those fields where you do your best work young, older people should be paid less. Older programmers can compete by somehow (credibly) asking for less money.

    That “credibly” is an issue though, since programming is something of a macho, boastful field. In such, a declining salary is easily taken as a big red flag rather than a realistic acknowledgement that–say–short-term memory and concentration are important to programming and both get worse with age.

Other thoughts that might be of help to my fellow coffin-dodgers?

Today’s fun ruby fact

It looks as if block_given? is scoped like a local variable. Try running this:

Shoulda uses this.

Mini-review of Freedom(tm) by Daniel Suarez

Killer ninja motorcycles chop people up. Doc Manhattan’s less-blue relative appears. Good guys win. They establish humane, John Robb-style resilient communities. Bad guys get their just deserts. Huzzahs all around. Except…

…the good guys would have gotten c-r-e-a-m-e-d were it not for the all-wise program on their side and the tech that—oddly—only it knew how to build. So, for us in a non-fiction world, this is actually a pessimistic book: the author sees a bad moon rising, but we seem to be short on inhumanly brilliant game designers to prevent it.

If you liked the previous book, you’ll like this one too.

Mocks and legacy code

While on my grand European trip, I stopped in for a day at a nice company with a fun group of people doing good work on a legacy code base. They challenged me to improve an existing test using mocks. The test was typical of those I’ve seen in legacy code situations: there was a whole lot of setup code because you couldn’t instantiate any single object without instantiating a zillion of them, and the complexity of the test made figuring out its precise purpose difficult.

After some talk, we figured out that what the test really wanted to check was that when a Quote is recalculated because it’s out-of-date, you get a brand-new Quote.

Rather than morph the test, I tried writing it afresh in my mockish style. A lot of the complexity of the test was in setting things up so that an existing quote should be retrieved. Since my style these days is to push off any hard work to a mocked-out new object, I decided we should have a QuoteFinder object that would do all that lookup for us. The test (in Ruby) would look something like this:

quote_finder = flexmock(”quote finder“)
quote = flexmock(”quote“)

during {
   some function 
}.behold! {
 quote_finder.should_receive(:find_quote).once.
              with(…whatever…).
              and_return(quote)
 
}

Next, the new quote had to be generated. The lazy way to do that would be to add that behavior to Quote itself:

quote_finder = flexmock(”quote finder“)
quote = flexmock(”quote“)

during {
   some function 
}.behold! {
  quote_finder.should_receive(:find_quote).once.
               with(…whatever…).
               and_return(quote)
  quote.should_receive(:create_revised_quote).once.
        with(…whatever…).
        and_return(”a new quote“)
}

Finally, the result of the function-under-test should be the new quote:

quote_finder = flexmock(”quote finder“)
quote = flexmock(”quote“)

during {
   some function 
}.behold! {
  quote_finder.should_receive(:find_quote).once.
               with(…whatever…).
               and_return(quote)
  quote.should_receive(:create_revised_quote).once.
        with(…whatever…).
        and_return(”a new quote“)
}

assert { @result == a new quote }

I felt a bit of a fraud, since I’d shoved a lot of important behavior into tests that would need to be written by someone else (including the original purpose of the test, making sure the next Quote was a different object than the last one.) The team, though, gave me more credit than I did. They’d had two Aha! moments. First, the idea of “finding a quote” was spread throughout the code, and it would be better localized in a QuoteFinder object. Second, they decided it really did make sense to have Quotes make new versions of themselves (rather than leave that responsibility somewhere else). So this test gave the team two paths they could take to improve their code.

In the beginning, the QuoteFinder and Quote#create_revised_quote would likely just delegate their work to the existing legacy code, but there were now two new organizational centers that could attract behavior. So this looks a lot like Strangling an App, but it avoids that trick’s potential “then a miracle occurs” problem of needing a good architecture to strangle with: instead, by following the make-an-object-when-you-hesitate strategy that mocking encourages, you can grow one.

I’ve not seen any writeup on using mocks to deal with legacy code. Have you?

P.S. It’s possible I’ve gotten details of the story wrong, but I think the essentials are correct.

TDD & Functional Testing: from collections to scalars

I’ve been fiddling around with top-down (mock-style) TDD of functional programs off-and-on for a few months. I’ve gotten obsessed with deferring the choice of data structures as long as possible. That seems appropriate in a functional language, where we should be talking about functions more than data. (And especially appropriate in Clojure, my language of choice, since Clojure lets you treat maps/dictionaries as if they were functions from keys to values.)

That is, I like to write these kinds of tests:

(example-of "saturating a terrain"
   (saturated? (... terrain ...)) => true
   (because
      (span-between-markers (... terrain ...)) => (... sub-span ...)
      (saturated? (... sub-span ...)) => true
)

… instead of committing to what a terrain or sub-span look like. That’s been working reasonably well for me.

I’ve also been saying that “maps are getters”. By that, I mean that—given that you’ve test-driven raise-position—it really makes no more sense to test-drive this:

(defn raise [terrain]
   (map raise-position terrain))

… than it does to test a getter: it’s too obvious. That leads to a nice flow of testing: I’m always testing the transformation of things to other things. I don’t have to worry, until the very end of test-driving, that the “things” are actually complex data.

The problem I’ve been running into recently, though, is handling cases where complex data structures are converted into single values. For example, I’ve been trying to show a top-down TDD of Conway’s Life. In that case, I have to reduce a set of facts about the neighborhood of a cell into a single yes-or-no decision: should that cell be alive or dead in the next iteration? But expressing that fact is rather awkward when you don’t want to say precisely what a “cell” is or how you know it’s “alive” or “dead” (other than that there’s some function from a cell and its environment to a boolean).

To be concrete, here’s something I want to claim: a cell is alive in the next iteration if (1) it is alive now and (2) exactly two of the cells in its neighborhood are alive. How do you say that while being not-specific? I’ve not found a way that makes me happy.

Part of the problem, I think, is that when you start talking about individual elements of collections, you’re moving from the Land of TDD, which is a land of functions-of-constants to a Land of Quantified Variables (like “there exists an element of the collection such that…”). That way lies madness.

informed-citizens.org

As part of my ongoing (since age 28) midlife crisis, I’ve been casting around for an ambitious product to build. I have an idea. I describe my personal motivation for it, then sketch what “it” is, on a separate blog. I encourage you to try to talk me out of it.

About “Business Value”

I’m notoriously long-winded on Twitter. Here’s what I wrote earlier today:

Bv2

Bv3

What do I mean by that?

In my research into your species, I’ve noticed that humans are social animals. With the exception of ideologues and other damaged people, you work best when your work is oriented toward other people. Not “people” (in the abstract), but actual individuals. Because you’re quite good at personifying objects—look how many of you act like your boats are people—you can also do well when oriented toward improving those object’s “lives”. (That is, you treat your software the way you treat your pets, which is not that different from the way you treat [some of] your loved ones.)

When it comes to software development, I’ve seen teams that are extremely… nurturing toward the product owner. I’ve seen teams that treat the product itself as a person to be groomed and prepared for its entry into the world. I’ve seen teams that identify strongly with the prototypical, personified, or actual end user and want to make her life easier.

Quite often these teams, especially Agile teams, seem obsessively focused on “Business Value”, but that’s in the context of personal relationships. “Business Value” is a shorthand, a way of keeping conversations from going astray, of keeping people focused. It is a term that signals or reminds of other things—it is not a thing in itself.

Increasingly these days, when I hear people theorizing about Agile and Lean, they are treating “Business Value” as a thing in itself. It is treated as an end, rather than as a means. (This is in keeping with the decline of Agile as a bottom-up team-oriented insurgency.)

Who cares? The good thing about old-style Agile is that it tamped down teams’ tendency to be overbearing while still involving them in the conversation about what makes the product better. Code is generative, and programmers could—and did—suggest directions the business could take based on what the code “naturally” “wanted” to do. This could lead to wonderful and illuminating conversations.

When Business Value is determined from On High and is a discrete thing in itself—a product of expertise not accessible to the hoi polloi—this conversation is short-circuited. Analysis (the province of Business) populates swimlanes on the Kanban board or the product backlog. When Development takes an item from the Analysis backlog or swimlane, the signal it sends upstream is “I need a new chunk of work” not “Let’s talk about the next good idea.”

That is: your species has another skill. Just like you’re good at turning objects into people, you’re good at turning people into objects. It’s easy for you to subordinate actual humans to the beauty of a System. You’re terribly prone to slip into ideology, to elevate objects to totems-to-be-deferred-to. “Business Value” is, I fear, becoming one such totem.

To forestall the inevitable comment: I know (I know!) that in a well-functioning organization with Respect for People, the ugliness I describe wouldn’t happen because wise philosopher-kings wouldn’t let it. I just believe there is a shortage of such wise philosopher-kings and—in their absence—we should cut with the grain of human nature.

A sort of thought about interaction (and perhaps state-based) tests

This here post is about making tests terse by specifying what has happened instead of (as in interaction tests) who did it or (as in state-based test) the different kinds of things-it-has-happened-to.

I have a test that says that the Availability object should use the TupleCache object to get particular values for: all animals, animals that are still working, and animals that have been removed from service. If one wants to show animals that can be removed from service, it’s this:

all animals - animals still working - animals already removed from service

Here’s a mock-style test that describes how the Availability uses the TupleCache:

 should use tuple cache to produce a list of animals do
      @availability.override(mocks(:tuple_cache))
      during {
        @availability.animals_that_can_be_removed_from_service
      }.behold! {
        @tuple_cache.should_receive(:all_animals).once.
                     and_return([{:animal_name => out-of-service jake‘},
                                 {:animal_name => working betsy‘},
                                 {:animal_name => some…‘},
                                 {:animal_name => …other…‘},
                                 {:animal_name => …animals‘}])
        @tuple_cache.should_receive(:animals_still_working_hard_on).once.
                     with(@timeslice.first_date).
                     and_return([{:animal_name => working betsy‘}])
        @tuple_cache.should_receive(:animals_out_of_service).once.
                     and_return([{:animal_name => out-of-service jake‘}])
       }
      assert_equal([”…animals“, …other…“, some…“], @result)
    end

I’m not wild about the amount of detail in the test, but let’s leave that to the side. Notice that the results of the test imply that the Availability is turning the tuples (think of them as hashes or dictionaries) into a simple list of strings. Notice also that the list of strings is sorted. Noticing that brings a couple of questions to mind:

  • That sorting - does it use ASCII sorting, which sorts all uppercase characters in front of lowercase? or is it the kind of sorting the users expect (where case is irrelevant)?

  • Are duplicates stripped out of the result?

As it happens, I want the responsibility of converting tuples into lists to belong to another object. I’d prefer Availability to have only the responsibility of asking the right questions of the persistent data, not also of massaging the results. I’d like to put that responsibility into a Reshaper object. Here’s an expanded test that does that:

    should use tuple cache to produce a list of animals do
      @availability.override(mocks(:tuple_cache, :reshaper))
      during {
        @availability.animals_that_can_be_removed_from_service
      }.behold! {
        @tuple_cache.should_receive(:all_animals).once.
                     and_return([”…tuples-all…“])
        @tuple_cache.should_receive(:animals_still_working_hard_on).once.
                     with(@timeslice.first_date).
                     and_return([”…tuples-work…“])
        @tuple_cache.should_receive(:animals_out_of_service).once.
                     and_return([”…tuples-os…“])
        # New lines
        @reshaper.should_receive(:extract_to_values).once.
                  with(:animal_name, [’…tuples-work…‘], [”…tuples-os…“], [”…tuples-all…“]).
                  and_return([[”working betsy“], [’out-of-service jake‘],
                              [’working betsy‘, out-of-service jake‘,
                              some…‘, …other…‘, …animals‘]])
        @reshaper.should_receive(:alphasort).once.
                  with([’some…‘, …other…‘, …animals‘]).
                  and_return([”…animals“, …other…“, some…“])
      }
      assert_equal([”…animals“, …other…“, some…“], @result)
    end

It shows that the Availability method calls Reshaper methods which we could see (if we looked) guarantee the properties that we want. But I don’t like this test. The relationship between Availability and Reshaper doesn’t seem to me nearly as fundamental as that between Availability and TupleCache. And I hate the notion that the general notion of “convert a pile of tuples into a sensible list” is made so specific: it will make maintenance harder. And I’m not thrilled (throughout this test) of the way that the human reader must infer claims about the code from the examples.

So how about this?:

   should use tuple cache to produce a list of animals do
      @availability.override(mocks(:tuple_cache))
      during {
        @availability.animals_that_can_be_removed_from_service
      }.behold! {
        @tuple_cache.should_receive(:all_animals).once.
                     and_return([{:animal_name => out-of-service jake‘},
                                 {:animal_name => working betsy‘},
                                 {:animal_name => some…‘},
                                 {:animal_name => …other…‘},
                                 {:animal_name => …animals‘}])
        @tuple_cache.should_receive(:animals_still_working_hard_on).once.
                     with(@timeslice.first_date).
                     and_return([{:animal_name => working betsy‘}])
        @tuple_cache.should_receive(:animals_out_of_service).once.
                     and_return([{:animal_name => out-of-service jake‘}])
      }
      assert_equal([”…animals“, …other…“, some…“], @result)
      assert { @result.history.alphasorted }

The last line of the test claims that—at some point in the past—the result list has been “alphasorted”. A list that’s been alphasorted has the properties we want, which we can check by looking at the tests for the Reshaper#alphasort method.

In essence, we check whether at some point in the past the object we’re looking at has been “stamped” with an appropriate description of its properties. Therefore, we don’t have to construct test input that checks the various ways that description can become true - we simply trust earlier tests of what the stamp means.

Here’s code that adds the stamp:

    def result.history()
      @history = OpenStruct.new unless @history
      @history
    end
    result.history.alphasorted = true
    result.freeze

(Notice that I “freeze” the object. In Ruby, that makes the object immutable. That’s in keeping with my growing conviction that maybe programs should consist of functional code sandwiched between carefully-delimited bits of state-setting code.)

Having said all that, I suspect that the original awkwardness in the tests is a sign that I need a different factoring of responsibilities, rather than making up this elaborate solution. But I haven’t figured out what that factoring should be, so I offer the alternative for consideration.

Changing basic assumptions in apps

One of the hardest things for me as an app developer is changing basic assumptions in a safe, gradual way. Here’s an example.

Critter4Us is an app that reserves animals for demonstrations and student practice in a veterinary college. There are business rules like “an animal can be used to practice injections no more than twice a week”.

In its original form, Critter4Us only made reservations for one point in time (the morning, afternoon, or evening of a particular day). I’m now making it take reservations for “timeslices”, where a timeslice has a start date, an end date, and any subset of {morning, afternoon, evening}. Doesn’t seem like a huge change, but it turns out to be pretty fundamental:

  • A lot of questions vaguely like “is this point within this range?” are now “do these two ranges overlap?” There’s an existing Timeslice object (which, despite the name, was about points in time). It has some time-related behavior, but responsibility for some other behavior leaked out of it because the data was so simple.

  • Some of the database operations were annoyingly slow after being hard to get right, quite likely because my SQL-fu is weak. Since the questions are becoming more complex, I want to do less calculating when questions are asked and more stashing partial answers in the database. So adding this feature requires more database work than just a simple schema migration.

    (This feels like premature optimization of a feature I don’t know wouldn’t be fast enough, but my main motivation is that I’m more confident of getting the right answers if I stash partial answers. This is the most important business logic in the app.)

  • The UI needs to change. Should that be done by upgrading the existing reserve-a-point-in-time page (adding clutter for a case that’s used seldom) or adding a new page?

I want to make all these changes in such a way that (1) the tests are all passing most of the time and (2) the app is deployable most of the time. I devised the following strategy after writing a partial spike and throwing it away:

  1. DONE Change the current Cappuccino front end. It used to deliver

    ?date=2010-09-02&time=morning
    

    to the Sinatra backend. Now it delivers

    timeslice={'firstDate':'2010-09-02', 'lastDate':'2010-09-02', 'times':['morning']}
    

    This isn’t hard because the front end doesn’t do any calculations on the date.

  2. DONE Have the backend controller that receives the data quickly convert the new format into the old.

  3. DONE Move conversion of new-format-to-old into the Timeslice object. (Only two classes make Timeslice objects, so that’s easy.)

  4. DONE Change the Reservations table to add new columns. Change the Reservation object to allow it to be constructed using either the old or new format. Change all the non-test code to use the new constructor. (It’s convenient to keep the terser form around for the tests that use it.)

  5. IN PROGRESS
    Reservation and Timeslice aren’t completely dumb objects — they probably get told more than they get asked — but they do have accessors for date and time. Those accessors are still meaningful because (at this point) the first_date and last_date are always the same and the set of times can only ever contain one element.

    However, change their names to faked_date_TODO_replace_me and faked_time_TODO_replace_me. Run the tests. For each no-such-method failure,

    • If the purpose of the call can easily be expressed in terms of the new interface, rewrite the call to use it.

    • If not, use the convoluted, soon-to-be-replaced name.

  6. Replace the old code that answers the question “for which procedures may this animal be used at this moment?” with code that assumes cached partial answers. Mock out the cached partial answers and thereby design the partial answer table.

  7. Add the cached table to the database. Change the code that creates reservations to cache partial answers.

  8. Working from the controllers down, examine each method that uses faked_date_TODO_replace_me and faked_time_TODO_replace_me. What does the method do? What makes sense in a world where reservations and uses of animals are not for a single point in time? Where does the method really belong? Fix them.

  9. Now that all uses are fixed, delete the methods with silly names.

  10. Generalize the question-answering code to answer questions about timeslices more complicated than single points. Much careful testing here.

  11. Change the UI to collect more complicated timeslices.

It’ll be interesting to see what the final structure of the code looks like. A lot of this code was written early in the project, so I’m sure it’ll improve a lot.

If you want to see the code, it’s on GitHub.