Exploration Through Example

My final word at the NSF workshop:

A Plug for Natural History
in the style of
Darwin’s Living Cirripedia (in 2 volumes!)

1. Ever since the TRW studies, Software Engineering has been primarily the Software Engineering of the large corporation. I maintain that many small businesses operating within platform ecosystems develop software differently than large enterprises, and that the differences are both important and interesting.

2. Many researchers think they understand Agile but - through no fault of their own - do not. (It’s like this: you can have read books, you can have read about writing books, you can have written articles - nevertheless, writing a book well is a different thing than writing an article.)

To put someone else’s time where my mouth is: Dave Hoover of Obtiva has volunteered to host researchers who’d like to visit. I can even imagine some more ambitious get-together that bridges the gap I see.

Result: Except for RPG, no interest yet.

Here’s an example of what I meant by my earlier position. Suppose I were a professor and someone wanted to do a PhD under me. He or she is sort of interested in databases, sort of interested in software engineering, sort of interested in whatever I can come up with. I would say something like this:

“There was a NoSQL workshop recently. People who developed non-relational databases (mostly because of scaling issues) got together to talk. Over the next few years, I expect you to become the academic expert on the, uh, ‘totality’ of that sort of database. That means things like these:

“You’re going to need to describe the databases with the same care and precision that Charles Darwin used to describe barnacles. That would include the obvious, like some sort of taxonomy for them—preferably one that you discover in the wild rather than bring to it. Here’s a wild idea: there’s a style of research they use in the social sciences for that, grounded theory. (Take a look at Angela Martin’s dissertation and what Robert Biddle’s students do.) You’ll have to figure out how to apply it to non-humans.

“But more than that, I want you to write some superb user documentation for at least one of the databases (pick the one that needs it most). I think writing about how to use something gives you great insight into that thing. You’re also contributing back to the world that’s giving you something to study, and the taxpayers who are paying part of both of our salaries.

“Now, before you can write superb user documentation, you have to use the thing being documented. You should become a better-than-just-competent user of at least several of these.

“I also want you to learn the internals, both from a design point of view and also from a nitty-gritty point of view. So you should become a committer on at least one of the open source projects.

“I also want you to travel and work at places that use the databases. Part of your research will be on how such databases have disrupted (or just changed) both normal development processes and also the systems being built. I can imagine you going to four kinds of places:

• “Places that are switching an existing app to the new-style database.

• “Places that are starting out with a new-style database.

• “Places that have been using the new technology for a while.

• “The places that first wrote one of the new databases.

“At the end of all that, I want you to be able to say something wise or at least suggestive about the introduction of disruptive technology.

“Since you’re going to be an expert on at least one of the databases, we can get companies to pay to have you visit (in at least the first two cases and—I bet—in the third). We’ll see if we can scrounge up grant money for the fourth kind, but you may have to do a Corey-Haines-style tour.

“That’s a start. I expect we’ll think of more things that push you toward learning everything there is to know about a particular part of the world, from every sensible angle.”

I got invited to a National Science Foundation workshop whose…

goal broadly speaking will be to reflect on software
engineering research and practice of the past, to determine some areas
in which “rethinking” is needed, and to engage in some rethinking.

I have a position (strongly held, of course). But when I look at the list of attendees and their affiliations, I worry that I may be the only person there representing, well, you: the kind of person who reads this blog or (especially) follows me on twitter. So I’d like to hear what you think needs rethinking. (Please include a tiny bit about what kind of person you are. Something like “rails developer in boutique shop”.) You can reply here, via email, or by responding to my tweet on the subject.

My basic position goes something like this:

Twenty years ago, the self-conception of the software engineering researcher went something like this, I think:

I am a scientist, though partly of the applied sort, discovering important principles which engineers will then put into practice when building systems. I aspire to be something like the Charles Darwin of The Origin

I think the world has moved on from that. What was once an intellectual structure that could be considered built has since exploded into something more like a genuine ecosystem. There are vast numbers of people, communicating in all kinds of strange ways, assembling practically innumerable pieces of software in scrapheap ways, collaborating—quite often across organizational boundaries—in a way more reminiscent of a rhizome than a command-and-control tree.

The appropriate response to that is to go back in time before The Origin to the Charles Darwin who spent eight years studying barnacles, grasping their differences, and publishing the definitive work on them. There is a lot to study in the software development ecosystem (which, note especially, will require collaboration with researchers who know more about people and groups than context-free grammars).

Time to switch from Alan Kay’s glamorous “the way to predict the future is to create it” to something more in keeping with Kuhn’s normal science: the world is out there; mine it for deep knowledge and extensive data (against, I hope, the background of the “paradigms” used by practitioners).