In Defense of Testability

Three facts for your consideration:

• There is an ongoing conversation in the blogsphere about testing, specifically whether the design and code should be modified to support testing.  See Roy Osherove, Eli Lopian, and Stop Designing for Testability.
• I recently worked on a project that involved legacy code that was from 20 to 30 years old (COBOL if that matters).  The project in question was to rehost the software to a new platform.  This was at least the fourth time that it had been rehosted.
• My wife and I are having the basement in our home finished.  This has led to relocating a number of pieces of plumbing and electrical equipment in order to make it possible for us to get to them once the drywall is up.  it has also led to some questions about what the previous owners of the house were thinking when they laid out the basement in the first place.  We are paying for the effort to correct that thinking.

How are all of these facts connected to each other? 

Let me start out with the premise that while it is not true of all software, some software can last a long time and during that lifetime the software will have to transition to several different environments.  I do know that the "old timers" at my COBOL re-hosting project were surprised that the software was still alive after all of this time. 

It is probably unreasonable to define a hard requirement during the gestation of the software that says that the software will be capable of transitioning to at least one new environment within its lifetime.  In most cases, the details of the new environment will not be known.  I still think that question ought to be asked, at least to the extent of saying that the expected lifetime of the software is short (a quick utility to load some data), medium (a web site to handle a specific event coming up in six months), or long-term (this software defines how we run our business). 

If the lifetime of the software is long-term, I think that you ought to think about some additional user stories or use cases.  These might be stories about the versioning of the software, or the migration to a new piece of equipment or a new platform.  That is what my wife and I are doing as we layout the plan for our basement.  We have a vague long-term vision but our financial resources are at best medium-term.  In talking to the people who are doing the work for us, one message comes through quite clearly: there are things that are much cheaper to do now (before drywall and paint) than later, so much cheaper that even though our vision is not particularly clear, we would be stupid not to add them to the work plan.  There have been a half-dozen things that we have done, ranging the adding a couple of electrical outlets in places that we had not thought about to moving some plumbing. 

If I apply some of the agile principles to this behavior I get a "tacky buzzer" sound signalling that "you are not going to need it" applies.  One defense here is that drywall is not nearly as malleable as software; it is much tougher to refactor drywall.  Given this difference, it make more sense to build in "as yet unneed" functionality in our basement.  The problem that I have is that I am not so sure that the differences between drywall and software are as great as the agile priniciples would have us believe. 

OK, I got two of them tied together.  Now to take a cleansing breath and go for three out of three.

Why do we do unit testing?  I am ignoring the issue of whether the unit tests arise out of Test-Driven Design/Development or (hopefully) shortly after the development.  One of the major reasons we do create and run unit tests is the confidence that the tests give us as we change the software.  Make a change and re-run the unit tests to demonstrate that we have not messed up. 

In the literature, the change most often discussed is "refactoring" but there are other kinds of changes.  If we change something external to the software, a successful execution of the unit tests can allay our fears of that external change.  For example, I might change the version of the compilier; the unit tests would reassure me that this external change did not cause any problems.  Ditto for a new version of the operating system, a third party component, or web service.  A change in a configuration item might be a reasonable trigger.  I would argue that any change in the external environment would be a good reason to run the unit tests; change and unit test to demonstrate that we have not messed up.  As above, we need to think about the user stories where the user is anyone who makes a change to any environmental factor.

If it makes sense to run the unit tests on any change in external factors, it should not be a huge step to get to the point where we just run the unit tests every so often to maintain our confidence that things are OK.  [Wait for it,  here it comes.]  "Gasp! You are talking about during production, aren't you?"  Yes, I am.  The reality is that the environment of the software changes all the time.  It would be nice if there was a warning and that the owners of the software could veto the change until sufficient testing demonstrated that there were not going to be any problems.  Good luck on getting that to happen all the time.

We would not dream of moving external input from a WinForm or Web page to a database without validating it.  We know from experience that would be a form of fatal madness.  We would not dream of skipping the error handling.  More madness there.  The reality is that for software (as for everything else), it is a cold and harsh world out there.  Trust no one!  Verify everything!  Including the correct execution of the function of our own software.  How do we verify the correct execution?  The answer is unit and functional testing.  For me, this strongly suggests that at least some of the unit tests ought to be included in the production release and that there should be a process that runs these unit tests periodically to determine if things are still OK.

There is precident for this kind of duplication.  Certain avionics boxes actually implement two or more algorithms to support a given function.  This is typically done for "mission critical" functions such as the auto-pilot.  There might be three different algorithms, developed by three different teams who should have no contact with each other.  Every input is processed independently by each algorithm.  If the outputs of all of the algorithms match (within some tolerance), the auto-pilot assumes that things are going OK.  If the outputs do not match, the vote-tabulation logic may disregard one of the algorithms ("two out of three is good enough") or shut the auto-pilot down (requiring the pilot to actually fly the plane manually).  This sounds a lot like running unit tests during production. 

tags: unit+testing, unit+tests, YAGNI