Assembly tests

“Lights on, lights off, no smoke. Works.”

A delivery manager from a large consulting firm once told me about a peculiar breed of automated system tests his team was using to catch regressions with minimal effort. He called those tests “assembly tests”, essentially they start an entire application, roughly verify that it is somehow plausibly reacting to elementary input and shut it down again.

I tried to come up with literature on assembly tests but all I could find is this link [1] which is not how I remember his description of assembly tests, and is definitely not how I used them ever since in my projects – again, to great success.Update 25.01.2016 Dan North pointed out that assembly tests are known commonly as smoke tests [5].

The gist of assembly testing is: if I power the thing on and it moves a bit without exploding, then it must be kind of working. A bit more formally: an assembly test runs parts of the system and verifies that no trivially observable malfunction occurs. Let’s look at some examples.

If we wanted to assembly-test a desktop application we would have a build tool first check out the application source code from a repository, build it, deploy the binaries on a testing machine and execute the binaries. The assembly test would then simply check that the application opens a window, that it can close the window again and that there are no error messages in the application log file.

Another example: If we wanted to test a web application we would, again, check the source code out from the source code repository, build the application, and deploy it to a web server. The assembly test would then access the URL under which the application is running and look for some marks in the HTML which verify that the application is running OK, like a welcome message. The test would also verify that there are no error messages in the server/application logs.

Unit tests [2] look at components in isolation and verify that a component works according to its specifications while integration tests [4] look at how multiple components work together in a (sub)system. How do assembly tests fit into that taxonomy?

Unit tests look at isolated components, they explicitly exclude dependencies (like databases, DAOs, services or other components) from testing and substitute those dependencies with mock objects [3] if they have to. Benefits of unit tests include speedy execution (only the component under test is loaded and run), robustness against change (changes to other components don’t affect the test as they are mocked) and expressiveness (one can infer the component’s behaviour by reading the test). An assembly test runs the entire system, or at least large parts of it and it doesn’t even name the component(s) it is testing. It doesn’t look at specific functions (does the credit card validation work?), instead it looks at the coarsest possible scale (light on, light off, no smoke). Not to mention that an assembly test will most likely run much slower than a component test.

Integration tests run multiple components together and verify their communication for correctness. Assembly tests seem closer to integration tests since they both run larger parts of the entire application, yet assembly tests don’t look at specific functions, nor do they pin-point which components exactly failed in case of an error. In my view, assembly tests are really, really sloppy integration tests, but they do serve a purpose.

My pitch for assembly tests

The key argument for assembly tests is that they implement a reasonable end-to-end regression test at a ridiculously low cost. The consulting firm where I first learned about assembly tests conducted an internal study which showed that assembly tests caught 30% of all errors – that is an awesome return for just looking at a light bulb. While assembly tests don’t verify a functional or technical aspect completely, they at least touch many if not all of them in a single run:

✓ Database found and schema (probably) correct
✓ Binary dependencies deployed and compatible
✓ No incompatibilities because of OS or Webserver updates
✓ HTTP is working
✓ External services found and responsive
✓ Logfiles are being written to
✓ No startup errors (e.g. NullPointerExceptions)
✓ No important components missing
✓ Bootstrap configuration OK

Assembly tests are cheap to write because they don’t do much

The moment your continuous integration server builds the application and deploys it to a test environment, all the assembly test has to do is invoke any non-trivial function of the running application, verify that it responded plausibly and that there are no errors in the log files. Unit tests are typically a lot of work: detailed input data must be constructed, depending on test coverage ambitions that input data has to come in variations, dependencies have to be mocked and when you are done, you have tested nothing more than a single component.

Assembly tests are cheap because there is only one per application

Assembly tests don’t need to be updated with every new feature or bug fix because they don’t care about precise results. As a matter of fact, because there is only one function (light on) there rarely would be more than one assembly test per application anyway.

Assembly test are cheap to maintain because they don’t require you to think about them

Since assembly tests verify only the most elementary functionality (light on, light off, no smoke) they don’t need to be part of the developer’s daily work: they are written only once per application, they are not updated like unit tests and generally they don’t show up anywhere until the application breaks.

Assembly tests verify the build chain

In these modern days of dynamically typed languages and run-time linking, a successful compilation doesn’t say much about whether the application will run or crash. A successful assembly tests most effectively verifies that the build chain (compiler, dependency resolution, configuration, deployment, target runtime environment) worked, which isn’t really part of unit testing and often is not verified by integration tests either.

Assembly tests verify the configuration

Again, in these modern days of dependency injection frameworks there is a lot that can go wrong and go unnoticed by other means of testing: spelling mistakes in URLs to Javascript libraries, syntax errors in bootstrapping XML files, wrong database credentials and much more. Unit tests would not catch these at all and most integration tests run in simplified (and simulated) environments where this configuration doesn’t apply at all. Since the assembly test runs in a real environment it includes that environment’s configuration in the functional verification.

Who should use assembly tests?

Assembly tests are for the lazy and the desperate. If you consider unit tests too synthetic and a waste of time, if integration tests are too hard to write or too unstable, if you are stuck with one of those projects that don’t have budget for QA, do yourself this one favour, write that one assembly test. It will catch many of those hasty mistakes, syntax errors, dependency mismatches, environment incompatibilities and merge mistakes which any tester would immediately find but require you to leave the IDE and log on to a browser.