Assembly tests: a cheap way of testing
“Lights on, lights off, no smoke. Works.”
A delivery manager from a large consulting firm once told me about a peculiar breed of automated system tests his team was using to catch regressions with minimal effort. He called those tests “assembly tests”, essentially they start an entire application, roughly verify that it is somehow plausibly reacting to elementary input and shut it down again.
If it moves and it doesn’t explode then it works
Comparing to unit tests and integration tests
Unit tests  look at components in isolation and verify that a component works according to its specifications while integration tests  look at how multiple components work together in a (sub)system. How do assembly tests fit into that taxonomy?
Unit tests look at isolated components, they explicitly exclude dependencies (like databases, DAOs, services or other components) from testing and substitute those dependencies with mock objects  if they have to. Benefits of unit tests include speedy execution (only the component under test is loaded and run), robustness against change (changes to other components don’t affect the test as they are mocked) and expressiveness (one can infer the component’s behaviour by reading the test). An assembly test runs the entire system, or at least large parts of it and it doesn’t even name the component(s) it is testing. It doesn’t look at specific functions (does the credit card validation work?), instead it looks at the coarsest possible scale (light on, light off, no smoke). Not to mention that an assembly test will most likely run much slower than a component test.
Integration tests run multiple components together and verify their communication for correctness. Assembly tests seem closer to integration tests since they both run larger parts of the entire application, yet assembly tests don’t look at specific functions, nor do they pin-point which components exactly failed in case of an error. In my view, assembly tests are really, really sloppy integration tests, but they do serve a purpose.
My pitch for assembly tests
The key argument for assembly tests is that they implement a reasonable end-to-end regression test at a ridiculously low cost. The consulting firm where I first learned about assembly tests conducted an internal study which showed that assembly tests caught 30% of all errors – that is an awesome return for just looking at a light bulb. While assembly tests don’t verify a functional or technical aspect completely, they at least touch many if not all of them in a single run:
✓ Database found and schema (probably) correct
✓ Binary dependencies deployed and compatible
✓ No incompatibilities because of OS or Webserver updates
✓ HTTP is working
✓ External services found and responsive
✓ Logfiles are being written to
✓ No startup errors (e.g. NullPointerExceptions)
✓ No important components missing
✓ Bootstrap configuration OK
Assembly tests are cheap to write because they don’t do much
The moment your continuous integration server builds the application and deploys it to a test environment, all the assembly test has to do is invoke any non-trivial function of the running application, verify that it responded plausibly and that there are no errors in the log files. Unit tests are typically a lot of work: detailed input data must be constructed, depending on test coverage ambitions that input data has to come in variations, dependencies have to be mocked and when you are done, you have tested nothing more than a single component.
Assembly tests are cheap because there is only one per application
Assembly tests don’t need to be updated with every new feature or bug fix because they don’t care about precise results. As a matter of fact, because there is only one function (light on) there rarely would be more than one assembly test per application anyway.
Assembly test are cheap to maintain because they don’t require you to think about them
Since assembly tests verify only the most elementary functionality (light on, light off, no smoke) they don’t need to be part of the developer’s daily work: they are written only once per application, they are not updated like unit tests and generally they don’t show up anywhere until the application breaks.
Assembly tests verify the build chain
In these modern days of dynamically typed languages and run-time linking, a successful compilation doesn’t say much about whether the application will run or crash. A successful assembly tests most effectively verifies that the build chain (compiler, dependency resolution, configuration, deployment, target runtime environment) worked, which isn’t really part of unit testing and often is not verified by integration tests either.
Assembly tests verify the configuration
Who should use assembly tests?
Assembly tests are for the lazy and the desperate. If you consider unit tests too synthetic and a waste of time, if integration tests are too hard to write or too unstable, if you are stuck with one of those projects that don’t have budget for QA, do yourself this one favour, write that one assembly test. It will catch many of those hasty mistakes, syntax errors, dependency mismatches, environment incompatibilities and merge mistakes which any tester would immediately find but require you to leave the IDE and log on to a browser.
 Assembly tests
 Unit tests
 Mock objects
 Integration tests
 Smoke tests