Ensuring JUnit test suite contains all tests

While cleaning up today one of my projects I found a JUnit test that was not part of a test suite and never was executed. ‘Never again’ I say and here is how: a test suite that scans base packages for tests.

The particular code:

  • Is tailored to the specific project’s needs 
  • Compares the tests which are declared in the test suite with the ones actually present and fails with a detailed report should they not match
  • Uses the Reflections [1] library to scan sub packages for executable tests which extend a given base class

Areas for improvement:

  • Tests must currently extend a base class (here: BaseTest) as I have not found a way for Reflections to discover tests in the sub packages when they are not annotated or do not extend a base class
  • When test suites are discovered their tests should also somehow be handled

@Suite.SuiteClasses({ Test1.class, Test2.class, Test3.class })
public class TestSuite {

static String basePackage = TestSuite.class.getPackage().getName();

private static Set<Class> getDeclaredTests() {
Annotation[] annotations = TestSuite.class.getAnnotations();
for (Annotation annotation : annotations) {
if (!annotation.annotationType().equals(SuiteClasses.class))
Suite.SuiteClasses testClasses = (Suite.SuiteClasses) annotation;
Set<Class> cset = new HashSet<Class>();
for (Class c : testClasses.value())
return cset;
return null;

private static boolean isAbstract(Class c){
return Modifier.isAbstract(c.getModifiers());

private static Set<Class> getActualTests() {

Reflections reflections = new Reflections(basePackage, MethodAnnotationsScanner.class);
Set<Class> tests = (Set)reflections.getSubTypesOf(BaseTest.class);
for (Iterator<Class> ite = tests.iterator();ite.hasNext();){
Class c = ite.next();
if (isAbstract(c))
return tests;

private static List<Class> getMissingTests(Set<Class> realTests, Set<Class> declaredTests) {
List<Class> missingTests = new ArrayList<Class>();
for (Class test : realTests)
if (!isAbstract(test) && !declaredTests.contains(test))
return missingTests;

public static void checkThatAllTestsAreIncluded() {
Set<Class> declaredTests = getDeclaredTests();
Set<Class> realTests = getActualTests();
List<Class> missingTests = getMissingTests(realTests, declaredTests);
String failMessage = "";
if (!missingTests.isEmpty()) {
String report = "";
for (Class test : missingTests) {
report += "\n" + test.getCanonicalName();
failMessage += "Found tests that are not included in the suite: "
+ report;
if (realTests.size() < declaredTests.size()){
failMessage += "\nFound real tests : " + realTests.size()
+ " while " + declaredTests.size() + " were declared. Missing tests:";
for (Class c:declaredTests)
if (!realTests.contains(c))
if (!StringUtils.isEmpty(failMessage))


[1] Reflections library

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s