Ensuring JUnit test suite contains all tests

While cleaning up today one of my projects I found a JUnit test that was not part of a test suite and never was executed. ‘Never again’ I say and here is how: a test suite that scans base packages for tests.

The particular code:

• Is tailored to the specific project’s needs 
• Compares the tests which are declared in the test suite with the ones actually present and fails with a detailed report should they not match
• Uses the Reflections [1] library to scan sub packages for executable tests which extend a given base class

Areas for improvement:

• Tests must currently extend a base class (here: BaseTest) as I have not found a way for Reflections to discover tests in the sub packages when they are not annotated or do not extend a base class
• When test suites are discovered their tests should also somehow be handled

@RunWith(Suite.class)
@Suite.SuiteClasses({ Test1.class, Test2.class, Test3.class })
public class TestSuite {
 
    static String basePackage = TestSuite.class.getPackage().getName();
 
    private static Set<Class> getDeclaredTests() {
        Annotation[] annotations = TestSuite.class.getAnnotations();
        for (Annotation annotation : annotations) {
            if (!annotation.annotationType().equals(SuiteClasses.class))
                continue;
            Suite.SuiteClasses testClasses = (Suite.SuiteClasses) annotation;
            Set<Class> cset = new HashSet<Class>();
            for (Class c : testClasses.value())
                cset.add(c);
            return cset;
        }
        return null;
    }
     
    private static boolean isAbstract(Class c){
        return Modifier.isAbstract(c.getModifiers());
    }
 
    private static Set<Class> getActualTests() { 

        Reflections reflections = new Reflections(basePackage, MethodAnnotationsScanner.class);
        Set<Class> tests = (Set)reflections.getSubTypesOf(BaseTest.class);
        for (Iterator<Class> ite = tests.iterator();ite.hasNext();){
            Class c = ite.next();
            if (isAbstract(c))
                ite.remove();
        }
        return tests;
    }
 
    private static List<Class> getMissingTests(Set<Class> realTests, Set<Class> declaredTests) {
        List<Class> missingTests = new ArrayList<Class>();
        for (Class test : realTests)
            if (!isAbstract(test) && !declaredTests.contains(test))
                missingTests.add(test);
        return missingTests;
    }
 
    @BeforeClass
    public static void checkThatAllTestsAreIncluded() {
        Set<Class> declaredTests = getDeclaredTests();
        Set<Class> realTests = getActualTests();
        List<Class> missingTests = getMissingTests(realTests, declaredTests);
        String failMessage = "";
        if (!missingTests.isEmpty()) {
            String report = "";
            for (Class test : missingTests) {
                report += "\n" + test.getCanonicalName();
            }
            failMessage += "Found tests that are not included in the suite: "
                    + report;
        }
        if (realTests.size() < declaredTests.size()){
            failMessage += "\nFound real tests : " + realTests.size()
                    + " while " + declaredTests.size() + " were declared. Missing tests:";
            for (Class c:declaredTests)
                if (!realTests.contains(c))
                    failMessage+="\n"+c.getCanonicalName();
        }
        if (!StringUtils.isEmpty(failMessage))
            fail(failMessage);
    }
} 

Resources

[1] Reflections library
http://code.google.com/p/reflections/