Ensuring JUnit test suite contains all tests

While cleaning up today one of my projects I found a JUnit test that was not part of a test suite and never was executed. ‘Never again’ I say and here is how: a test suite that scans base packages for tests.

The particular code:

  • Is tailored to the specific project’s needs 
  • Compares the tests which are declared in the test suite with the ones actually present and fails with a detailed report should they not match
  • Uses the Reflections [1] library to scan sub packages for executable tests which extend a given base class

Areas for improvement:

  • Tests must currently extend a base class (here: BaseTest) as I have not found a way for Reflections to discover tests in the sub packages when they are not annotated or do not extend a base class
  • When test suites are discovered their tests should also somehow be handled
@Suite.SuiteClasses({ Test1.class, Test2.class, Test3.class })
public class TestSuite {
    static String basePackage = TestSuite.class.getPackage().getName();
    private static Set<Class> getDeclaredTests() {
        Annotation[] annotations = TestSuite.class.getAnnotations();
        for (Annotation annotation : annotations) {
            if (!annotation.annotationType().equals(SuiteClasses.class))
            Suite.SuiteClasses testClasses = (Suite.SuiteClasses) annotation;
            Set<Class> cset = new HashSet<Class>();
            for (Class c : testClasses.value())
            return cset;
        return null;
    private static boolean isAbstract(Class c){
        return Modifier.isAbstract(c.getModifiers());
    private static Set<Class> getActualTests() { 

        Reflections reflections = new Reflections(basePackage, MethodAnnotationsScanner.class);
        Set<Class> tests = (Set)reflections.getSubTypesOf(BaseTest.class);
        for (Iterator<Class> ite = tests.iterator();ite.hasNext();){
            Class c = ite.next();
            if (isAbstract(c))
        return tests;
    private static List<Class> getMissingTests(Set<Class> realTests, Set<Class> declaredTests) {
        List<Class> missingTests = new ArrayList<Class>();
        for (Class test : realTests)
            if (!isAbstract(test) && !declaredTests.contains(test))
        return missingTests;
    public static void checkThatAllTestsAreIncluded() {
        Set<Class> declaredTests = getDeclaredTests();
        Set<Class> realTests = getActualTests();
        List<Class> missingTests = getMissingTests(realTests, declaredTests);
        String failMessage = "";
        if (!missingTests.isEmpty()) {
            String report = "";
            for (Class test : missingTests) {
                report += "\n" + test.getCanonicalName();
            failMessage += "Found tests that are not included in the suite: "
                    + report;
        if (realTests.size() < declaredTests.size()){
            failMessage += "\nFound real tests : " + realTests.size()
                    + " while " + declaredTests.size() + " were declared. Missing tests:";
            for (Class c:declaredTests)
                if (!realTests.contains(c))
        if (!StringUtils.isEmpty(failMessage))


[1] Reflections library

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.