JUnit4, JUnit5 and Spock framework comparison

11 January 2018

This article has been published on DZone with editor revision so I recommend you to read it there.

Introduction and motivation

Recently I gave a talk in my local Java User Group about unit testing. Some of the content of the talk was about some popular libraries you can use in your Java project. I’ve reviewed JUnit4, JUnit5 and Spock framework. Many of the attendees were quite surprised with the differences. In this post, I will summarize the most commented: assert, parametrized tests and mocking.

I always like to demonstrate the concepts with examples and live coding so I chose a simple algorithm: Fibonacci number calculator. If you don’t know it, it’s just to generate numbers which are the sum of the two previous ones in the series: 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377.

I used the typical (and with very bad performance) implementation:

    private static int fibonacci(int n) {
        if (n <= 1) return n; else
            return fibonacci(n-1) + fibonacci(n-2);
    }

JUnit4

I started explaining JUnit4. It makes sense because it’s the most popular library and the base for many others. I started explaining assertTrue and then, more advances usages, including assertEquals.

    @Test
    public void improvedFibonacciTestSimple() {
        FibonacciWithJUnit4 f = new FibonacciWithJUnit4();
        assertEquals(f.fibonacci(4), 3);
    }

If it’s false, it would give you an error like:

java.lang.AssertionError:
Expected :3
Actual   :2

Not very spectacular but quite useful.

The next thing was to show how to write a parametrized test, a nice feature very useful for test algorithms. In JUnit4 is quite tricky. You need to create a Collection with the annotation @Parameters.

    @Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][]{
                {0, 0}, {1, 1}, {2, 1}, {3, 2}, {4, 3}, {5, 5}, {6, 8}
        });
    }

Then we create some local variables and a constructor:

    private int fInput;
    private int fExpected;

    public ParametrizedFibonacciJUnit4(int input, int expected) {
        fInput = input;
        fExpected = expected;
    }

and finally, we can use the assertEquals:

    @Test
    public void test() {
        FibonacciWithJUnit4 f = new FibonacciWithJUnit4();
        assertEquals(fExpected, f.fibonacci(fInput));
    }

It’s quite verbose and if the test fails, you would obtain a message which doesn’t indicate clearly the order or the parameters used (but your IDE probably will help on that):

java.lang.AssertionError:
Expected :0
Actual   :1

I didn’t explain mocking here because an external library as Mockito is usually required when you want to use mocking with JUnit4. Mockito is great but I didn’t have enough time to explain it.

JUnit5

Since September 2017, JUnit5 is considered stable and it should be your choice over JUnit4 for several reasons. It has better support for Java 8 and Lambdas, it’s compatible with JUnit4 (you can have both which it’s great to migrate in a progressive way) and it provides new runners and better integrations.

I repeated the same process. First, show an assertEquals:

    @Test
    public void bestFibonacciTestSimple() {
        FibonacciWithJUnit5 f = new FibonacciWithJUnit5();
        Assertions.assertEquals(f.fibonacci(4), 3);
    }

There are important advances in other assert, for instance the timeout, but for assertEqual with integers, it’s practically the same. Also the message is similar if there is an error:

org.opentest4j.AssertionFailedError:
Expected :3
Actual   :2

Where we can find important changes is in the parametrized test. First, we need to use the @ParametrizedTest annotation and we can specify a name using { and } to indicate important parameters as index as arguments.

    @ParameterizedTest(name = "run #{index} with [{arguments}]")

Now we can define our test. We start defining the entries to test function with the annotation @CsvSource. Each item will replace the parameters in the test function, in this case, input and expected.

    @CsvSource({"1, 1", "4, 3"})
        public void test2(int input , int expected) {
                FibonacciWithJUnit5 f = new FibonacciWithJUnit5();
             	Assertions.assertEquals(f.fibonacci(input), expected);
         }

This is lot better than the JUnit4 implementation. Also, if there is a fail in the test, we obtain a better message indicating the difference, the index causing the fail and the used parameters.

Finally, it’s the same for mocking as JUnit4: you normally use an external library so I didn’t explained it.

Spock framework

The last one was the Spock framework. It’s based in Apache Groovy. If you don’t know Groovy, it’s a language which you can use with the JVM and it interact very well with Java. It allows write less code in a more clear way. It’s very powerful. Some years ago we started to use it for "non critical" development: tests, dependency management, Continuous Integration, load testing and in any place where we need some configuration file avoiding XML, JSON or any format like that. We continue to develop in Java the core of our software and it isn’t a problem because both languages play very well together. If you know Java, you know Groovy…​ so we have the best of both worlds.

Write a test in Spock is quite different, it would be like this:

    def "Simple test"() {
        setup:
        BadFibonacci f = new BadFibonacci()

        expect:
        f.fibonacci(1) == 1
        assert f.fibonacci(4) == 3
    }

Basically, we can use def where we don’t care about the type. The name of the function can be defined between quotation marks, which allow us to use better naming for our tests. We have some special words as setup, when, expect, and, etc. to define our test in a more descriptive and structured way. And we have a power assert, which is part of the language itself, proving nice messages:

Condition not satisfied:

f.fibonacci(4) == 2
| |            |
| 3            false
BadFibonacci@23c30a20

Expected :2

Actual   :3

It provides all the information: the returned value (actual), the expected value, the function, the parameter, etc. assert is Groovy is really handy.

Now it’s the turn for the parametrized test. It would be something like this:

    def "parametrized test"() {
        setup:
        BadFibonacci f = new BadFibonacci()

        expect:
        f.fibonacci(index) == fibonacciNumber

        where:
        index | fibonacciNumber
        1     | 1
        2     | 1
        3     | 2
    }

After show this I heard some 'oooh' in the audience. The magic of this code is: you don’t need to explain it! There is a table in the where: section and the values in expect: are automagically replace it in each iteration. If there is a fail, the message is crystal clear:

Condition not satisfied:

f.fibonacci(index) == fibonacciNumber
| |         |      |  |
| 2         3      |  4
|                  false
BadFibonacci@437da279

Expected :4

Actual   :2

Then I’ve introduced very shortly mocks and stubs. A mock is a object you create in your test to avoid to use a real object. For example, you don’t want to do real web requests or print a page in your tests, so you can use a mock from an interface or another object.

    Subscriber subscriber = Mock()

    def "mock example"() {
        when:
        publisher.send("hello")

        then:
        1 * subscriber.receive("hello")

Basically, you create the subscriber interface as Mock and then you can invoke the methods. The 1 * is another nice feature of Spock, it specify how many messages you should receive. Cool, right?.

In some occasions, you need to define what return the methods of your mocks. For that, you can create a stub.

    def "stub example"() {
        setup:
        subscriber.receive(_) >> "ok"

        when:
        publisher.send("message1")

        then:
        subscriber.receive("message1") == 'ok'
    }

In this case, with the >> notation we are defining the method receive should return ok independently of the parameter (_ means any value). The test pass without any problem.

Conclusions

I don’t like to recommend one library or another: all of the them have their use cases. It’s pretty clear we have great options in Java and I just give some examples. Now it’s your turn to decide which it’s better for you. The only thing I can say: write test and master your library of choice, it would make you a better developer!

If you want to take a deeper look to the examples, you will find them in this GitHub repository. Enjoy!

Comments