Performance comparison

In this article, we'll be showing the results of performance measurements including Java testing. We'll be comparing the testing performance of the saker.java.testing package against the Gradle build tool.

If you're intereseted in Java compilation performance, see here.

All the comparisons are done on the same and single machine. The comparions are done on Windows 10, with the Windows Defender being turned off all times. When doing the comparisons, we'll be using the build daemons of each participating tool when applicable.

During the measurements, we'll be doing some warm-up round(s) for each tools which won't be part of the result. Then we compute the average of the measured times and that will be the result of a measurement. We don't use any benchmarking/profiling tools, and don't set any options that instrument the tools for benchmarking.

We'll use the Measure-Command PowerShell cmdlet to measure the time it takes to execute a build. (We use the time the cmdlet displays instead that the given tool displays to have a consistent measuring practice.) Note that when the tools are used inside an IDE, the startup cost of the Java processes may be omitted.

When clean builds are measured, the previous outputs are manually cleaned from the output directory rather than running clean commands for a given tool.

Large monolithic project

We measure the testing performance on the single-large-project that is published by Gradle specifically for performance comparisons. It contains 50000 source files and 50000 associated test cases.

We've modified the project a little bit, in order to make the comparison fairer. We disabled report generation for Gradle, and increased the number of JVM forks for testing to match the settings we use for saker.build. The following was set for the build.gradle file:

test {
    reports {
        junitXml.enabled = false
		html.enabled = false
    }
	maxParallelForks = 4
}

Let's see the results:

Based on the results we can see that clean compilation and testing is performed 44% by saker.build. In case of incremental changes, the performance increase is around 90%! We can clearly see an advantage of the saker.java.test() task in that it tracks the dependencies of each invoked test case. In the incremental scenarios, it only needed to reinvoke the modified test case instead of invoking all 50000 of them.