In bundle: saker.java.testing
Executes tests for Java classes.
The task will execute the specified tests in a separate Java Virtual Machine using the given test runner.
The task provides incremental support for testing, meaning that only tests which have their inputs change will be reinvoked next time. The incremental support includes modifying Java classes, accessed files, and class loader resources. Some aspects of incremental support may not be complete, e.g. reflection based Java class tracking.
This task is best suited for unit testing of Java classes, and assumes that test cases don't share a common state.
The task takes 3 class path inputs that are loaded for testing:
1. The user ClassPath. This contains the classes that are being tested.
2. The TestClassPath. This contains the test cases that are being run.
3. The TestRunnerClassPath. This contains the classes that are used to invoke the tests in TestClassPath.
The class paths are loaded in a way that the TestClassPath sees classes from both the user ClassPath and TestRunnerClassPath,while the user ClassPath is loaded in an isolated way.
The invoker API of the saker.java.testing bundle is available for the TestRunnerClassPath.
Summary
| AbortOnFail | Specifies if the task should be considered as failed if a failed test is encountered. |
| ClassPath | Required parameter. Specifies the class path of the classes being tested.
|
| FailFast | Specifies if the testing should continue after a failed test has been encountered. |
| JavaSDK | Specifies the Java installation that should be used to execute the tests. |
| MaxJVMCount | Specifies how many JVM processes may be spawned at maximum to run the tests. |
| NonDeterministicTests | Wildcard specifications of the non-deterministic Java test case classes. |
| ProcessJVMParameters | Additional command line parameters to be passed to the spawned JVM that is used to run the test cases. |
| SuccessExitCodes | Specifies the exit codes that should be considered as a result of a successful test case. |
| TestClassParameters | Specifies arbitrary key-value pairs that are passed to the TestInvokerClass for each test case. |
| TestClassPath | Specifies the class path which contains the test case classes. |
| TestClasses | Wildcard specifications of the Java test case classes. |
| TestInvokerClass | Specifies the Java class name of the test invoker that is used to run the tests. |
| TestInvokerParameters | Arbitrary key-value pairs that are passed to the TestInvokerClass during initialization. |
| TestRunnerClassPath | Specifies the class path which contains the classes that are used to run the test cases. |
| Verbose | Sets the output of the testing task to be verbose. |
| WorkingDirectory | Specifies the working directory that the tests should use. |
Parameters
Specifies if the task should be considered as failed if a failed test is encountered.
If set to true, then the task will report an error, and will cause the build execution to fail if a test case fails.
If set to false, then a failed test will not cause the build execution to fail.
The default is true.
Collection of JavaClassPathTaskOption
Required parameter.
Specifies the class path of the classes being tested.
These are the classes which are accessible from the TestClassPath and are the subject of testing.Specifies if the testing should continue after a failed test has been encountered.
If set to true, the testing will end if a failed task is encountered. If set to false, all tests will be run.
The default is false.
Specifies the Java installation that should be used to execute the tests.
If not specified, the same installed Java Runtime Environment is used as the one used to run the build.
Specifies how many JVM processes may be spawned at maximum to run the tests.
Collection of String
Wildcard specifications of the non-deterministic Java test case classes.
If a non-deterministic test case fails, it will be reinvoked next time the task is run. By default, all test cases are considered to be deterministic. Meaning that if they fail, they won't be re-run if none of its dependencies have changed. Specifying a test case to be non-deterministic will cause the task to re-run tests even if their dependencies haven't changed.
If you encounter any incremental errors or unexpected outputs for failed test-cases, specifying "**" will result in all failed tests to be reinvoked.
Collection of String
Additional command line parameters to be passed to the spawned JVM that is used to run the test cases.
The specified parameters are directly appended to the started Java process right after the executable argument, and before any task related arguments.
This parameter is useful for example to start a test process with debugging enabled. Using "-agentlib:jdwp=transport=dt_socket,address=localhost:5432,server=y,suspend=y,quiet=y" will start the process by listening on the 5432 port for a debugger after being started. We recommend connecting to the test process using your Java IDE of choice. When you specify this argument make sure to add "quite=y", else the task may not initialize the process successfully.
Some parameters may be filtered out based on the supported options of the java executable. E.g. --enable-preview won't be passed to the test process if it is below the Java major version 11.
Collection of int
Specifies the exit codes that should be considered as a result of a successful test case.
If a test runner decides to call System.exit() as the result of the test execution, it will be considered as a failed test. However, if the exit code is one of the values specified in this parameter, the test case will be considered as successful.
By default, the exit code of 0 is considered to be a successful test.
Specifies arbitrary key-value pairs that are passed to the TestInvokerClass for each test case.
Each entry in the specified map is a class name wildcard mapped to the string key-value pairs that are passed for any test case which class name matches the associated wildcard.
The parameters for the test cases can be used to configure the behaviour of the test runner for the invoked test case.
If the default TestInvokerClass is used, the following test case parameter(s) can be used:
PrefixArguments: Specifies parameters for the test runner class that are inserted before the test class name.
SuffixArguments: Specifies parameters for the test runner class that are inserted after the test class name.
To summarize, for each test case class name, the TestRunnerClass.main(String[]) method is called with the arguments constructed as { <PrefixArguments...>, <test-case-class-name>, <SuffixArguments...> }.
Collection of JavaClassPathTaskOption
Specifies the class path which contains the test case classes.
Each class that matches the specified TestClasses will be passed to the test invoker to execute the test.
The test classes are passed to the invoker one-by-one, instead of in bulk. The test invoker class specified in TestInvokerClass will be used to invoke the test cases.
Collection of String
Wildcard specifications of the Java test case classes.
The parameter accepts wildcard patterns matching Java class binary names. The patterns are to be specified in a dot ('.') separated format, and they work the same way as wildcard paths.
E.g. to specify all classes that end with "Test" to be used as test cases: "**.*Test"
All matching classes will be passed to the TestInvokerClass for testing.
Specifies the Java class name of the test invoker that is used to run the tests.
The test invoker is loaded from the TestRunnerClassPath, and must implement saker.java.testing.api.test.invoker.JavaTestInvoker. The specified test cases will be passed to the test invoker to be run.
The test invoker can be configured using TestInvokerParameters.
By default, a test invoker is used that calls a main method of the specified TestRunnerClass test invoker parameter with the test case name passed as an argument. See TestInvokerParameters and TestClassParameters for the available parameters for the default test invoker.
Arbitrary key-value pairs that are passed to the TestInvokerClass during initialization.
The specified entries can be used to configure the test invoker. These parameters are passed to the invoker at the start of testing, and don't change for each test case.
If the default TestInvokerClass is used, the following parameter(s) can be used:
TestRunnerClass: Specifies the class that runs the test. It must have a static main(String[]) method which is called for each test case.
Collection of JavaClassPathTaskOption
Specifies the class path which contains the classes that are used to run the test cases.
This class path contains the classes that invoke the tests. The specified TestInvokerClass will be loaded from it, and it is used to call the test cases. This class path usually contains the testing library like JUnit or similar libraries.
Sets the output of the testing task to be verbose.
Verbose output includes more detailed information about the test cases that are being executed.
Specifies the working directory that the tests should use.
If tests use relative paths, they will be resolved against the specified working directory. The path is an execution path, not a path on the local file system.