Parallelizing unit test execution

As of framework version 11.0.244.0 it is possible to parallelize unit test execution. The following steps must be performed once to make use of this functionality going forward.

Setup

Database setup

Each parallel process requires a different database so the unittest database name is suffixed with the property ${test.clone}. Thus the expected database names for each parallel process are of the form: unittest, unittest1, unittest2... etc. To setup the additional databases, run the Database Schema Tool for each clone by clicking on Advanced in the tool and adding the following argument to the existing VM Arguments:

BASH

-Dnexj.server.property.override.test.clone=N



e.g.
1st: -Dnexj.server.property.override.test.clone=1
2nd: -Dnexj.server.property.override.test.clone=2

So for 4 clones the Database Schema Tool should be run 3 times with N being 1, 2 and 3 respectively to produce a setup script for each database. Once done, verify that you now have 3 databases: unittest, unittest1, unittest2, etc. Generated scripts should be executed before running the tool again, otherwise they will be overwritten.

Specify the number of parallel processes

The number of parallel processes is defined by the test.clones configuration property and defaults to 1. In Development.environment in finance this value is set to the property ${cloneCount} and expected to be defined in the associated property file. To override this without modifying version-controlled environment or property files the following VM argument can be added to the unit test tool in Eclipse: Window -> Preferences -> NexJ Studio -> Launch Settings -> Tool Settings -> Run Unit Tests -> VM arguments:

BASH

-Dnexj.server.property.override.cloneCount=4

The presence of this argument has no effect on environments where cloneCount is undefined so running unit tests on environments that have no provisions for parallelization simply execute sequentially in a single process instead of resulting in runtime failures. This property also has no effect on any other tools.

Please note that the optimal number of clones will depend on the hardware configuration of the machine executing the tests, e.g. on an 8GB machine and a JVM configured with a 2GB heap, the optimal number of clones will likely be 2 or 3. It will also likely be one less than the number of physical CPU cores on the system.

Exclude the finance integration tests

All tests will run nightly, however for development purposes the Integration tests (ie. Exchange) can be excluded. Within Studio's Unit Test view, select the Change Filter Mode option, Exclude Suite from Run All menu, and select SanityExclusion. This suite is to be considered Read Only for development purposes.

CODE

Exclude from Run All: SanityExclusion
Run All

Execution

Once the above steps are completed parallelization occurs automatically and unit tests selected and executed through studio are broken up and parallelized as much as possible. Individual test cases are not parallelized, so if a single unit test is selected no parallelization is attempted. The console output shows interleaved output from each clone but the lines are prefixed with [#N] for clone N to make it possible to filter the logs for a particular process. Please note that the optimal number of clones will depend on the hardware configuration of the machine executing the tests, e.g. on an 8GB machine and a JVM configured with a 2GB heap, the optimal number of clones will likely be 2 or 3.

Debugging

When debugging unit tests for now it is easiest to avoid parallelization, as only the main process will hit breakpoints by default. If necessary it is possible to debug tests running on the clones by changing the test.clone value in the property file and launching a debugger manually through studio for each clone. Work to seamlessly integrate parallelization with debugging is currently underway.

Feedback

We would be interested to get some empirical feedback from any developers that choose to use this feature, such as timings for running a suite of tests along with the amount of parallelization providing the best performance. Please include hardware information or any other JVM arguments used.