I'll soon start writing a new test runner - a common test runner for the
JVM, codename CTR4J. It will provide a superset of JUnit's test runner's
features, while at the same time overcoming some of JUnit's limitations
and helping integration with development tools. I would like to hear
your opinions on this project.
MOTIVATION
JUnit's test runner - especially the org.junit.runner.RunWith annotation
- has been very successful in providing a common interface for running
tests from multiple testing frameworks. If a testing framework provides
a org.junit.runner.Runner implementation, every Java build tool and IDE
will automatically support the framework.
However with the rise of Scala testing frameworks (Specs, ScalaTest,
Specsy - the last one being written by me), some limitations of JUnit's
test runner are coming to the fore:
- JUnit makes an implicit assumption that all tests are known before any
test code is executed. For example IntelliJ IDEA calls
Runner.getDescription() before calling Runner.run() and has problems if
the descriptions change during test execution. But in the Scala testing
frameworks, in order to achieve a concise syntax, test declarations are
implemented as method calls in the class constructor, which means that
at least some of the test code must be executed before it is known that
what tests there are.
- JUnit doesn't know the concept of nested tests. It knows the concept
of suites containing suites, but not "test methods" which contain other
"test methods". But at least Specs and Specsy allow organizing tests
into unlimitedly nested closures. For them this limitation means that
IDEs are not able to show the right mental model of what is really
happening during test execution, which makes understanding test output
harder.
- There is an implicit assumption that test names are the same as method
names, and that the tests are implemented as methods. For example in
IntelliJ IDEA it's possible to navigate from the test results into test
methods. But with the Scala frameworks this is not possible, because the
test names to not correspond any method declaration (and a string search
is not 100% reliable because the test name could be generated
dynamically or there could be multiple tests with the same name; at
least Specsy allows duplicate names).
There are also some other limitations which affect all Java testing
frameworks:
- When a test prints something, IntelliJ IDEA collects what is printed
to stdout and stderr, so that when you select a test in the test
results, it will show only what that test printed. This is very useful
for debugging with println-statements. But because there is no
synchronization between the test runner and the IDE which reads the
stdout/err, IDEA doesn't always associate what a test printed with the
correct test, especially when the tests are very fast.
- No built-in support for executing the tests in parallel. Third-party
tools are needed for it.
Possibly because of being faced with the above limitations, at least
Specs, ScalaTest and TestNG have implemented their own test runners.
This in turn means that the IDEs need to support each test runner
individually, which results in varying levels of quality and features
between the integration with different test runners. For example, at
least some time ago, IDEA's Scala plugin did not support automatically
finding all Specs tests in the project and executing them.
There is also code duplication inside and between all IDEs, build tools
and CI servers because they have had to write code to keep track of the
test execution state, and that needs to be repeated for each test runner.
THE PROJECT
I'll soon start writing a test runner to solve the above mentioned
limitations and to ease the integration with build tools, IDEs, CI
servers etc. The license will be Apache License 2.0. I'm also thinking
of making this not only open source, but also "open development", by
screencasting similar to
http://jamesshore.com/Blog/Lets-Play/What the test runner will provide to testing frameworks, is a superset
of the features provided by these JUnit classes: RunWith, Runner,
Description, RunNotifier. The API will be a bit more generic and with
new abstractions, in order to support for example nested tests.
What the test runner will provide to IDEs, build tools and CI servers,
is a library for launching the tests in a new VM instance, monitoring
the test execution status, collecting the test results and what was
printed to stdout/err etc. The tools just need to configure the
classpath and choose which tests to run (e.g. using file name patters).
I will also provide a reference implementation of an UI for running the
tests, because running tests as part of the TDD cycle is too important a
feature for its usability to be left in the hands of programmers without
interaction design skills.
What the test runner will provide to users, is running tests from all
testing frameworks (which support this runner or JUnit's test runner) in
one suite, running them in parallel on multiple CPU cores (I'll tackle
parallelization over multiple machines as a separate project, probably
as a commercial tool), integration with every development tool on the
JVM (I intend to contact all tool vendors and gather requirements from
them to ease the integration) and reliability (backwards compatiblity is
very important to me, and I plan on making it possible for each tool
vendor to write integration tests, which will be run as part of the test
runner's development builds, to detect any breaking changes).
So, I would like to ask for your opinion on this project. Here are some
questions:
- Do you find this useful?
- What do you think would be a good name for the project? One suggestion
is CTR4J (actually named similar to SLF4J) which is googleable, but
perhaps not very pronounceable or memorable.
- What would be a good name for the annotation which corresponds
@RunWith? Its name needs to be different to be googleable and to make it
easy to annotate a class with both JUnit's and this test runner's
annotation.
- Any other thoughts?
--
Esko Luontola
www.orfjackal.net
Locked
