Want faster Ant build times? One of the most common culprits I've seen for slow builds is the use of
batchtest under the junit task while it has fork="yes"
set.
People end up in this state for a variety of reasons. The batchtest example in
the Ant docs has fork="yes" set. So does the example in the cruise
control docs. Perhaps they started not forking a VM, but got clashes with XML (or other)
libraries, and that was the easiest fix. Whatever the reason, I've seen lots of people do it,
and there is a better way.
Lets start with a typical example. Here's what your Ant test target might look like to start off with:
<target name="test">
<junit fork="yes">
<classpath refid="test.classpath"/>
<formatter type="plain" usefile="false"/>
<batchtest>
<fileset dir="${test.classes.dir}">
<include name="**/*Test.class"/>
<exclude name="**/acceptance/**/*Test.class"/>
</fileset>
</batchtest>
</junit>
</target>
The basic approach is to convert this Ant script which runs lots of tests into an Ant script
that runs only one test. Several things need to happen though. Firstly, the virtual machine
still needs to be forked, to make sure classpath issues are not re-introduced. Secondly, the
test has to become a suite so that all of the tests will still be run. Finally, some sort of
filter mechanism needs to be put into the suite that matches the fileset filtering mechanism
in the batchtest
The Ant script now becomes a bit simpler. The batchtest section is gone, and
a single test name is in its place: com.thoughtworks.todolist.AllTests.
<target name="test">
<junit fork="yes">
<classpath refid="test.classpath"/>
<formatter type="plain" usefile="false"/>
<test name="com.thoughtworks.todolist.AllTests"/>
</junit>
</target>
To get the test suite to work as per batchtest, I use a utility from the
JUnit Addons project. It
dynamically builds up a test suite from a directory structure with custom filters to
include or exclude classes. My test suite looks like this:
package com.thoughtworks.todolist;
import junit.framework.Test;
import junitx.util.DirectorySuiteBuilder;
import junitx.util.SimpleTestFilter;
public class AllTests {
public static Test suite() throws Exception {
DirectorySuiteBuilder builder = new DirectorySuiteBuilder();
builder.setFilter(new SimpleTestFilter() {
public boolean include(Class clazz) {
return super.include(clazz) &&
!getPackageName(clazz).equals("com.thoughtworks.todolist.acceptance");
}
});
return builder.suite("build/test/classes");
}
}
The use of the DirectorySuiteBuilder class replaces the batchtest
component of the Ant build script, and the use of the SimpleTestFilter replaces
the need for the nested fileset element.
That's all there is to it. In my simple application that so far has a grand total of 11 unit tests, I managed to cut my build time from 4 seconds down to 2 seconds. Phew! Pretty quick huh? Well - maybe in my case it didn't matter so much, but I've seen this cut minutes of builds for "real" projects before. Let me know how it goes for you.
Oh yeah - as an added bonus, you now have a stand-alone JUnit test suite that will always be up to date with all of your unit tests. It has no dependancy on Ant, so you can run it by itself inside your IDE (or whatever environment you use for JUnit tests).
Comments
I would love to use a Suite for my tests, but the formatting of output is still my reason for not switching from batchtests. The Ant Junit task mishandles output of Suites, as I wrote about a few weeks ago:
http://uncommentedbytes.blogspot.com/2004/06/ant-junit-task-mishandles-suites.html
There will be a new fork option in Ant 1.6.2 to speed up the batchtests also.
Posted by: Jeff Sheets | July 13, 2004 11:38 PM
Two points that may be of interest:
1) In Ant 1.6.2 (currently in beta), there is a "forkmode" attribute to the test runners. forkmode="once" will mean only one fork, not one per test class.
2) "Real IDEs" already have good ways of running all your unit tests. In eclipse, I simply select the root of the test source tree and tell it to run. :)
Posted by: Robert Watkins | July 13, 2004 11:40 PM
I like how automated this solution is. I have been creating a PackageNameSuite in each package and then adding the TestClasses.suite() from each class in that package. Then I add the PackageNameSuite of any sub-packages.
Posted by: Ozten | July 14, 2004 02:12 AM
I love this project. This has brought the build time on my current project down from > 10 minutes to < 4 minutes.
I agree that the only downside is the JUnit Report. However, it's not really a downside as I see it. I don't really care which package my failing test is in, as long as the JUnit failure message tells me.
At least until the new version of JUnit, this is a great option.
Posted by: Geoff Oliphant | July 14, 2004 04:49 AM
Robert - I did know about the fork once option coming up. I look forward to seeing it. I disagree about the "Real IDE" thing though. They do know how to run all of your JUnit tests, but have no way of knowing which ones are your /unit/ tests (and not functional or integration tests).
Posted by: Marty Andrews | July 14, 2004 08:41 AM
However, it's not really a downside as I see it. I don't really care which package my failing test is in, as long as the JUnit failure message tells me
Heh.
When you've got a couple of thousand tests all on one page, the scroll wheel gets a little overused :)
Posted by: Tim Vernum | July 14, 2004 11:57 AM
I am hoping ya'll might have some feedback for me on a similar issue. We are using Junit and JFCUnit to put in place test automation. I want all the tests to run every night with the nightly build. BUT, I am adding a menu into our Swing application that will list the various functional areas that can be tested. The primary purpose is to allow developers before checking in changes, to sync up with the source control thus grabbing anyone else's changes, resolve any merge conflicts, then run the functional tests they wish (at the UI level) to verify that nothing broke. They could just as easily wait for the next morning, but they can use this process to run through various tests in an area they may be working on without having to wait until morning. Especially useful if they come in in the morning, see tests that have failed, make some quick fixes and want to see if those tests will now pass. Rather than go through the full build cycle, especially as we add more and more tests that are going to take a lot longer than a few minutes to run, I figured adding it directly into the app with the use of jfcUnit, Marathon, etc, can allow much faster small functional area tests.
Now, my dilema. I am going to make a simple main class that reads in an xml config file. The ant build script calls this class and defaults to running all tests. The application menu system would also call this class but pass a selected list of tests to run. Thus, the same code base is used to run the tests, only the ant script is running them all, where as the UI menu I add can allow a developer to choose a selection of tests to run, from a group of them, to a sub-group within a group, to a specific test. So... does this seem alright? Or is there a potential better solution that does something similar?
Another thing, I have broken tests into a separate dir (below /src) so that all junit/jfcUnit/etc test classes are NOT embeded within the main application src. I find it easier to maintain, anyone else agree with this logic? Lastly, because I am writing automation tests based on functional testing, but the developers write unit tests (mostly non-UI, but some UI as well for component level testing), I broke up the package into a com.company.test.unit. and com.company.test.functional. package structure. I think this works because it allows faster navigation to failed tests for unit tests, but also allows better maintainability of functional tests which are not package specific tests as they may cover tests that drive the app through a rigourous amount of stuff. Anyone like this package structure? The unit tests below the test.unit. conform to the same package structure as the classes they test in the primary source base.
Thanks, look forward to some thoughts.
Posted by: Kevin | July 14, 2004 04:09 PM
Kevin,
I don't understand why you are adding a menu item to your swing app to run functional tests against itself. That seems odd to me.
I do like breaking tests into functional groups, which may address the same problem you were trying to with the menu. See the point on packages below.
I think having tests in a separate source tree to application code is a good idea. You can ensure your application code is compiled and deployed independanty of any test classes (lots of other good reasons omitted).
I don't like your package structure. I would rather have tests in the same packages as the corresponding application code they are testing. I usually then use a class naming scheme to differentiate between unit and functional tests.
So if I had a class com.wrytradesman.app.Customer in my application source tree, then I might have a class com.wrytradesman.app.CustomerTest (which is a unit test) and a class com.wrytradesman.app.CustomerFunctionalTest in my test source tree.
The matching packages makes it easy to navigate (especially in modern IDE's) between the class you are writing and its corresponding tests.
Posted by: Marty Andrews | July 14, 2004 04:37 PM
Marty, you could always organise your tests into multple source trees if that's causing you grief.
e.g....
src/
test/unit
test/int
test/accept
Posted by: Robert Watkins | July 14, 2004 08:44 PM
Marty,
Thanks. I have a question for you on package naming. A functional test, at least in our app, will test MANY areas at one time, which could cover several components using 10's of classes. So in my mind, this makes more sense separating it into a .functional (or .acceptance might be better). I don't know how a test that opens a project, clicks a few things, changes some text, submits it, saves then closes the project can be put in a package/class that matches a single class it tests. That isn't the case as I see it. So, that said, does it still make sense to try to do it the way you describe?
As for adding a menu to the app, there are a few reasons. I want to add record/playback script capability, which jfcUnit, Abbot and soon Marathon all support. By being able to run the app, hit record and start clicking/typing, then stop recording, I can save it out to a script. My goal, however is not to use scripts per se, mostly because I think they are much more brittle than coded tests. But to take these xml output files and change them into code, which wont be terribly hard. That way, we can rapidly record a script, have it translated to code, then modify it as needed. Ahh, but you say, "What happens when you need to re-record that same script because something it tests changed..". Well, the thing is, once it is in code, we generally would keep it in code and modify the code a wee bit as changes occur to the UI. I don't know that this is 100% fullproof, just one of many thoughts I have.
The other reason, as I said before is to use the SAME tests that I write/use to test/regression, to allow developers to run small areas of tests before checking in code to ensure that there stuff works. If it's something new, they obviously have to manual test it. But if it is a fix for something that already exists and a test is there, they can just run through the tests. More so, each morning after a nightly buid, they'll see if anything failed. If something did and it's in there area, they can do a fix, and not have to wait for the nightly build to run ALL the tests. They can simply run through the specific test(s) that failed and see if it works. That way, they aren't greeted the next day with another failure. The idea here is that rather than wait each day to see what is broken, if you know, fix it and run the same test that ran and failed in the morning.
Sure, they could use Ant, and try to modify the script to run specific tests, but why not have a nice GUI that allows them faster selection of specific tests. By breaking our test into functional area, sub-category, then test_001 - test_xxx, they can quickly navigate to a specific test, category of functional area to run tests for.
Posted by: Kevin | July 15, 2004 01:42 AM
Robert, you're right, but it's not causing me grief. In fact, I find it works quite well. More info in the next comment for Kevin...
Posted by: Marty Andrews | July 15, 2004 04:05 PM
Kevin,
I think we have some naming differences going on.
Unit tests for me usually test a specific class. Putting them in a .unit package seems dumb, as it's common to have hundreds of them. Making them match the application package structure seems much more sensible.
Functional tests for me sometimes test groups of classes, and sometimes test the application via its interface (eg. http). In either case, I ususally have some package structure for the tests that has some meaning for what's going on. Sometimes I do have a .acceptance package (or similar) package as a parent package for interface driving tests.
Posted by: Marty Andrews | July 15, 2004 04:27 PM
Marty,
Ok, you may be right about unit. I read something today that made a lot of sense, but I dislike to a degree. By keeping unit tests in the same location as the classes they tets, they get compiled with the class they test and forces you thus to update the unit test class if you change the class it tests, or it wont compile (depending..). However, I prefer to keep them separated.
Anyway, the problem I have with NOT using .unit in its package name is that I have /src/test/com/company/test/functional/xxx and /test/unit/xxx. If I get rid of the unit, I still have a package with functional in it and not entirely sure it makes a lot of sense. I am not opposed to removing it, but I kind of like the unit and functional (or acceptance as I think I may change it to) name in the package structure. It does make it simple to see any test class I am working on belongs to a unit test or a functional test without any further ado. Don't you agree?
Posted by: Kevin | July 15, 2004 04:39 PM
I don't understand how keeping a test with the class forces you to update the test when the class changes.
I agree with your idea about immediately seeing if something is a unit test or functional test. As I said before, thats exactly why I use the test naming scheme. Packages name don't have to solve that problem...
Posted by: Marty Andrews | July 15, 2004 05:02 PM
Marty,
If you keep tests with the files they test, and you need to modify a method signature within the class, the test wouldn't even compile. The train of thought here is that usually when you place tests in another folder as I do (and I think you do), you compile the application source separately, and you may not compile tests for months (no idea why anyone would not do this every day as well...but I am assuming this is what the article author was thinking). Thus, the tests are NOT updated until the next time you compile the tests, by which time the main class(es) may have changed quite a bit. Thus, the tests are out of date. From a TDD view, this would never happen, you would modify the test first, watch it fail, then modify the class to make it pass regardless of where the test source is. But we all know that a lot of companies don't operate this way. I am very thankful that my new job is bent like hell on quality! My job, in fact is to employ UI automation tests to test the crap out of our UI product. While my title is QA, I am actually mostly writing java code to do so.
As for the package/class name, again ;), it is easy enough to just use XxxTest and know its a unit test I suppose, and for functional tests use the .acceptance in its package name or even the end of the file could be XxxAcceptanceTest, and XxxUnitTest. I don't know, I'll see what some of the other developers/qa think as well. In the grand scheme of things, the name is very easy to change. We are using ClearCase so repackaging is not too bad like it is in CVS, but with Subversion quickly gaining momentum, the whole package refactoring thing may be a past problem.
Posted by: kevin | July 16, 2004 01:00 AM