When I report an issue in bug tracking system about a nightly build's JUnit test error I usually get simple answer that it is supposed to be working because it is working on developers workstation. After that routine conversation starts and it turns out that tests were running from development environment under Eclipse. Here I usually have to explain again and again that's not the same running tests from development environment and in build.
The right way to make yourself sure your tests will work in most cases without errors in nightly build is to start tests the same way as nightly build does. It was not easy for JBoss Tools tests until we created experimental branch and switched to Maven Tycho project. That means it is fairly easy to run tests now. Basically you need to change current directory and execute maven install goal. If it runs in development environment and in maven your tests are good and in most cases it should be fine in nightly build. Problems begin if it runs in development environment but it doesn't in maven. In this scenario you need to debug tests running in Tycho somehow and fix it. Fortunately it can be done using Java remote debugging support.
First of all you need to be sure you have built your sources you're going to debug and there is no differences between .java and .class files. If you're going find problem from previous build just get right tagged version and build it before debugging session.
Then open pom.xml for your Eclipse Test Plug-in and add Java VM arguments like it shown below (actual port numbers, server names and other parameters may be different, it depends from your environment)
org.sonatype.tycho
maven-osgi-test-plugin
${tycho-version}
-Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y
This snipped configured for remote debugging in OpenJDK 6, if your is different you might need to read [1] and configure it right for your java version.
Configure java projects with sources you're going to debug in Eclipse and create Remote Java Application configuration in Eclipse Debug Configuration dialog. Fill 'host' and 'port' fields with the same values from pom.xml argLine element. Press Apply button to save your changes and start your test plug-in from terminal like
$mvn install
It will go through build process and finally you ll see something like
[INFO] Expected eclipse log file: /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work/data/.metadata/.log
[INFO] Command line:
/bin/sh -c cd /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test && /usr/lib/jvm/java-6-openjdk/jre/bin/java -Dosgi.noShutdown=false -Dosgi.os=linux -Dosgi.ws=gtk -Dosgi.arch=x86 -agentlib:jdwp=transport=dt_socket,address=8001,server=y,suspend=y -jar /home/eskimo/.m2/repository/p2/osgi/bundle/org.eclipse.equinox.launcher/1.0.201.R35x_v20090715/org.eclipse.equinox.launcher-1.0.201.R35x_v20090715.jar -data /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work/data -dev file:/home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/dev.properties -install /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work -configuration /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work/configuration -application org.codehaus.tycho.surefire.osgibooter.uitest -testproperties /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/surefire.properties
Listening for transport dt_socket at address: 8001
At this point build is waiting for you to attach remote debugger to the process because of using
suspend=y
in argLine element of your pom.xml file. So you need to return to eclipse and hit 'Debug' button in dialog opened before.
Build will continue at this point and stop on your break points so you can find out what is wrong with tests during nightly build.
[1] Java Platform Debugger Architecture - http://java.sun.com/javase/technologies/core/toolsapis/jpda/#Invocation
