How to get LauncherDiscoveryRequest to see pre-compiled tests?

[jramsley]

Hello,

Full disclosure, I am not much of a java developer so a lot of the concepts around JARs and classpaths have a cursory knowledge of.

We've been trying to update our test infrastructure to be more programmatic as well as handle a level of test sharding for integration tests that isn't default out-of-the-box with gradle Junit 5. Our goal is to be able to write a custom LauncherDiscoveryRequest to generate a TestPlan by pointing to some pre-compiled tests/JARs and then do some filtering + ordering based on that. However, we are never able to get any tests found.

1. I can see that I have a JAR of tests on my local file system, /Users/jramsley/stage/test/libs/tests.jar
2. I can inspect the JAR to see that there are compiled classes in them which look like our test classes

jramsley:stage jramsley$ jar tvf ./test/lib/tests.jar | grep Test
...
   870 Mon May 20 12:55:08 EDT 2024 com/company/restapi/web/controller/RestReportTest$17.class
   943 Mon May 20 12:55:08 EDT 2024 com/company/restapi/web/controller/RestPermissionsTest$1.class
  2724 Mon May 20 12:55:08 EDT 2024 com/company/restapi/web/controller/TranslatorTest.class
...

3. If I run a javap -v -classpath tests.jar <class> I can see that the @Test annotation is on the class

jramsley:stage jramsley$ javap -v -classpath ./test/lib/tests.jar com.company.restapi.web.controller.RestPermissionsTest
...
RuntimeVisibleAnnotations:
      0: #922()
        org.junit.jupiter.api.Test

  public void <test name>() throws java.lang.Exception;
    descriptor: ()V
...

4. If I point to this JAR in our custom test runner, the test plan discovers no tests

@TaskAction
void runTestDiscovery() {
    Path classPath = Path.of("/Users/jramsley/stage/test/libs/tests.jar");
    println(classPath);

    LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
        .selectors(
            selectClasspathRoots(Collections.singleton(classPath))
        )
        .build();

    try (LauncherSession session = LauncherFactory.openSession()) {
        TestPlan testPlan_ = session.getLauncher().discover(request);
        println('Print testPlan');
        println(testPlan_);
        println(testPlan_.containsTests());
        println(testPlan_.getRoots().size());
        println(testPlan_.getRoots());
        testPlan_.getRoots().forEach(root -> {
            println('print root');
            println(root);
            try {
                Integer printDescendantsSize = testPlan_.getDescendants(root).size();
                println('print descendants size');
                println(printDescendantsSize);

                println('print descendants');
                println(testPlan_.getDescendants(root));
                println("print children");
                println(testPlan_.getChildren(root));
            } catch (Exception e) {
                println('error getting descendants');
                println(e);
            }
        })
    }
    catch (Exception error) {
        println('There is an error');
        println(error);
    }
}

with output

> Task :java:test:discoveredTests
/Users/jramsley/stage/test/libs/tests.jar
Print testPlan
org.junit.platform.launcher.core.InternalTestPlan@3d8c6278
false
1
[TestIdentifier [uniqueId = [engine:junit-jupiter], parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER]]
print root
TestIdentifier [uniqueId = [engine:junit-jupiter], parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER]
print descendants size
0
print descendants
[]
print children
[]

I would expect this to scan the compiled test classes, find the annotations and add them as children to the root/containers. At minimum I would expect testPlan_.containsTests() to be returning true.

So at this point I am sort of at a loss how to get the selector for LauncherDiscoveryRequest to be able to scan and find these annotations. I've tried following Discovering Tests but unable to get the launcher to find any tests. Can anyone help point me in the right direction to debug this?

[mpkorstanje]

Have you verified that the test jar is on your class path with the expected class path root?

[jramsley]

That seems to be the issue. Looking at https://junit.org/junit5/docs/5.0.3/api/org/junit/platform/engine/discovery/ClasspathResourceSelector.html there is a note that the classpath has to be on the engine. I'm able to get this working outside of running through gradle, but in the context of gradle it looks like the jar is on the discovery JVM and not the engine JVM.

[marcphilipp]

@jramsley If you provide a reproducer, I'd be happy to take a look.