How to include/exclude tests given a list of test cases

[mumrah]

Over in apache/kafka#17725 I am attempting to separate a set of quarantined tests from the main test suite. The how/why can be seen in our design.

The issue I am facing with JUnit is how to exclude a set of test cases programatically. I have a file with a list of tests I do not want to run as part of a suite. The contents are class name + method name:

org.apache.kafka.clients.ApiVersionsTest#testFinalizedFeaturesUpdate
org.apache.kafka.clients.ApiVersionsTest#testMaxUsableProduceMagic
org.apache.kafka.clients.ApiVersionsTest#testMaxUsableProduceMagicWithRaftController

My current solution is to use a custom ExecutionCondition class to exclude these tests at runtime. However, this results in the tests being SKIPPED whereas I would prefer that they not be selected in the first place.

I can see a declarative approach with the suite annotations (@Suite, @SelectClass, @SelectMethod), but what I think I need is a programmatic approach.

Short of writing a new TestEngine, is it possible to plug in to JUnit to create a list of tests to include or exclude based on the class + method name? We are using Gradle as our JUnit entry point if that makes a difference.

Thanks!

[marcphilipp]

You can implement a PostDiscoveryFilter (and register it via ServiceLoader) that inspect the TestSource of each TestDescriptor and exclude it if it's a MethodSource for one of the methods in your file.

Something like this should work:

import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.List;

import org.junit.platform.engine.FilterResult;
import org.junit.platform.engine.TestDescriptor;
import org.junit.platform.engine.support.descriptor.MethodSource;

public class ExcludedMethodsFromFileFilter implements PostDiscoveryFilter {

	private final List<String> exclusions;

	public ExcludedMethodsFromFileFilter() {
		try {
			exclusions = Files.readAllLines(Paths.get("excluded-tests.txt"));
		}
		catch (IOException e) {
			throw new UncheckedIOException("Failed to read excluded tests file", e);
		}
	}

	@Override
	public FilterResult apply(TestDescriptor descriptor) {
		return descriptor.getSource()
				.filter(source -> source instanceof MethodSource)
				.map(source -> (MethodSource) source)
				.map(source -> String.format("%s#%s", source.getClassName(), source.getMethodName()))
				.filter(exclusions::contains)
				.map(it -> FilterResult.excluded("excluded via file"))
				.orElseGet(() -> FilterResult.included("included via file"));
	}
}

[mumrah]

Thanks, @marcphilipp! This is exactly what I needed.