Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: DH-18477: Java coverage generated conditionally #6586

Merged
Prev Previous commit
Next Next commit
Reworked coverage merge into its own project
stanbrub committed Jan 24, 2025
commit f4605deeac5076bd667584e89c1cac28e5e268e4
Original file line number Diff line number Diff line change
@@ -7,21 +7,17 @@ jacoco {
toolVersion = '0.8.12'
}

test {
jacoco {
destinationFile = layout.buildDirectory.file('jacoco/jacoco.exec').get().asFile
}
finalizedBy jacocoTestReport
}

jacocoTestReport {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to look into hooking up integration tests as part of the coverage path. Right now, it seems like only the "test" task is included for the reporting. https://docs.gradle.org/current/userguide/jacoco_plugin.html#sec:jacoco_tasks

Copy link
Contributor Author

@stanbrub stanbrub Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the README.md I have the expected command line for running everything. Are there more than check, testSerial, testParallel, testOutOfBand? How do I run anything that's missing?

dependsOn test
sourceSets sourceSets.main
executionData = fileTree(buildDir).include("/jacoco/*.exec")
reports {
csv.required = true
csv.destination = layout.buildDirectory.file('reports/jacoco/java-coverage.csv').get().asFile
xml.required = true
xml.outputLocation = layout.buildDirectory.file('reports/jacoco/java-coverage.xml').get().asFile
html.outputLocation = layout.buildDirectory.dir('reports/jacoco/html')
csv.required = true
xml.required = true
html.required = true
}
}

task coverage {
dependsOn jacocoTestReport
}

Original file line number Diff line number Diff line change
@@ -5,7 +5,6 @@ plugins {
// Apply Jacoco instrumentation and coverage reporting
if (project.findProperty('coverage.enabled') == 'true') {
project.apply plugin: 'io.deephaven.java-jacoco-conventions'
rootProject.apply plugin: 'io.deephaven.coverage-merge'
}

def testJar = project.tasks.register 'testJar', Jar, { Jar jar ->
47 changes: 47 additions & 0 deletions coverage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Overview

Code coverage for Deephaven Community Core manages multiple languages like Java, Python, R, Go and C++. This is handled in the gradle build at the individual project level but also supports collection of normalized results rolled up to the top level. For convenience, both top-level Java HTML and a top-level all-language CSV are created.

## Running for Coverage

A typical run looks like the following that is run from the root of the multi-project build
```
./gradlew -Pcoverage.enabled=true check
./gradlew -Pcoverage.enabled=true testSerial
./gradlew -Pcoverage.enabled=true testParallel
./gradlew -Pcoverage.enabled=true testOutOfBand
./gradlew -Pcoverage.enabled=true coverage
./gradlew -Pcoverage.enabled=true coverage-merge
```
Running the second command is not contingent upon the first command succeeding. It merely collects what coverage is available.

## Result Files

Results for individual project coverage are stored in the project's _build_ output directory. Depending on the language and coverage tools, there will be different result files with slightly different locations and names. For example, Java coverage could produce a binary _jacoco.exec_ file, while python coverage produces a tabbed text file.

Aggregated results produce a merged CSV file for each language under the top-level _build_ directory. Those CSV files are further merged into one _all-coverage.csv_.

## Exclusion Filters

In some cases, there may be a need to exclude some packages from coverage, even though they may be used during testing. For example, some Java classes used in GRPC are generated. The expectation is that the generator mechanism has already been tested and should produce viable classes. Including coverage for those classes in the results as zero coverage causes unnecessary noise and makes it harder to track coverage overall.

To avoid unneeded coverage, the file _exclude-packages.txt_ can be used. This is a list of values to be excluded if they match the "Package" column in the coverage CSV. These are exact values and not wildcards.

## File Layout

Top-level Build Directory (Some languages TBD)
- `coverage/` This project's directory
- `gather-coverage.py` Gather and normalize coverage for all languages
- `exclude-packages.txt` A list of packages to exclude from aggregated results
- `buildSrc/src/main/groovy/`
- `io.deephaven.java-jacoco-conventions.gradle` Applied to run coverage on Java projects
- `io.deephaven.java-test-conventions.gradle` Applies the above conditionally base on the _coverage.enabled_ property
- `coverage/build/reports/coverage/`
- `java-coverage.csv` Normalized coverage from all Java projects
- `python-coverage.py` Normalized coverage from all Python projects
- `cplus-coverage.py` Normalized coverage from all C++ projects
- `r-coverage.py` Normalized coverage from all R projects
- `go-coverage.oy` Normalized coverage from all Go projects
- `all-coverage.csv` Normalized and filtered coverage from all covered projects
- `coverage/build/reports/jacoco/jacoco-merge/html/`
- `index.html` Root file to view Java coverage down to the branch level (not filtered)
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
plugins {
id 'base'
id 'io.deephaven.project.register'
id 'java'
id 'jacoco-report-aggregation'
}

jacoco {
toolVersion = '0.8.12'
}

tasks.register("coverage-merge", JacocoReport) {
def jprojects = allprojects.findAll { p-> p.plugins.hasPlugin('java') }
tasks.register("jacoco-merge", JacocoReport) {
def jprojects = rootProject.allprojects.findAll { p-> p.plugins.hasPlugin('java') }
additionalSourceDirs = files(jprojects.sourceSets.main.allSource.srcDirs)
sourceDirectories = files(jprojects.sourceSets.main.allSource.srcDirs)
classDirectories = files(jprojects.sourceSets.main.output)
@@ -17,6 +18,15 @@ tasks.register("coverage-merge", JacocoReport) {
csv.required = true
xml.required = false
}
def projRootDir = project.rootDir.absolutePath
def projRootDir = rootProject.rootDir.absolutePath
executionData fileTree(projRootDir).include("**/build/jacoco/*.exec")
}

tasks.register("coverage-merge", Exec) {
dependsOn("jacoco-merge")
def projDir = projectDir.absolutePath
def script = projDir + '/gather-coverage.py'
commandLine 'python', script, projDir
standardOutput = System.out
}

5 changes: 5 additions & 0 deletions coverage/exclude-packages.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
io.deephaven.tuple.generated
io.deephaven.engine.table.impl.tuplesource.generated
io.deephaven.proto.backplane.grpc
io.deephaven.proto.backplane.script.grpc
io.deephaven.proto
46 changes: 46 additions & 0 deletions coverage/gather-coverage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#
# Copyright (c) 2016-2025 Deephaven Data Labs and Patent Pending
#
import sys, glob, csv, os, shutil

# Aggregate coverage data for all languages. Each language has a different way of doing
# coverage and each normalization mechanism is used here. Class/file exclusions are
# handled here, since coverage tools are inconsistent or non-functional in that regard.

proj_root_dir = sys.argv[1]
script_dir = os.path.dirname(os.path.abspath(__file__))
coverage_dir = proj_root_dir + '/build/reports/coverage'
coverage_output_path = coverage_dir + '/all-coverage.csv'
coverage_input_glob = coverage_dir + '/*-coverage.csv'
exclude_path = script_dir + '/exclude-packages.txt'

if os.path.exists(coverage_dir):
shutil.rmtree(coverage_dir)
os.makedirs(coverage_dir)

# Aggregate and normalize coverage for java projects
input_glob = proj_root_dir + '/build/reports/jacoco/jacoco-merge/jacoco-merge.csv'
with open(f'{coverage_dir}/java-coverage.csv', 'w', newline='') as outfile:
csv_writer = csv.writer(outfile)
csv_writer.writerow(['Language','Project','Package','Class','Missed','Covered'])
for filename in glob.glob(input_glob, recursive = True):
with open(filename, 'r') as csv_in:
csv_reader = csv.reader(csv_in)
next(csv_reader, None)
for row in csv_reader:
new_row = ['java',row[0],row[1],row[2],row[3],row[4]]
csv_writer.writerow(new_row)

# Load packages to be excluded from the aggregated coverage CSV
with open(exclude_path) as f:
excludes = [line.strip() for line in f]

# Collect coverage CSVs into a single CSV without lines containing exclusions
with open(coverage_output_path, 'w', newline='') as outfile:
csv_writer = csv.writer(outfile)
for csv_file in glob.glob(coverage_input_glob):
with open(csv_file, 'r') as csv_in:
for row in csv.reader(csv_in):
if row[2] in excludes: continue
new_row = [row[0],row[1],row[2],row[3],row[4],row[5]]
csv_writer.writerow(new_row)
1 change: 1 addition & 0 deletions coverage/gradle.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
io.deephaven.project.ProjectType=BASIC
1 change: 1 addition & 0 deletions settings.gradle
Original file line number Diff line number Diff line change
@@ -51,6 +51,7 @@ project(':configs').projectDir = file('props/configs')
include(':test-configs')
project(':test-configs').projectDir = file('props/test-configs')

include 'coverage'
include 'combined-javadoc'

include 'grpc-java:grpc-servlet-jakarta'