chore(opossum reporter): Migrate serialization to kotlinx #9484

alexzurbonsen · 2024-11-21T18:16:29Z

[FOR REVIEW]: Best reviewed commit by commit. First commit of this PR contains the actual changes. Second commit just moves code around to improve readability.

Context

Closes #8839

Summary

Migrates the OpossumReporter plugin away from Jackson serialization to use kotlinx serialization.

To get properly typed data structures for serialization the previous untyped Map<*, *>s are substituted by properly typed data classes. This leads to the creation of an OpossumInputCreator class that holds intermediate data structures needed for processing of the Reporter input.

Test Plan

Automated tests:

OpossumReporterTest
OpossumReporterFunTest

Manual test: Compare two opossum files with ORT builds from main and this branch

repeat for each branch:

./gradlew installDist
cli/build/install/ort/bin/ort analyze -i <INPUT_PROJECT> -o ./RESULT -f JSON 
cli/build/install/ort/bin/ort report -i /path/to/analyzer-result.json -o ./RESULT -f Opossum

Unzip report.opossum files and validate the input.json s with this little python script:
https://gist.github.com/alexzurbonsen/69e38586dcffbb6b988b010bbd096759

sschuberth · 2024-11-22T19:09:06Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

+            fun create(
+                source: String,
+                id: Identifier? = null,
+                url: String? = null,
+                license: SpdxExpression? = null,
+                copyright: String? = null,
+                comment: String? = null,
+                preSelected: Boolean = false,
+                followUp: Boolean = false,
+                excludeFromNotice: Boolean = false
+            ): OpossumSignal {


You may suppress this via @Suppress("LongParameterList").

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

-        }
-
-        return opossumInput
+    internal fun createOpossumInput(input: ReporterInput, maxDepth: Int = Int.MAX_VALUE): OpossumInput {


codecov · 2024-11-21T19:13:02Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.64%. Comparing base (9103ac2) to head (91b1609).
Report is 9 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9484      +/-   ##
============================================
- Coverage     67.93%   67.64%   -0.30%     
  Complexity     1289     1289              
============================================
  Files           249      249              
  Lines          8792     8830      +38     
  Branches        913      926      +13     
============================================
  Hits           5973     5973              
- Misses         2433     2471      +38     
  Partials        386      386

Flag	Coverage Δ
funTest-non-docker	`33.33% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

Migrates the OpossumReporter plugin away from Jackson serialization to use kotlinx serialization. To get properly typed data structures for serialization the previous untyped Map<*, *> are substituted by properly typed data classes. This leads to the creation of an OpossumInputCreator class that holds intermediate data structures needed for processing of the Reporter input. Signed-off-by: alexzurbonsen <[email protected]>

Put public method at the top, put caller above callees etc. Signed-off-by: alexzurbonsen <[email protected]>

alexzurbonsen · 2024-11-22T18:13:41Z

plugins/reporters/opossum/src/test/kotlin/OpossumReporterTest.kt

-            opossumInput.packageToRoot.values.forAll { levelForPath ->
-                levelForPath.keys.forAll { path ->
-                    fileList shouldContain resolvePath(path)
-                }
-            }


Deleting this because otherwise I would have to write a separate test for OpossumInputCreator (and make the property packageToRoot non private, I guess).

But it seems like this test does not add much value. The same file paths are in resourcesToAttributions, via addSignal and later processing of pathToSignals. And we have a test for paths in resourcesToAttributions against the same list.

alexzurbonsen · 2024-11-22T18:18:19Z

Hi @sschuberth, it is ready for review now. Let me know what you think. Removed the untyped maps.

Serialization of the recursive data structure OpossumResources was a bit tricky and I decided to not support deserialization. I assume there is no use case for it anyways.

alexzurbonsen · 2024-11-22T18:22:56Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

@@ -563,3 +622,31 @@ private fun DependencyNode.getDependencies(): List<DependencyNode> =
            this += dependencyNodes.map { it.getStableReference() }
        }
    }
+
+private object UUIDSerializer : KSerializer<UUID> {


There wasn't a UUID serializer yet and only few other usages of UUIDs. So I put it here as a private class for now. But let me know if you want to see this somewhere else.

Again Ks3 comes to the rescue 😉 It already provides a UUID serializer:

https://github.com/Kantis/ks3/blob/f2ec0f5811c73ce140718b355ec61ddf751eaec5/doc/jdk.md?plain=1#L42

sschuberth · 2024-11-22T18:48:38Z

plugins/reporters/opossum/build.gradle.kts

@@ -20,6 +20,9 @@
 plugins {
    // Apply precompiled plugins.
    id("ort-plugin-conventions")
+


Commit message nits:

We prefer imperative mood, i.e. "Migrates" should say "Migrate".

We usually start capitalized after ":" in the title.

sschuberth · 2024-11-22T19:06:43Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

-        val followUp: Boolean = false,
-        val excludeFromNotice: Boolean = false,
-        val uuid: UUID = UUID.randomUUID()
+        @Transient


Is this a left-over from not having an UUID serializer yet?

sschuberth · 2024-11-22T19:09:06Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

+            fun create(
+                source: String,
+                id: Identifier? = null,
+                url: String? = null,
+                license: SpdxExpression? = null,
+                copyright: String? = null,
+                comment: String? = null,
+                preSelected: Boolean = false,
+                followUp: Boolean = false,
+                excludeFromNotice: Boolean = false
+            ): OpossumSignal {


You may suppress this via @Suppress("LongParameterList").

sschuberth · 2024-11-22T19:10:02Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

+                followUp: Boolean = false,
+                excludeFromNotice: Boolean = false
+            ): OpossumSignal {
+                return OpossumSignal(


Nit: You could turn this into a function expression via = instead of return.

sschuberth · 2024-11-22T19:10:27Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

+                    preSelected = preSelected,
+                    followUp = OpossumFollowUp.FOLLOW_UP.takeIf { followUp },
+                    excludeFromNotice = excludeFromNotice,
+                    comment = comment,


Nit: Please avoid trailing commas.

Same in other places.

sschuberth · 2024-11-22T19:23:48Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

+        val json = Json {
+            explicitNulls = false
+            encodeDefaults = true
+        }


This should probably be made available as an internal top-level property for other places like tests to use. When doing that, please name the property JSON.

sschuberth · 2024-11-22T19:34:19Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

+            explicitNulls = false
+            encodeDefaults = true
+        }
+        jsonFile.writeText(json.encodeToString(OpossumInput.serializer(), opossumInput))


If files may get big, maybe better use a streaming approach like at

ort/plugins/reporters/aosd/src/main/kotlin/Aosd20Model.kt

Line 198 in 76c7958

internal fun File.writeReport(model: AOSD20): File = apply { outputStream().use { JSON.encodeToStream(model, it) } }

sschuberth · 2024-11-22T20:01:42Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

-
-        return opossumInput
+    internal fun createOpossumInput(input: ReporterInput, maxDepth: Int = Int.MAX_VALUE): OpossumInput {
+        return OpossumInputCreator().create(input, maxDepth)


This could as well be a function expression.

sschuberth · 2024-11-22T20:05:36Z

plugins/reporters/opossum/src/main/kotlin/OpossumReporter.kt

@@ -563,3 +622,31 @@ private fun DependencyNode.getDependencies(): List<DependencyNode> =
            this += dependencyNodes.map { it.getStableReference() }
        }
    }
+
+private object UUIDSerializer : KSerializer<UUID> {


Again Ks3 comes to the rescue 😉 It already provides a UUID serializer:

https://github.com/Kantis/ks3/blob/f2ec0f5811c73ce140718b355ec61ddf751eaec5/doc/jdk.md?plain=1#L42

sschuberth · 2024-11-22T20:07:57Z

plugins/reporters/opossum/src/test/kotlin/OpossumReporterTest.kt

@@ -213,18 +201,18 @@ class OpossumReporterTest : WordSpec({

        "create issues containing all issues" {
            val issuesFromFirstPackage =
-                opossumInput.getSignalsForFile("/pom.xml/compile/first-package-group/[email protected]")
+                opossumInput.getSignalsForFile("/pom.xml/compile/first-package-group/[email protected]/")


I guess the trailing slash is not needed and could be removed again to reduce the diff?

alexzurbonsen force-pushed the migrate-opossum-reporter-to-kotlinx-serialization branch 3 times, most recently from 7b1a948 to b3aefdb Compare November 21, 2024 18:26

github-advanced-security bot found potential problems Nov 21, 2024

View reviewed changes

alexzurbonsen force-pushed the migrate-opossum-reporter-to-kotlinx-serialization branch from b3aefdb to 98ee401 Compare November 21, 2024 19:50

alexzurbonsen changed the title ~~chore(opossum reporter): migrate serialization to kotlinx~~ chore(opossum reporter): Migrate serialization to kotlinx Nov 21, 2024

alexzurbonsen force-pushed the migrate-opossum-reporter-to-kotlinx-serialization branch from 98ee401 to 91b1609 Compare November 22, 2024 16:56

alexzurbonsen added 2 commits November 22, 2024 19:04

refactor(opossum reporter): adjust file structure

4bf27e2

Put public method at the top, put caller above callees etc. Signed-off-by: alexzurbonsen <[email protected]>

alexzurbonsen force-pushed the migrate-opossum-reporter-to-kotlinx-serialization branch from 91b1609 to 4bf27e2 Compare November 22, 2024 18:04

alexzurbonsen commented Nov 22, 2024

View reviewed changes

alexzurbonsen marked this pull request as ready for review November 22, 2024 18:16

alexzurbonsen requested a review from a team as a code owner November 22, 2024 18:16

alexzurbonsen commented Nov 22, 2024

View reviewed changes

sschuberth requested changes Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(opossum reporter): Migrate serialization to kotlinx #9484

chore(opossum reporter): Migrate serialization to kotlinx #9484

alexzurbonsen commented Nov 21, 2024 •

edited

Loading

sschuberth Nov 22, 2024

codecov bot commented Nov 21, 2024 •

edited

Loading

alexzurbonsen Nov 22, 2024

alexzurbonsen commented Nov 22, 2024

alexzurbonsen Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

sschuberth Nov 22, 2024

chore(opossum reporter): Migrate serialization to kotlinx #9484

Are you sure you want to change the base?

chore(opossum reporter): Migrate serialization to kotlinx #9484

Conversation

alexzurbonsen commented Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Nov 21, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

alexzurbonsen commented Nov 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexzurbonsen commented Nov 21, 2024 •

edited

Loading

codecov bot commented Nov 21, 2024 •

edited

Loading