feat: Use lockfile scalibr interface (#1330)

This PR contains all the code required to move to osv-scalibr while making the existing code compile and pass all tests (container tests not passing because of a bug in the scalibr alpine extractor). Changes not mentioned in the following list will be split off in separate PRs which should land before this PR. Those are: - [x] #1337 - [x] #1331 - [x] #1338 - [x] #1341 - [x] #1345 Changes in this PR: - Fixture changes: - Scalibr Python requirements.txt extractor currently doesn't support packages without versions, so added some version strings to the test files - Image package required quite a bit of reworking to successfully update. - Add the ability to iterate through a directory via the pathtree library - Support scalibr FS interface for Layers - Add conversion code to convert inventories from osv-scalibr back to v1's lockfile and Inventory - This is done to minimize snapshot changes. Followup PRs should remove this conversion - Add `internal/lockfilescalibr` package: - `errors.go` adds common extraction errors we want to translate. - `translation.go` adds helper functions and translation logic between osv-scanner v1 extractor names, and osv-scalibr extractor names. Changes in followup PRs: - Delete lockfiles package and migrate everything to use osv-scalibr extractors - Remove conversion code in image --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Gareth Jones <[email protected]> Co-authored-by: Xueqin Cui <[email protected]> Co-authored-by: Michael Kedar <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
google · Nov 1, 2024 · b15b566 · b15b566
1 parent 1638434
commit b15b566
Show file tree

Hide file tree

Showing 32 changed files with 660 additions and 231 deletions.
diff --git a/cmd/osv-scanner/__snapshots__/main_test.snap b/cmd/osv-scanner/__snapshots__/main_test.snap
@@ -349,9 +349,9 @@ overriding license for package Packagist/league/flysystem/1.0.8 with 0BSD
 | LICENSE VIOLATION | ECOSYSTEM | PACKAGE                                        | VERSION | SOURCE                                                |
 +-------------------+-----------+------------------------------------------------+---------+-------------------------------------------------------+
 | 0BSD              | Packagist | league/flysystem                               | 1.0.8   | fixtures/locks-insecure/composer.lock                 |
-| UNKNOWN           |           | https://github.com/flutter/buildroot.git       |         | fixtures/locks-insecure/osv-scanner-flutter-deps.json |
-| UNKNOWN           |           | https://github.com/brendan-duncan/archive.git  |         | fixtures/locks-insecure/osv-scanner-flutter-deps.json |
 | UNKNOWN           |           | https://chromium.googlesource.com/chromium/src |         | fixtures/locks-insecure/osv-scanner-flutter-deps.json |
+| UNKNOWN           |           | https://github.com/brendan-duncan/archive.git  |         | fixtures/locks-insecure/osv-scanner-flutter-deps.json |
+| UNKNOWN           |           | https://github.com/flutter/buildroot.git       |         | fixtures/locks-insecure/osv-scanner-flutter-deps.json |
 | UNKNOWN           | RubyGems  | ast                                            | 2.4.2   | fixtures/locks-many/Gemfile.lock                      |
 | 0BSD              | Packagist | sentry/sdk                                     | 2.0.4   | fixtures/locks-many/composer.lock                     |
 +-------------------+-----------+------------------------------------------------+---------+-------------------------------------------------------+
@@ -1856,6 +1856,7 @@ Filtered 16 vulnerabilities from output
 | https://osv.dev/DLA-3325-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/DLA-3449-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/DLA-3530-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
+| https://osv.dev/DLA-3942-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/DSA-4539-3          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/CVE-2017-12837      | 7.5  | Debian    | perl                           | 5.24.1-3+deb9u7                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/CVE-2017-12883      | 9.1  | Debian    | perl                           | 5.24.1-3+deb9u7                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
@@ -2034,6 +2035,7 @@ Filtered 16 vulnerabilities from output
 | https://osv.dev/DLA-3325-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/DLA-3449-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/DLA-3530-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
+| https://osv.dev/DLA-3942-1          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/DSA-4539-3          |      | Debian    | openssl                        | 1.1.0l-1~deb9u5                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/CVE-2017-12837      | 7.5  | Debian    | perl                           | 5.24.1-3+deb9u7                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
 | https://osv.dev/CVE-2017-12883      | 9.1  | Debian    | perl                           | 5.24.1-3+deb9u7                    | fixtures/sbom-insecure/postgres-stretch.cdx.xml |
@@ -2328,7 +2330,7 @@ No issues found
 ---
 
 [TestRun_LockfileWithExplicitParseAs/empty_works_as_an_escape_(no_fixture_because_it's_not_valid_on_Windows) - 2]
-open <rootdir>/path/to/my:file: no such file or directory
+stat <rootdir>/path/to/my:file: no such file or directory
 
 ---
 
@@ -2337,7 +2339,7 @@ open <rootdir>/path/to/my:file: no such file or directory
 ---
 
 [TestRun_LockfileWithExplicitParseAs/empty_works_as_an_escape_(no_fixture_because_it's_not_valid_on_Windows)#01 - 2]
-open <rootdir>/path/to/my:project/package-lock.json: no such file or directory
+stat <rootdir>/path/to/my:project/package-lock.json: no such file or directory
 
 ---
 
@@ -2346,7 +2348,7 @@ open <rootdir>/path/to/my:project/package-lock.json: no such file or directory
 ---
 
 [TestRun_LockfileWithExplicitParseAs/files_that_error_on_parsing_stop_parsable_files_from_being_checked - 2]
-(extracting as Cargo.lock) could not extract from <rootdir>/fixtures/locks-insecure/my-package-lock.json: toml: line 1: expected '.' or '=', but got '{' instead
+(extracting as rust/Cargolock) could not extract from <rootdir>/fixtures/locks-insecure/my-package-lock.json: toml: line 1: expected '.' or '=', but got '{' instead
 
 ---
 
@@ -2404,7 +2406,7 @@ No issues found
 ---
 
 [TestRun_LockfileWithExplicitParseAs/parse-as_takes_priority,_even_if_it's_wrong - 2]
-(extracting as package-lock.json) could not extract from <rootdir>/fixtures/locks-many/yarn.lock: invalid character '#' looking for beginning of value
+(extracting as javascript/packagelockjson) could not extract from "<rootdir>/fixtures/locks-many/yarn.lock": invalid character '#' looking for beginning of value
 
 ---
 
@@ -2586,17 +2588,17 @@ Scanning image ../../internal/image/fixtures/test-node_modules-npm-empty.tar
 
 [TestRun_OCIImage/scanning_node_modules_using_npm_with_some_packages - 1]
 Scanning image ../../internal/image/fixtures/test-node_modules-npm-full.tar
-+-------------------------------------+------+--------------+----------+------------+-------------------------------------------------------------------------------------------------------+
-| OSV URL                             | CVSS | ECOSYSTEM    | PACKAGE  | VERSION    | SOURCE                                                                                                |
-+-------------------------------------+------+--------------+----------+------------+-------------------------------------------------------------------------------------------------------+
-| https://osv.dev/CVE-2023-42363      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                    |
-| https://osv.dev/CVE-2023-42364      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                    |
-| https://osv.dev/CVE-2023-42365      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                    |
-| https://osv.dev/CVE-2023-42366      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                    |
-| https://osv.dev/GHSA-38f5-ghc2-fcmv | 9.8  | npm          | cryo     | 0.0.6      | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/usr/app/node_modules/.package-lock.json |
-| https://osv.dev/GHSA-vh95-rmgr-6w4m | 9.8  | npm          | minimist | 0.0.8      | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/usr/app/node_modules/.package-lock.json |
-| https://osv.dev/GHSA-xvch-5gv4-984h |      |              |          |            |                                                                                                       |
-+-------------------------------------+------+--------------+----------+------------+-------------------------------------------------------------------------------------------------------+
++-------------------------------------+------+--------------+----------+------------+--------------------------------------------------------------------------------------------------------+
+| OSV URL                             | CVSS | ECOSYSTEM    | PACKAGE  | VERSION    | SOURCE                                                                                                 |
++-------------------------------------+------+--------------+----------+------------+--------------------------------------------------------------------------------------------------------+
+| https://osv.dev/CVE-2023-42363      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                     |
+| https://osv.dev/CVE-2023-42364      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                     |
+| https://osv.dev/CVE-2023-42365      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                     |
+| https://osv.dev/CVE-2023-42366      | 5.5  | Alpine:v3.19 | busybox  | 1.36.1-r15 | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/lib/apk/db/installed                     |
+| https://osv.dev/GHSA-38f5-ghc2-fcmv | 9.8  | npm          | cryo     | 0.0.6      | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/prod/app/node_modules/.package-lock.json |
+| https://osv.dev/GHSA-vh95-rmgr-6w4m | 9.8  | npm          | minimist | 0.0.8      | ../../internal/image/fixtures/test-node_modules-npm-full.tar:/prod/app/node_modules/.package-lock.json |
+| https://osv.dev/GHSA-xvch-5gv4-984h |      |              |          |            |                                                                                                        |
++-------------------------------------+------+--------------+----------+------------+--------------------------------------------------------------------------------------------------------+
 
 ---
 

diff --git a/cmd/osv-scanner/fixtures/locks-requirements/my-requirements.txt b/cmd/osv-scanner/fixtures/locks-requirements/my-requirements.txt
@@ -1 +1 @@
-flask
+flask==1.0.0
diff --git a/cmd/osv-scanner/fixtures/locks-requirements/requirements-dev.txt b/cmd/osv-scanner/fixtures/locks-requirements/requirements-dev.txt
@@ -1 +1 @@
-black
+black==1.0.0
diff --git a/cmd/osv-scanner/fixtures/locks-requirements/requirements.txt b/cmd/osv-scanner/fixtures/locks-requirements/requirements.txt
@@ -1,3 +1,3 @@
-flask
-flask-cors
+flask==1.0.0
+flask-cors==1.0.0
 pandas==0.23.4
diff --git a/cmd/osv-scanner/fixtures/locks-requirements/the_requirements_for_test.txt b/cmd/osv-scanner/fixtures/locks-requirements/the_requirements_for_test.txt
@@ -1 +1 @@
-pytest
+pytest==1.0.0
diff --git a/cmd/osv-scanner/main_test.go b/cmd/osv-scanner/main_test.go
@@ -518,7 +518,12 @@ func TestRun_LockfileWithExplicitParseAs(t *testing.T) {
 		t.Run(tt.name, func(t *testing.T) {
 			t.Parallel()
 
-			testCli(t, tt)
+			stdout, stderr := runCli(t, tt)
+
+			testutility.NewSnapshot().MatchText(t, stdout)
+			testutility.NewSnapshot().WithWindowsReplacements(map[string]string{
+				"CreateFile": "stat",
+			}).MatchText(t, stderr)
 		})
 	}
 }

diff --git a/go.mod b/go.mod
@@ -18,7 +18,7 @@ require (
 	github.com/go-git/go-git/v5 v5.12.0
 	github.com/google/go-cmp v0.6.0
 	github.com/google/go-containerregistry v0.20.2
-	github.com/google/osv-scalibr v0.1.4-0.20241016092100-7e7f0c6a01ec
+	github.com/google/osv-scalibr v0.1.4-0.20241031120023-761ca671aacb
 	github.com/ianlancetaylor/demangle v0.0.0-20240912202439-0a2b6291aafd
 	github.com/jedib0t/go-pretty/v6 v6.6.0
 	github.com/muesli/reflow v0.3.0

diff --git a/go.sum b/go.sum
@@ -109,8 +109,8 @@ github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
 github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/go-containerregistry v0.20.2 h1:B1wPJ1SN/S7pB+ZAimcciVD+r+yV/l/DSArMxlbwseo=
 github.com/google/go-containerregistry v0.20.2/go.mod h1:z38EKdKh4h7IP2gSfUUqEvalZBqs6AoLeWfUy34nQC8=
-github.com/google/osv-scalibr v0.1.4-0.20241016092100-7e7f0c6a01ec h1:pbByndoAmqND/Vkj3wYLS2aDAq+/2dll7rKzIM3ezCU=
-github.com/google/osv-scalibr v0.1.4-0.20241016092100-7e7f0c6a01ec/go.mod h1:MbEYB+PKqEGjwMdpcoO5DWpi0+57jYgYcw2jlRy8O9Q=
+github.com/google/osv-scalibr v0.1.4-0.20241031120023-761ca671aacb h1:A7IvUJk8r3wMuuAMWxwbkE3WBp+oF/v7CcEt3nCy+lI=
+github.com/google/osv-scalibr v0.1.4-0.20241031120023-761ca671aacb/go.mod h1:MbEYB+PKqEGjwMdpcoO5DWpi0+57jYgYcw2jlRy8O9Q=
 github.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=
 github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=

diff --git a/internal/image/__snapshots__/image_test.snap b/internal/image/__snapshots__/image_test.snap
@@ -4,7 +4,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -186,7 +186,7 @@
   "Lockfiles": [
     {
       "filePath": "/go/bin/more-vuln-overwrite-less-vuln",
-      "parsedAs": "go-binary",
+      "parsedAs": "go/binary",
       "packages": [
         {
           "name": "github.com/BurntSushi/toml",
@@ -214,7 +214,7 @@
     },
     {
       "filePath": "/go/bin/ptf-1.2.0",
-      "parsedAs": "go-binary",
+      "parsedAs": "go/binary",
       "packages": [
         {
           "name": "github.com/BurntSushi/toml",
@@ -242,7 +242,7 @@
     },
     {
       "filePath": "/go/bin/ptf-1.3.0",
-      "parsedAs": "go-binary",
+      "parsedAs": "go/binary",
       "packages": [
         {
           "name": "github.com/BurntSushi/toml",
@@ -270,7 +270,7 @@
     },
     {
       "filePath": "/go/bin/ptf-1.3.0-moved",
-      "parsedAs": "go-binary",
+      "parsedAs": "go/binary",
       "packages": [
         {
           "name": "github.com/BurntSushi/toml",
@@ -298,7 +298,7 @@
     },
     {
       "filePath": "/go/bin/ptf-1.4.0",
-      "parsedAs": "go-binary",
+      "parsedAs": "go/binary",
       "packages": [
         {
           "name": "github.com/BurntSushi/toml",
@@ -326,7 +326,7 @@
     },
     {
       "filePath": "/go/bin/ptf-vulnerable",
-      "parsedAs": "go-binary",
+      "parsedAs": "go/binary",
       "packages": [
         {
           "name": "github.com/BurntSushi/toml",
@@ -354,7 +354,7 @@
     },
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -536,7 +536,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -754,7 +754,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -963,8 +963,8 @@
       ]
     },
     {
-      "filePath": "/usr/app/node_modules/.package-lock.json",
-      "parsedAs": "node_modules",
+      "filePath": "/prod/app/node_modules/.package-lock.json",
+      "parsedAs": "javascript/nodemodules",
       "packages": [
         {
           "name": "cryo",
@@ -1011,7 +1011,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -1229,7 +1229,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -1447,7 +1447,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",
@@ -1665,7 +1665,7 @@
   "Lockfiles": [
     {
       "filePath": "/lib/apk/db/installed",
-      "parsedAs": "apk-installed",
+      "parsedAs": "os/apk",
       "packages": [
         {
           "name": "alpine-baselayout",