RPM Prefetching Integration Test Proposal

[brunoapimentel]

1- E2E test

A test to cover the prefetching of RPMs, injection of project-files (i.e. creating the repofiles) and installing the RPMs hermetically during the container build. After the image is built, the installed package needs to generate some type of output that can be validated, so we can assure it was properly installed. The generated SBOM also needs to be checked and its accuracy validated.

The test repository needs to have a lockfile containing a target RPM with several dependencies, as well as contain valid checksums, file size and corresponding source RPMs.

This test will only target builds on a x86_64 arch. This way, it can still be run in the Github CI.

2- Missing checksums

Covers the reporting of missing checksums in the SBOM. The test repository needs to contain a very simple RPMs lockfile with some missing checksums.

This test will only validate the prefetch step, which means that no image build is required. The SBOM needs to be verified to make sure the RPMs with missing checksums were properly reported.

3- Invalid checksum

Validates that an incorrect checksum in the lockfile will cause the prefetch to fail. The test repository needs to contain a very simple RPMs lockfile with at least one invalid checksum. The test will also validate that the error message is correct.

4- Invalid file size

Validates that an incorrect file size in the lockfile will cause the prefetch to fail. The test repository needs to contain a very simple RPMs lockfile with at least one incorrect file size. The test will also validate that the error message is correct.

5- Multiple packages

Validates that Cachi2 is able to process multiple lockfiles in the same repository, as well as appropriatly reporting the prefetched RPMs in the SBOM. The test repository needs to cover the following scenarios:

• Contain the same RPM in both lockfiles, but only report the checksum in one of them (the RPM still needs to be reported as having missing hashes).
• Contain the same package, but with slightly different versions in one of the lockfiles (both need to be reported accurately).

6- Multi Arch

Validates that Cachi2 is able to process multiple architectures in the same lockfile, as well as appropriatly reporting the prefetched RPMs in the SBOM. The test repository needs to contain at least two different architectures, with the same packages reflected in both, as well as contain the corresponding source RPMs. The test will validate the every package is reported correctly with the PURL indicating which arch it corresponds to.

The following scenarios also need to be covered:

• Contain an RPM with missing hashes in only one of the archs (only the RPM corresponding to this arch should be reported with missing hashes).
• Contain an RPM that only exists in one of the archs (it should only be reported with a PURL that points to the correct arch).

Note that this test will not cover any container builds, the checks will be done on the content generated by Cachi2 only.

7- Creation of the .repo file and repository metadata

Validates that the inject-files command is able to generate correct output for RPM prefetch content. The test repository needs to contain a very simple RPMs lockfile that can be prefetched correctly. The test will then execute the inject-files command and verify that repos.d/cachi2.repo and repodata/repomd.xml were properly created. There is no need to actually perform the container build in this test`.

It would be interesting to also test the passing of dnf options (currently being worked on #522).

[brunoapimentel]

@eskultety @onosek @taylormadore @ben-alkov @ejegrova PTAL

[ben-alkov]

I have no objections here, but I am not an RPM SME, so I won't ACK/NACK

[taylormadore]

LGTM

[eskultety]

@brunoapimentel validation of the generated .repo files would nice too

[brunoapimentel]

Do we need to cover only the .repo files, or also the repo metadata that is created by the inject command?

[brunoapimentel]

Were you imagining validating the .repo files in all RPM integration tests (just like we do with the SBOM and project files)? Or have a single test to cover the DNF options feature?

[eskultety]

Do we need to cover only the .repo files, or also the repo metadata that is created by the inject command?

Good question, I think there's value in checking whether repo metadata has been created, but I'd honestly limit this to a mere repodata/repomd.xml existence check since I don't think it makes much sense to do thorough content validation of the created metadata in isolated test scenarios (could potentially make sense for the e2e test, but it is for a deeper analysis to figure out which of the generated files is deterministic enough to be compared with some expected output data).

Were you imagining validating the .repo files in all RPM integration tests (just like we do with the SBOM and project files)? Or have a single test to cover the DNF options feature?

Well, repetitive thorough checks only increase the execution time, hence increasing the load on computation resources, but regardless of that I wouldn't expect us to run inject-files with every isolated integration test scenario for RPM, would we? But for scenarios where inject-files would be executed (i.e. a dedicated inject-files test and e2e test) we should validate the contents of the generated repo file.

[brunoapimentel]

Well, if we're limiting this to the e2e test, I'm wondering if we get much added value by checking the contents of repofiles and the existence of the repodata/repomd.xml. I think the main measure we have that these files are properly generated is that the image is built successfully, and that we can run the main RPM installed.

I don't object adding more thorough checks, though, so let me know if you still think we should do it. I'm thinking that if we want to go that route, we could simply add the two checks mentioned (repofile content and repomd.xml exists) to the currently existing e2e test.

[eskultety]

Well, if we're limiting this to the e2e test, I'm wondering if we get much added value by checking the contents of repofiles and the existence of the repodata/repomd.xml. I think the main measure we have that these files are properly generated is that the image is built successfully, and that we can run the main RPM installed.

Hmm, true about the e2e test. So what about adding an inject-files scenario with that file existence check and keeping e2e as is?

I don't object adding more thorough checks, though, so let me know if you still think we should do it. I'm thinking that if we want to go that route, we could simply add the two checks mentioned (repofile content and repomd.xml exists) to the currently existing e2e test.

Nah, for the e2e you're right, as long as it builds hermetically it's enough of a proof.

[brunoapimentel]

So what about adding an inject-files scenario with that file existence check and keeping e2e as is?

I'm fine with this. I can update the design to include a "inject-files" scenario.