cachi2 should also cache bundler of the locked version

[gsaslis]

This issue was originally reported by @akostadinov on the Cachito issue tracker. I am recreating the feature request here.

cachi2 presently does not cache bundler because it is usually (or never?) not pinned as a dependency with version in the Gemfile.lock [1].

But it is important for the same version of bundler to be installable and used for resolving dependencies.

At the bottom of a Gemfile.lock, there is

BUNDLED WITH
2.3.4
Cachito needs to cache bundler of this version so that builds can install it into the system. Using the bundler from the build image does not usually fit.

P.S. If BUNDLED WITH is not present, only then do not cache bundler. For example if project uses gem install -g instead of bundle install.

Acceptance criteria

• contribute BUNDLED WITH support to the scancode-toolkit gemfile.lock parser (https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/gemfile_lock.py)
• sync changes from the upstream parser to the vendored one at https://github.com/containerbuildsystem/gemlock-parser
• make cachi2 download (and report) bundler at the locked version

As an additional piece of information, I wanted to add that the capability to parse the BUNDLED WITH information has already been contributed (and merged!) to the upstream projects, so that it can be used by cachi2.

[akostadinov]

FYI the gemlock-parser already has BUNDLED WITH support applied with containerbuildsystem/gemlock-parser#3

So the only further change needed is with cachi2, regards.

[brunoapimentel]

I believe we don't have any precedents for actually fetching the exact version package manager binary during a prefetch. I do see why you need it, though, so I was wondering if you could use the generic fetcher to simply download the Gem you need from rubygems.org?

https://github.com/containerbuildsystem/cachi2/blob/main/docs/generic.md

It would essentially require a tiny lockfile to be added to the repo you're building, then Bundler would be properly downloaded and reported in the resulting SBOM.

[eskultety]

@gsaslis Can you please provide your thoughts on ^^^ (#755 (comment))

[gsaslis]

@eskultety / @brunoapimentel I'm a little confused.

The generic fetcher docs mention:

It is highly discouraged for this feature to be used for anything already supported by cachi2 in other ways (such as e.g. pip packages), because the produced SBOM component will not be accurate.

Am I correct in understanding that, while this does offer a way to get the right version of bundler in the build, the downside of that approach would be that this version of bundler is not, in fact, recorded correctly in the SBOM ?

Or does that sentence refer to something else?

With regards to the extra lockfile itself, it does appear unnecessary (since the source of truth for that information already exists with BUNDLED WITH), but it is a solution. That is just my opinion though - @akostadinov should decide on this one.

[lkolacek]

It is highly discouraged for this feature to be used for anything already supported by cachi2 in other ways (such as e.g. pip packages), because the produced SBOM component will not be accurate.

Hi Yorgos, documentation tries to stress, that if there is a package manager support for your use case, you should use that one and not the generic fetcher, because you will receive more precise SBOM among the other benefits.

Your use case is different and it's OK to use generic fetcher to get required version of bundler and then use ruby/bundler package manager for your project as designed.

[eskultety]

With regards to the extra lockfile itself, it does appear unnecessary (since the source of truth for that information already exists with BUNDLED WITH), but it is a solution.

@gsaslis I completely agree with you here - it is far from an ideal UX and UX is something that needs to be one of our top priorities since we're developing this project for our users otherwise, what's the point!? That said, the limitation behind @brunoapimentel's earlier comment comes from some downstream requirements, but if you argued that this IS NOT a downstream space for the project I would not be able argue against. I'm wondering if we could explore a solution where we'd make this a conditional behaviour, i.e. have a backend-specific flag (doesn't make sense globally since not all backends are able to leverage this kind of functionality) and so we could fetch the given version of bundler for you to use in your build. However, this would not be without a tradeoff - we'll have to find a way to mark this kind of prefetch as "tainted" (maybe a custom SBOM property?) due to the fact that the toolchain would not be coming from a trusted base image and so that in case you wish to leverage this functionality in downstream builds, whatever compliance process you need follow has to be aware of such build to be "tainted" and you'll have to deal with the process on your own. I believe that's the best we can propose.
Note though, that despite ^this proposal (without a major team consensus) I am not able to promise if and what priority this kind of work would get, especially since a "workaround" approach exists.

If the team's consensus (@tmadore, @ben-alkov, @a-ovchinnikov, @slimreaper35, @brunoapimentel) is that we could provide this functionality to the described extent, then patches are always welcome :) if this kind of support is somewhat urgent for you in any way.

[gsaslis]

Thanks all for your replies.

I do see that - depending on the ecosystem - the necessity of allowing the packaged manager to be prefetched is not the same.

However, this would not be without a tradeoff - we'll have to find a way to mark this kind of prefetch as "tainted" (maybe a custom SBOM property?) due to the fact that the toolchain would not be coming from a trusted base image and so that in case you wish to leverage this functionality in downstream builds, whatever compliance process you need follow has to be aware of such build to be "tainted" and you'll have to deal with the process on your own.

Speaking for ruby/bundler specifically, I don't see why this should be so. Bundler is a ruby library - exactly like all the other ruby libraries that the project relies on. From the UX point of view, if cachi2 prefetches the project dependencies found in Gemfile.lock I don't see why it can't do the same for bundler. It is a library coming from whatever source all the other libraries are coming from.

And in terms of the SBOM, I would expect the SBOM to include 2 versions of bundler (the system one, if there is one installed in the base image, and the project one, which is the one that is used by the project).

With that said, I'd still defer to @akostadinov to drive this forward, as I'll be moving onto other duties soon.