RFC: Vendoring dependencies

[winterqt]

Rendered

[ljharb]

Vendoring is a resoundingly bad practice that npm should not in any way encourage or ease - there's a reason checking node_modules into version control is similarly discouraged.

With a lockfile, and an internal registry, there should be no issues with reproducibility I'm aware of. Can you elaborate on what I might be missing here?

[ljharb]

the third option is the best practice for a long time - use a lockfile, and an internal registry, and any resulting differences between systems of node_modules are intentional and necessary.

[winterqt]

I understand that there are differences between different node_modules, but having to run an internal registry certainly begs for a more lightweight solution, IMHO. (Why use an internal registry over the public one, assuming all of your dependencies are there, if you have a lockfile?)

[ljharb]

Because deploying from the internet is hugely unsafe and exposes you to DNS/man-in-the-middle attacks?

Any discussion about reproducibility rests on first having no uncontrolled sources of content.

[winterqt]

How is this the case with a known-good lockfile, which contains hashes for all dependencies, though?

[ljharb]

That's fair, but you still don't want the contents denied to you entirely - that breaks your build too.

[winterqt]

Can you elaborate on what I might be missing here?

Sure.

The motivation for this RFC is my effort to rework how we package npm projects in Nixpkgs -- which presents certain challenges surrounding reproducibility.

The current solution used is to generate large files (warning: 6 MB file) containing dependency trees.

I've proposed an alternate solution which does what I've described as the second alternative -- generating a cacache using a small, purpose-built fetcher, which only relies on a lockfile. (I've done it this over e.g. shelling out to npm to further ensure reproducibility.)

This has proven to work very well for this use case, which is great. But as I said, relying on this is a worse idea than an official solution, since the (historically very low) chances of breakage are there.

We currently use similar solutions for Go, Cargo, and Yarn (v1), utilizing the aforementioned vendoring solutions for the first two, while a home-grown(-ish?) solution for Yarn.

(I'm guessing this comes off as selfish if I'm only doing it for this reason, but I also believe that Go and Cargo must have implemented this for a good reason.)

[winterqt]

Vendoring is a resoundingly bad practice that npm should not in any way encourage or ease - there's a reason checking node_modules into version control is similarly discouraged.

Why do you(/others, I'm assuming) believe this? Assuming the vendored dependencies include any and all dependencies that are required for the project, no matter the OS etc, it provides more reproducibility than checking in node_modules.

[ljharb]

Reproducibility isn't remotely more important than a manageable git repository or a navigable/comprehensible git diff/log.

[winterqt]

Sure, but alternate uses exist, which is evident by Cargo producing a tarball as opposed to a folder.

[wraithgar]

Is this functionally any different than including all of your dependencies as bundled and running npm pack?

[winterqt]

I don't believe so, except for the fact that npm pack depends on the current state of the node_modules folder. Assuming npm ci --ignore-scripts wasn't used for installation, this could introduce impurities depending on what the install/prepare scripts for the dependencies do (e.g. grabbing a prebuilt version of a native dependency).

I think the two serve different purposes, though, and one doesn't require modifying existing projects.

[wraithgar]

npm pack mirrors as exactly as possible building the tgz file that gets attached during npm publish.

If there is a potential for having things bundled that are missing, this should be solved in the pack/publish process itself so that npm pack becomes as reliable as you need it to be. I don't know what other problem this RFC would be solving, so let's solve that problem itself (if indeed it is determined there is one)

[winterqt]

npm pack mirrors as exactly as possible building the tgz file that gets attached during npm publish.

Right, but it still depends on the local state of the node_modules folder, no?

[wraithgar]

Right, but it still depends on the local state of the node_modules folder, no?

Yep. And if that's a problem, then that's the problem to fix.

[winterqt]

Looks like I made a pretty dumb factual error here -- both Cargo and Go use folders.

I'll push a revision fixing that, as well as also mentioning npm pack as an alternative.

[winterqt]

Fixed.