-
Notifications
You must be signed in to change notification settings - Fork 28
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Introduces the ADR for enabling Yarn v4 support
The ADR contains a thorough investigation on the changes that v4 introduced, and how they can impact Cachi2's prefetching. Signed-off-by: Bruno Pimentel <[email protected]>
- Loading branch information
1 parent
354d9f4
commit 98e99a2
Showing
1 changed file
with
248 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| # Extend Yarn Berry support to v4 | ||
|
|
||
| ## Context | ||
|
|
||
| Yarn v4 was released in October 23, 2023, which coincided with the time that the support for Yarn v3 was being introduced in Cachi2. To limit the scope of implementation, we decided to leave support for v4 to a later time. Now, as more projects migrate to v4, introducing proper support has become a priority. | ||
|
|
||
| This document analyzes the most impactful changes introduced with v4, and how they relate to the current v3 implementation. | ||
|
|
||
| Main references: | ||
| - [v4 blog post](https://yarnpkg.com/blog/release/4.0) | ||
| - [Breaking changes for Yarn 4](https://github.com/yarnpkg/berry/issues/3591) | ||
|
|
||
| ### Changes in Yarn Behavior | ||
|
|
||
| #### Global cache is enabled by default | ||
|
|
||
| The [enableGlobalCache](https://yarnpkg.com/configuration/yarnrc#enableGlobalCache) option allows the user to set a shared location for the cache folder. Cachi2 already treats this option the following way: | ||
|
|
||
| - Set to `true` during the prefetch, and point it to a specific location in order to keep the prefetched dependencies. | ||
| - Set to `false` during the build, since the build needs to read from the cache, but the dependencies need to be installed to a local folder since they'll be used during the runtime. | ||
|
|
||
| The only action that might be necessary is to document this behavior, since now during build-time, Cachi2 is setting the opposite option that would be expected as default. | ||
|
|
||
| #### All official plugins are enabled by default | ||
|
|
||
| Official plugins can no longer be disabled by `.yarnrc.yml` configuration. The current official plugins don't seem to introduce any behavior that would taint the accuracy of the prefetched dependencies, though. Most of them only add support to protocols (which will still be filtered using the same rules of Yarn v4) or CLI commands. | ||
|
|
||
| The list of all official plugins can be found under the "Default Plugins" section in the [API](https://yarnpkg.com/api) page on the official Yarn documentation. Here's a short summary of every official plugin (as of 4.5.3): | ||
|
|
||
| <details> | ||
| <summary>Plugins that add support for a protocol</summary> | ||
|
|
||
| - plugin-exec | ||
| - plugin-file | ||
| - plugin-git | ||
| - plugin-http | ||
| - plugin-link | ||
| - plugin-npm | ||
| - plugin-patch | ||
| </details> | ||
|
|
||
| <details> | ||
| <summary>Plugins that enable a CLI command</summary> | ||
|
|
||
| - plugin-essentials | ||
| - plugin-init | ||
| - plugin-interactive-tools | ||
| - plugin-npm-cli | ||
| - plugin-pack | ||
| - plugin-stage | ||
| - plugin-workspace-tools | ||
| - plugin-version | ||
| </details> | ||
|
|
||
| <details> | ||
| <summary>Other plugins</summary> | ||
|
|
||
| - plugin-compat: patches packages that aren't compatible with Plug'n'Play | ||
| - plugin-constraints: support for [constraints](https://yarnpkg.com/features/constraints) | ||
| - plugin-dlx: install a package in temporary environment | ||
| - plugin-github: improves the performance when cloning from Github | ||
| - plugin-nm: support for installing packages in `node_modules` | ||
| - plugin-pnp: support for [Plug'n'Play](https://yarnpkg.com/features/pnp) | ||
| - plugin-pnpm: support for installing packages using symlinks | ||
| - plugin-typescript: Automatically adds `@types/` packages into your dependencies | ||
| </details> | ||
| <br> | ||
|
|
||
| We should still be able to proceed only by disabling non-official plugins without introducing any arbitrary code execution, and keep the current behavior for v3 projects unchanged. | ||
|
|
||
| ##### A note about plugin-typescript | ||
|
|
||
| The [typescript](https://yarnpkg.com/api/plugin-typescript) plugin is used to automatically include types when adding a dependency that does not package them by default. Since these changes are reflected in the `package.json` file, Cachi2 will handle them normally. | ||
|
|
||
| <details> | ||
| <summary>Example of how the Typescript plugin works</summary> | ||
|
|
||
| ``` | ||
| $ yarn add lodash | ||
| ➤ YN0000: · Yarn 4.5.3 | ||
| ➤ YN0000: ┌ Resolution step | ||
| ➤ YN0085: │ + @types/lodash@npm:4.17.13, lodash@npm:4.17.21 | ||
| ➤ YN0000: └ Completed | ||
| ➤ YN0000: ┌ Fetch step | ||
| ➤ YN0013: │ A package was added to the project (+ 957.26 KiB). | ||
| ➤ YN0000: └ Completed in 0s 252ms | ||
| ➤ YN0000: ┌ Link step | ||
| ➤ YN0000: └ Completed | ||
| ➤ YN0000: · Done in 0s 313ms | ||
| $ cat package.json | ||
| { | ||
| "name": "yarn-types", | ||
| "packageManager": "[email protected]", | ||
| "dependencies": { | ||
| "lodash": "^4.17.21" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/lodash": "^4" | ||
| } | ||
| } | ||
| ``` | ||
| </details> | ||
|
|
||
| #### pnpDataPath is no longer configurable | ||
|
|
||
| This used to be a configurable key in the `.yarnrc.yml` file. The previous default path, `./.pnp.data.json`, is now hard-coded and can't be changed. | ||
|
|
||
| The only mention to `pnpDataPath` in Cachi2 is the [check](https://github.com/containerbuildsystem/cachi2/blob/a5f19c6f9be90be4289beee35ecccd2827bbb328/cachi2/core/package_managers/yarn/main.py#L43) for paths pointing outside of the repo. The check can be kept the same way so Yarn v3 can still be covered, and it will simply be skipped in v4. | ||
|
|
||
| #### Yarn now caches npm version metadata | ||
|
|
||
| Although Yarn only seems to create the cache folder (`{globalFolder}/metadata/npm`) when the [hardened mode](https://yarnpkg.com/configuration/yarnrc#enableHardenedMode) is enabled. Simply running `yarn install` on a local project or even prefetching with Cachi2 doesn't seem to generate this extra folder. | ||
|
|
||
| The metadata, in itself, is a collection of json files with a few kilobytes each. In case we decide to enable the hardened mode, it'd be ideal to delete this npm metadata this from the output folder, although there's no harm in keeping it. | ||
|
|
||
| #### Changes to .yarnrc.yml options | ||
|
|
||
| - **enableConstraintsChecks**: when set to true, it will automatically execute [constraint checks](https://yarnpkg.com/features/constraints) after right after `yarn install` finishes. We need to explicitly disable it. | ||
|
|
||
| *Curiously, the documentation says this [option](https://yarnpkg.com/configuration/yarnrc#enableConstraintsChecks) is `true` by default, but during the [tests](https://github.com/brunoapimentel/cachi2-experiments/commit/01adc210f9e8f52e4c6afa12e1c08c0831e35c6d) I made, the constraint checks are only executed when we explicitly set this option to true in `.yarnrc.yml`* | ||
|
|
||
| ##### Some new options worth of notice | ||
|
|
||
| - **cacheMigrationMode**: determines behavior when dealing with outdated cache. We never reuse cache, so this won't affect us. | ||
| - **enableOfflineMode**: tells Yarn to use the local cache instead of making a network request. Since it is not enforcing (it tries to use the local cache only if possible), it won't help setting it as a build environment variable. | ||
| - **tsEnableAutoTypes**: enable/disable the installing of types for packages that don't provide their own. This only happens during `yarn add`, so it won't affect Cachi2. | ||
|
|
||
| #### Changes to yarn.lock | ||
|
|
||
| When updating v3 projects to v4, some differences in the lockfile appear. These are some picks from updating the [disallowed-protocols](https://github.com/cachito-testing/cachi2-yarn-berry/tree/disallowed-protocols) branch on the cachi2-yarn-berry test project: | ||
|
|
||
| **`npm:` locator added to npm dependencies:** | ||
| ``` | ||
| dependencies: | ||
| - chownr: ^2.0.0 | ||
| - fs-minipass: ^2.0.0 | ||
| - minipass: ^5.0.0 | ||
| - minizlib: ^2.1.1 | ||
| - mkdirp: ^1.0.3 | ||
| - yallist: ^4.0.0 | ||
| - checksum: ... | ||
| + chownr: "npm:^2.0.0" | ||
| + fs-minipass: "npm:^2.0.0" | ||
| + minipass: "npm:^5.0.0" | ||
| + minizlib: "npm:^2.1.1" | ||
| + mkdirp: "npm:^1.0.3" | ||
| + yallist: "npm:^4.0.0" | ||
| + checksum: ... | ||
| ``` | ||
|
|
||
| **Changes to some instances of the file locator:** | ||
| ``` | ||
| - resolution: "strip-ansi-tarball@file:external-packages/strip-ansi-4.0.0.tgz::locator=berryscary%40workspace%3A." | ||
| + resolution: "strip-ansi-tarball@file:external-packages/strip-ansi-4.0.0.tgz#external-packages/strip-ansi-4.0.0.tgz::hash=e17689&locator=berryscary%40workspace%3A." | ||
| ``` | ||
|
|
||
| Yarn v4 introduced the subdirectory (`#external-packages/strip-ansi-4.0.0.tgz`) and the hash (`::hash=e17689`) to the previously existing locator. These parts are already [handled](https://github.com/containerbuildsystem/cachi2/blob/a5f19c6f9be90be4289beee35ecccd2827bbb328/cachi2/core/package_managers/yarn/locators.py#L77-L79) by the v3 implementation in Cachi2, though. | ||
|
|
||
| **Changes to some instances of the patch locator:** | ||
| ``` | ||
| - resolution: "typescript@patch:typescript@npm%3A5.1.6#~builtin<compat/typescript>::version=5.1.6&hash=5da071" | ||
| + resolution: "typescript@patch:typescript@npm%3A5.1.6#optional!builtin<compat/typescript>::version=5.1.6&hash=5da071" | ||
| ``` | ||
|
|
||
| Instances of the patch locator are currently [being ignored](https://github.com/containerbuildsystem/cachi2/blob/a5f19c6f9be90be4289beee35ecccd2827bbb328/cachi2/core/package_managers/yarn/resolver.py#L264-L268) in the v3 implementation in Cachi2. | ||
|
|
||
| #### Hardened mode | ||
|
|
||
| Yarn v4 has introduced a [hardened mode](https://yarnpkg.com/blog/release/4.0#hardened-mode) to avoid lockfile poisoning attacks (i.e. when the resolved url for a dependency points to an non-standard malicious location). The downside to it is that the time to perform `yarn install` is increased. | ||
|
|
||
| According to a small subset of tests, the time to perform `yarn install` will increase by 1.5 to 2 times when enabling the hardened mode. We very likely would benefit by toggling this on during the prefetch, but the extra prefetch time is something to consider. In any case, this can be decided and implemented as a follow-up. | ||
|
|
||
| <details> | ||
| <summary>Tests on hardened mode on "cachi2 fetch-deps" time</summary> | ||
|
|
||
| **Test method**: Call `cachi2 fetch-deps` while setting the [enableHardenedMode](https://yarnpkg.com/configuration/yarnrc#enableHardenedMode) `.yarnrc.yml` option. | ||
|
|
||
| **plop**<br> | ||
| 884 deps - 72M<br> | ||
| https://github.com/plopjs/plop<br> | ||
| e0122279d1376ee62604acbbff1e76a88935b1af | ||
|
|
||
| non-hardened:<br> | ||
| 37.53s user 9.72s system 321% cpu 14.682 total<br> | ||
| 38.00s user 9.37s system 339% cpu 13.957 total<br> | ||
| 36.16s user 9.31s system 327% cpu 13.873 total<br> | ||
| *average time 14.17*<br> | ||
|
|
||
| hardened:<br> | ||
| 45.00s user 13.99s system 290% cpu 20.319 total<br> | ||
| 50.67s user 15.45s system 299% cpu 22.099 total<br> | ||
| 56.10s user 14.89s system 352% cpu 20.148 total<br> | ||
| *average time 20.855*<br> | ||
| ~47% increase | ||
|
|
||
| =============================================== | ||
|
|
||
| **porta**<br> | ||
| 1387 deps - 505M | ||
| https://github.com/3scale/porta<br> | ||
| bcbaea6392b58a89291932b2289127e44864e288<br> | ||
| (upgraded to yarn-4.5.3) | ||
|
|
||
| non-hardened:<br> | ||
| 29.32s user 9.66s system 369% cpu 10.554 total<br> | ||
| 32.41s user 10.69s system 406% cpu 10.604 total<br> | ||
| 29.43s user 9.97s system 362% cpu 10.864 total<br> | ||
| *average time 10.674*<br> | ||
|
|
||
| hardened:<br> | ||
| 41.66s user 16.28s system 294% cpu 19.646 total<br> | ||
| 65.75s user 19.94s system 416% cpu 20.576 total<br> | ||
| 64.89s user 20.47s system 404% cpu 21.106 total<br> | ||
| *average time 20.442*<br> | ||
| *~91% increase* | ||
|
|
||
| ================================================ | ||
|
|
||
| **twenty**<br> | ||
| 4139 deps - 2.2G<br> | ||
| https://github.com/twentyhq/twenty<br> | ||
| e492efb79e8fd8d8f8292c2fccd991d4448ac249<br> | ||
|
|
||
| non-hardened:<br> | ||
| 122.67s user 75.31s system 305% cpu 64.91 total<br> | ||
| 173.75s user 92.09s system 336% cpu 78.99 total<br> | ||
| 111.99s user 69.36s system 315% cpu 57.568 total<br> | ||
| *average time 67.155* | ||
|
|
||
| hardened:<br> | ||
| 245.61s user 103.60s system 351% cpu 99.30 total<br> | ||
| 274.81s user 109.91s system 361% cpu 106.42 total<br> | ||
| 244.61s user 107.89s system 349% cpu 100.81 total<br> | ||
| *average time 102.176*<br> | ||
| *~52% increase* | ||
|
|
||
| </details> | ||
|
|
||
| ## Decision | ||
|
|
||
| The changes needed to support Yarn v4 boil down to raising the maximum supported version, adding a few integration tests to cover v4 scenarios and updating the documentation. The Corepack shim is compatible with v4, so the project will still be processed with the exact Yarn version that is defined in its configuration files. | ||
|
|
||
| Enabling the newly introduced hardened mode during the prefetch might prove useful to further increase the security of the process, but is by no means necessary to introduce basic support for v4. The impact on the prefetching speed is something that needs to be investigated in depth before we decide to enable it, but the decision can be deferred to a later moment. | ||
|
|
||
| ## Consequences | ||
|
|
||
| Support for Yarn v4 can be made available as soon as this ADR is merged, and no impacts for v3 users are expected. |