[rush-lib] Support pnpm lockfile v9

[fzxen]

Summary

pnpm lockfile v9 have some breaking changes on the lockfile format. Rush cannot parse pnpm lockfile v9 correctly with latest version.
After i execute rush install or rush update with pnpm v9, the content of .rush/temp/shrinkwrap-deps.json is not correct. It may break rush cache system.
The expected output should correctly display the hash of each dependency., but actually:

Details

pnpm have some breaking changes on lockfile v9 format.

1. The packages field has been divided into two parts, packages and snapshots.
2. In non-workspace mode, the top-level dependencies field is moved into importers['.'].
3. specifier is not same as lockfile v6

slolution to 1:

Rush will load the lockfile and parse the information in the lockfile by itself.

rushstack/libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

Lines 318 to 347 in 3c6117e

	public static loadFromFile(
	shrinkwrapYamlFilePath: string,
	{ withCaching }: ILoadFromFileOptions = {}
	): PnpmShrinkwrapFile | undefined {
	let loaded: PnpmShrinkwrapFile | undefined;
	if (withCaching) {
	loaded = PnpmShrinkwrapFile._cacheByLockfilePath.get(shrinkwrapYamlFilePath);
	}
	// TODO: Promisify this
	loaded ??= (() => {
	try {
	const shrinkwrapContent: string = FileSystem.readFile(shrinkwrapYamlFilePath);
	return PnpmShrinkwrapFile.loadFromString(shrinkwrapContent);
	} catch (error) {
	if (FileSystem.isNotExistError(error as Error)) {
	return undefined; // file does not exist
	}
	throw new Error(`Error reading "${shrinkwrapYamlFilePath}":\n ${(error as Error).message}`);
	}
	})();
	PnpmShrinkwrapFile._cacheByLockfilePath.set(shrinkwrapYamlFilePath, loaded);
	return loaded;
	}
	public static loadFromString(shrinkwrapContent: string): PnpmShrinkwrapFile {
	const parsedData: IPnpmShrinkwrapYaml = yamlModule.safeLoad(shrinkwrapContent);
	return new PnpmShrinkwrapFile(parsedData);
	}

To ensure consistency with pnpm's parsing logic, I copied the relevant logic from @pnpm/lockfile.fs to PnpmshrinwrapFileConverters.ts.
The convertLockfileV9ToLockfileObject method will automatically merge the snapshots field information into the packages field.

public static loadFromString(shrinkwrapContent: string): PnpmShrinkwrapFile {
  const shrinkwrapJson: IPnpmShrinkwrapYaml = yamlModule.safeLoad(shrinkwrapContent);
  if ((shrinkwrapJson as LockfileFileV9).snapshots) {
    const lockfile: IPnpmShrinkwrapYaml | null = convertLockfileV9ToLockfileObject(
      shrinkwrapJson as LockfileFileV9
    );
    if (lockfile) {
      lockfile.dependencies = lockfile.importers['.' as ProjectId]?.dependencies;
    }
    return new PnpmShrinkwrapFile(lockfile);
  }

  return new PnpmShrinkwrapFile(shrinkwrapJson);
}

solution to 2:

In the PnpmShrinkwrapFile.loadFromString method, read importers['.'].dependencies instead of top-level dependencies.

// if ((shrinkwrapJson as LockfileFileV9).snapshots)  {
// ...
if (lockfile) {
  lockfile.dependencies = lockfile.importers['.' as ProjectId]?.dependencies;
  shrinkwrapFileJson = lockfile;
}
// }

solution to 3:

rush try to parse an encoded pnpm dependency key in parsePnpmDependencyKey method. However, the logic here is no longer applicable to lockfile v9. For example, lockfile v9 will not add a / prefix in the specifier field.

rushstack/libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

Lines 188 to 198 in 3c6117e

	// Example: "path.pkgs.visualstudio.com/@scope/depame/1.4.0" --> 0="@scope/depame" 1="1.4.0"
	// Example: "/isarray/2.0.1" --> 0="isarray" 1="2.0.1"
	// Example: "/sinon-chai/2.8.0/[email protected][email protected]" --> 0="sinon-chai" 1="2.8.0/[email protected][email protected]"
	// Example: "/[email protected]" --> 0=typescript 1="5.1.6"
	// Example: [email protected] --> no match
	// Example: 1.2.3_@[email protected] --> no match
	// Example: 1.2.3([email protected]) --> no match
	// Example: 1.2.3(@scope/[email protected]) --> no match
	const packageNameMatch: RegExpMatchArray | null = /^[^\/(]*\/((?:@[^\/(]+\/)?[^\/(]+)[\/@](.*)$/.exec(
	dependencyKey
	);

Summary of changes to specifier in lockfile v9：

1. remove prefix '/‘: '/@babel/[email protected](@babel/[email protected])' -> @babel/[email protected](@babel/[email protected])
2. URLs specifier always prefix with https:.
3. it will prefix with <PACKAGE_NAME>@ if resolved package name is not same as dependency name in package.json

category	specifier	lockfilev6 version	lockfilev9 version
regular	"@babel/plugin-preset-env": "^7.26.0"	7.26.0(@babel/[email protected])	7.26.0(@babel/[email protected])
URLs	"pad-left": "https://github.com/jonschlinkert/pad-left/tarball/2.1.0"	@github.com/jonschlinkert/pad-left/tarball/2.1.0	https://github.com/jonschlinkert/pad-left/tarball/2.1.0
	"pad-left": "https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz"	@xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz	https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz
	"pad-left": "git://github.com/jonschlinkert/pad-left#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
	"pad-left": "git+ssh://[email protected]:jonschlinkert/pad-left.git#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
file:	"project1": "file:../pnpm_no_workspace/projects/project1"	file:../pnpm_no_workspace/projects/project1	file:../pnpm_no_workspace/projects/project1
path	"project1": "../pnpm_no_workspace/projects/project1"	link:../pnpm_no_workspace/projects/project1	link:../pnpm_no_workspace/projects/project1
Npm alias	"test-pkg": "npm:@babel/[email protected]"	/@babel/[email protected](@babel/[email protected])	@babel/[email protected](@babel/[email protected])
Alias with URLs	"@scope/myDep1": "https://github.com/jonschlinkert/pad-left/tarball/2.1.0"	@github.com/jonschlinkert/pad-left/tarball/2.1.0	pad-left@https://github.com/jonschlinkert/pad-left/tarball/2.1.0
	"@scope/myDep2": "https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz"	@xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz	pad-left@https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz
	"@scope/myDep3": "git://github.com/jonschlinkert/pad-left#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	pad-left@https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
	"@scope/myDep4": "git+ssh://[email protected]:jonschlinkert/pad-left.git#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	pad-left@https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
Alias with file:	"test-pkg": "file:../pnpm_no_workspace/projects/project1"	file:../pnpm_no_workspace/projects/project1	project1@file:../pnpm_no_workspace/projects/project1

It is difficult to maintain those complex regular expressions in the original function. Therefore, I added a new function named as parsePnpm9DependencyKey. It will be called when the expression shrinkwrapFileMajorVersion >= 9 is true in runtime.

const result: DependencySpecifier | undefined =
        this.shrinkwrapFileMajorVersion >= ShrinkwrapFileMajorVersion.V9
          ? parsePnpm9DependencyKey(dependencyName, pnpmDependencyKey)
          : parsePnpmDependencyKey(dependencyName, pnpmDependencyKey);

This MR will not cause any breaking changes. But, pnpm9 and rush subspaces still cannot work together, because pnpm-sync not support pnpm9 yet. tiktok/pnpm-sync#37

I tried use commonly rush command in my own project.

• rush install/update -> The content of shrinkwrap-deps.json file is correct.
• rush build -> Build cache and Cobuild work as expected.
  I added some unit test cases in PnpmShrinkwrapFile.test.ts and ShrinkwrapFile.test.ts.
  At the same time, unit tests have also been added for the PnpmShrinkWrapFileConverters.ts file.

[fzxen]

@microsoft-github-policy-service agree