From 2a383cd70ee138af641714065af422cf7a87410d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Thu, 15 Aug 2024 15:11:37 +0200 Subject: [PATCH 1/4] First draft --- proposals/0167-loader-v4.md | 296 ++++++++++++++++++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 proposals/0167-loader-v4.md diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md new file mode 100644 index 00000000..0ce6523d --- /dev/null +++ b/proposals/0167-loader-v4.md @@ -0,0 +1,296 @@ +--- +simd: '0167' +title: Loader-v4 +authors: + - Alexander Meißner +category: Standard +type: Core +status: Draft +created: 2024-08-15 +feature: TBD +--- + +## Summary + +A new upgradeable loader which only requires a single account per program. + +## Motivation + +Loader-v3, which is currently the only deployable loader, requires two accounts +per program. This was a workaround to circumvent the finality of the +`is_executable` flag, which is removed in SIMD-0162. Consequentially, this +setup of the program account being a proxy account, containing the address of +the actual programdata account, is no longer necessary and should be removed. + +Another issue with loader-v3 is that the executable file stored in the +programdata account is misaligned relative to the beginning of the account. + +Additionally, there currently is no complete specification of the loaders +program management instructions. This proposal would thus fill that gap once +loader-v4 goes into production. + +See impact for further motivation. + +## Alternatives Considered + +A delay-visibility-free redeployment could be achieved by keeping the swap +program around until the end of the slot. This would however mean that two +accounts per program must be loaded until the dapp developer reclaims the +second one. That would defy the purpose of this proposal which is to get rid +of the proxy account. + +## New Terminology + +None. + +## Detailed Design + +`LoaderV411111111111111111111111111111111111` program management and execution +must be enabled with the associated feature gate, which simultaneously disables +new deployments on loader-v3 (`BPFLoaderUpgradeab1e11111111111111111111111`), +throwing `InvalidIstructionData` if `DeployWithMaxDataLen` is called. + +### Owned Program Accounts + +Accounts of programs owned by loader-v4 must have the following layout: + +- Header (which is 48 bytes long): + - `u64` Slot in which the program was last deployed, retracted or + initialized. + - `[u8; 32]` Authority address which can send program management + instructions. + - `u64` status enum: + - Enum variant `0u64`: Retracted, program is in maintenance + - Enum variant `1u64`: Deployed, program is ready to be executed + - Enum variant `2u64`: Finalized, same as `Deployed`, but can not be + modified anymore +- Body: + - `[u8]` The programs executable file + +Verification the program account checks in the following order that: + +- the owner of the program account is loader-v4, +otherwise throw `InvalidAccountOwner` +- the program account is at least as long enough for the header, +otherwise throw `AccountDataTooSmall` +- the program account is writable, otherwise throw `InvalidArgument` +- the provided authority (instruction account at index 1) signed, +otherwise throw `MissingRequiredSignature` +- the authority stored in the program account is the one provided, +otherwise throw `IncorrectAuthority` +- the status stored in the program account is not finalized, +otherwise throw `Immutable` + +### Execution / Invocation + +Invoking programs owned by loader-v4 checks in the following order that: + +- the owner of the program account is loader-v4, +otherwise throw `UnsupportedProgramId` +- the program account is at least as long enough for the header, +otherwise throw `AccountDataTooSmall` +- the status stored in the program account is not retracted, +otherwise throw `UnsupportedProgramId` +- the program account was not deployed recently (delay-visibility), +otherwise throw `UnsupportedProgramId` +- the executable file stored in the program account passes executable +verification, otherwise throw `UnsupportedProgramId` + +Otherwise the execution semantics stay the same as in loader-v2 and v3. +Delay-visibility also stays the same as in loader-v3: + +- Re/deployed programs act as closed until the end of the slot, +only then becoming available for execution +- The feature set that the executable file is verified against is not +necessarily the current one, but the one of the epoch of the next slot + +### Program Management Instructions + +#### Write + +- Instruction accounts: + - `[writable]` The program account to write to. + - `[signer]` The authority of the program. +- Instruction data: + - Enum variant `0u32` + - `u32` Offset at which to write the given bytes + - `[u8]` Chunk of the programs executable file +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check the status stored in the program account is retracted, + otherwise throw `InvalidArgument` + - Check that the end offset (sum of offset and length of the chunk) does + not exceed the maximum (program account length minus the header size), + otherwise throw `AccountDataTooSmall` + - Copy the chunk into the program account at the offset shifted by the + header size + +#### Truncate + +- Instruction accounts: + - `[(signer), writable]` The program account to change the size of. + - `[signer]` The authority of the program. + - `[writable]` Optional, the recipient account. +- Instruction data: + - Enum variant `1u32` + - `u32` The new size after the operation. +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - If this is an initialization (program account length is too short to + contain the header and the requested new size is greater 0): + - the owner of the program account is loader-v4, + otherwise throw `InvalidAccountOwner` + - the program account is writable, otherwise throw `InvalidArgument` + - the program account (instruction account at index 0) signed, + otherwise throw `MissingRequiredSignature` + - the provided authority (instruction account at index 1) signed, + otherwise throw `MissingRequiredSignature` + - If this is not an initialization: + - Verify the program account + - Check that the status stored in the program account is retracted, + otherwise throw `InvalidArgument` + - Check that there are enough funds in the program account for rent + exemption, otherwise throw `InsufficientFunds` + - If there are more than enough funds: + - Check there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Check that the recipient account (instruction account at index 2) is + writable, otherwise throw `InvalidArgument` + - Transfer the surplus from the program account to the recipient account + - If the requested new size is zero: + - Delete the entire program account, including the header + - If the requested new size is greater than zero: + - Set the length of the program account to the requested new size plus + the header size + - In case that this is an initialization, also initialize the header: + - Set the `is_executable` flag to `true` + - Set the slot to zero, **not** the current slot + - Set the authority address (from the instruction account at index 1) + - Set the status to retracted + +#### Deploy + +- Instruction accounts: + - `[writable]` The program account to deploy. + - `[signer]` The authority of the program. + - `[writable]` Optional, an undeployed source program account to take data + and lamports from. +- Instruction data: + - Enum variant `2u32` +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check that the deployment cooldown expired, + otherwise throw `InvalidArgument` + - Check that the status stored in the program account is retracted + otherwise throw `InvalidArgument` + - In case a source program was provided (instruction account at index 2): + - Verify the source program account + - Check that the status stored in the source program account is retracted, + otherwise throw `InvalidArgument` + - Check that the executable file stored in the source program account + passes executable verification + - Copy the entire source program account into the program account + - Set the length of the source program account to zero + - Transfer all funds of the source program account to the program + account + - In case no source program was provided: + - Check that the executable file stored in the program account passes + executable verification + - Change the slot in the program account to the current slot + - Change the status stored in the program account to deployed + +#### Retract + +- Instruction accounts: + - `[writable]` The program account to retract. + - `[signer]` The authority of the program. +- Instruction data: + - Enum variant `3u32` +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check that the deployment cooldown expired, + otherwise throw `InvalidArgument` + - Check that the status stored in the program account is deployed, + otherwise throw `InvalidArgument` + - Note: The slot is **not** set to the current slot to allow a + retract-modify-redeploy-sequence within the same slot + - Change the status stored in the program account to retracted + +#### TransferAuthority + +- Instruction accounts: + - `[writable]` The program account to change the authority of. + - `[signer]` The current authority of the program. + - `[signer]` Optional, the new authority of the program. +- Instruction data: + - Enum variant `4u32` +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - In case a new authority was provided (instruction account at index 2): + - Check that it signed as well, + otherwise throw `MissingRequiredSignature` + - Check that the authority stored in the program account is different + from the one provided, otherwise throw `InvalidArgument` + - Copy the new authority address into the program account + - In case no new authority was provided: + - Check that the status stored in the program account is deployed, + otherwise throw `InvalidArgument` + - Change the status stored in the program account to finalized + +## Impact + +This proposal: + +- covers all the use cases loader-v3 had but in a cleaner way and comes with +a specification. +- makes deployment slightly cheaper for dapp developers as they would no longer +have to burn funds for the rent exception of the proxy account. +- provides an alternative redeployment path which does not require a big +deposit of funds for rent exception during the upload. +- enables dapp developers to withdrawl the surplus of funds required for rent +exception when shortening the length of program accounts or closing them. +- shortens the workflow of temporarily closing a program to a single +instruction, instead of having to build and redeploy an empty program. +- properly alignes the executable file relative to the beginning of the +account. In loader-v3 it is misaligned. +- once all loader-v3 programs are migrated: + - allows transaction account loading to be simplifed, because every program + would load exactly one account, no need to load the proxy account to get to + the actual program data (which is not listed in the transaction accounts). + - allows the removal of the write lock demotion exception if loader-v3 is + present in a transaction. + - corrects the miscounting of the proxy account size towards the total + transaction account loading limit. + +## Security Considerations + +None. + +## Backwards Compatibility + +This proposal does not break any existing programs. However, dapp developers +might want to profit from the new program mangement instructions without +influencing their users work flows. To do so they would need a way to turn the +program accounts of loader-v3 to program accounts of loader-v4, changing the +account owner but keeping the program address. A potential issue is that the +programdata header of loader-v3 is only 45 bytes long while loader-v4 takes 48 +bytes. An automatic mechanism in the program runtime (triggered by feature +activation) could then perform the following steps per program: + +- loader-v3 clears the program proxy account (setting its size to zero) +- loader-v3 transfers all funds from the programdata to the proxy account +- loader-v3 gifts the program proxy account to loader-v4 +- loader-v4 initializes it via `Truncate` +- loader-v4 copies the data from the programdata account via `Write` +- loader-v4 deploys it via `Deploy` +- Optinally, loader-v4 finalizes it without a next version forwarding +- loader-v3 closes the programdata account (setting its size to zero) From 0b3d4c51b4e5796215f5dc405039b0a65081a7a7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Mon, 26 Aug 2024 12:23:49 +0200 Subject: [PATCH 2/4] Adds explicit finalization and next version forwarding. --- proposals/0167-loader-v4.md | 51 ++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 0ce6523d..0ddf03de 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -58,7 +58,8 @@ Accounts of programs owned by loader-v4 must have the following layout: - `u64` Slot in which the program was last deployed, retracted or initialized. - `[u8; 32]` Authority address which can send program management - instructions. + instructions. Or if the status is finalized, then the address of the next + version of the program. - `u64` status enum: - Enum variant `0u64`: Retracted, program is in maintenance - Enum variant `1u64`: Deployed, program is ready to be executed @@ -228,23 +229,46 @@ necessarily the current one, but the one of the epoch of the next slot - Instruction accounts: - `[writable]` The program account to change the authority of. - `[signer]` The current authority of the program. - - `[signer]` Optional, the new authority of the program. + - `[signer]` The new authority of the program. - Instruction data: - Enum variant `4u32` - Behavior: - - Check there are at least two instruction accounts, + - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - In case a new authority was provided (instruction account at index 2): - - Check that it signed as well, - otherwise throw `MissingRequiredSignature` - - Check that the authority stored in the program account is different - from the one provided, otherwise throw `InvalidArgument` - - Copy the new authority address into the program account - - In case no new authority was provided: - - Check that the status stored in the program account is deployed, + - Check that the new authority (instruction account at index 2) + signed as well, otherwise throw `MissingRequiredSignature` + - Check that the authority stored in the program account is different + from the one provided, otherwise throw `InvalidArgument` + - Copy the new authority address into the program account + +#### Finalize + +- Instruction accounts: + - `[writable]` The program account to change the authority of. + - `[signer]` The current authority of the program. + - `[]` Optional, the reserved address for the next version of the program. +- Instruction data: + - Enum variant `5u32` +- Behavior: + - Check there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check that the status stored in the program account is deployed, otherwise throw `InvalidArgument` - - Change the status stored in the program account to finalized + - for the program account of the next version + (instruction account at index 2) check that: + - the owner of the program account is loader-v4, + otherwise throw `InvalidAccountOwner` + - the program account is at least as long enough for the header, + otherwise throw `AccountDataTooSmall` + - the authority stored in the program account is the one provided, + otherwise throw `IncorrectAuthority` + - the status stored in the program account is not finalized, + otherwise throw `Immutable` + - Copy the address of the next version into the next version field stored in + the previous versions program account + - Change the status stored in the program account to finalized ## Impact @@ -252,6 +276,9 @@ This proposal: - covers all the use cases loader-v3 had but in a cleaner way and comes with a specification. +- allows finalized programs to mark which other program supersedes them which +can then be offered as an option in forntends. This provides a more secure +alternative to redeployment / upgrading of programs at the same address. - makes deployment slightly cheaper for dapp developers as they would no longer have to burn funds for the rent exception of the proxy account. - provides an alternative redeployment path which does not require a big From 80d9562335007e2843173b8016e906a23bfd9eef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 1 Oct 2024 14:30:08 +0200 Subject: [PATCH 3/4] Clearifies delay visibility and deployment cooldown. --- proposals/0167-loader-v4.md | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 0ddf03de..3afb0397 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -92,19 +92,11 @@ otherwise throw `UnsupportedProgramId` otherwise throw `AccountDataTooSmall` - the status stored in the program account is not retracted, otherwise throw `UnsupportedProgramId` -- the program account was not deployed recently (delay-visibility), -otherwise throw `UnsupportedProgramId` +- the program account was not deployed within the current slot +(delay visibility), otherwise throw `UnsupportedProgramId` - the executable file stored in the program account passes executable verification, otherwise throw `UnsupportedProgramId` -Otherwise the execution semantics stay the same as in loader-v2 and v3. -Delay-visibility also stays the same as in loader-v3: - -- Re/deployed programs act as closed until the end of the slot, -only then becoming available for execution -- The feature set that the executable file is verified against is not -necessarily the current one, but the one of the epoch of the next slot - ### Program Management Instructions #### Write @@ -185,8 +177,8 @@ necessarily the current one, but the one of the epoch of the next slot - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - Check that the deployment cooldown expired, - otherwise throw `InvalidArgument` + - Check that the slot stored in the program account is not the current + (deployment cooldown), otherwise throw `InvalidArgument` - Check that the status stored in the program account is retracted otherwise throw `InvalidArgument` - In case a source program was provided (instruction account at index 2): @@ -195,6 +187,8 @@ necessarily the current one, but the one of the epoch of the next slot otherwise throw `InvalidArgument` - Check that the executable file stored in the source program account passes executable verification + - The feature set that the executable file is verified against is not +necessarily the current one, but the one of the epoch of the next slot - Copy the entire source program account into the program account - Set the length of the source program account to zero - Transfer all funds of the source program account to the program @@ -216,8 +210,8 @@ necessarily the current one, but the one of the epoch of the next slot - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - Check that the deployment cooldown expired, - otherwise throw `InvalidArgument` + - Check that the slot stored in the program account is not the current + (deployment cooldown), otherwise throw `InvalidArgument` - Check that the status stored in the program account is deployed, otherwise throw `InvalidArgument` - Note: The slot is **not** set to the current slot to allow a From a4f4116855e447b6cd3128ee6aef9df372579508 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 19 Nov 2024 16:04:17 +0100 Subject: [PATCH 4/4] Further clarifications after implementing integration tests. --- proposals/0167-loader-v4.md | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 3afb0397..d57de874 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -5,7 +5,7 @@ authors: - Alexander Meißner category: Standard type: Core -status: Draft +status: Review created: 2024-08-15 feature: TBD --- @@ -45,9 +45,13 @@ None. ## Detailed Design -`LoaderV411111111111111111111111111111111111` program management and execution -must be enabled with the associated feature gate, which simultaneously disables -new deployments on loader-v3 (`BPFLoaderUpgradeab1e11111111111111111111111`), +The associated feature gate must: + +- add loader-v4 to the write lock demotion exceptions +- enable loader-v4 `LoaderV411111111111111111111111111111111111` program +management and execution +- simultaneously disable new deployments on loader-v3 +(`BPFLoaderUpgradeab1e11111111111111111111111`), throwing `InvalidIstructionData` if `DeployWithMaxDataLen` is called. ### Owned Program Accounts @@ -86,19 +90,20 @@ otherwise throw `Immutable` Invoking programs owned by loader-v4 checks in the following order that: -- the owner of the program account is loader-v4, -otherwise throw `UnsupportedProgramId` -- the program account is at least as long enough for the header, -otherwise throw `AccountDataTooSmall` -- the status stored in the program account is not retracted, -otherwise throw `UnsupportedProgramId` -- the program account was not deployed within the current slot -(delay visibility), otherwise throw `UnsupportedProgramId` +- the owner of the program account is loader-v4 +- the program account is at least as long enough for the header +- the status stored in the program account is not retracted +- the program account was not deployed within the current slot (delay +visibility) - the executable file stored in the program account passes executable -verification, otherwise throw `UnsupportedProgramId` +verification + +failing any of the above checks must throw `UnsupportedProgramId`. ### Program Management Instructions +All program management instructions must cost 2000 CUs. + #### Write - Instruction accounts: @@ -187,8 +192,10 @@ verification, otherwise throw `UnsupportedProgramId` otherwise throw `InvalidArgument` - Check that the executable file stored in the source program account passes executable verification - - The feature set that the executable file is verified against is not -necessarily the current one, but the one of the epoch of the next slot + - The feature set that the executable file is verified against is not + necessarily the current one, but the one of the epoch of the next slot + - Also, during deployment certain deprecated syscalls are disabled, + this stays the same as in the older loaders - Copy the entire source program account into the program account - Set the length of the source program account to zero - Transfer all funds of the source program account to the program