From c1a05afd833293df59740a70ea3bef3383670596 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 26 Apr 2023 18:53:32 +0530 Subject: [PATCH 01/67] Create 0023-Diet-client-v0.md --- proposals/0023-Diet-client-v0.md | 91 ++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 proposals/0023-Diet-client-v0.md diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-Diet-client-v0.md new file mode 100644 index 000000000..a3dc14e3c --- /dev/null +++ b/proposals/0023-Diet-client-v0.md @@ -0,0 +1,91 @@ +--- +simd: '0023' +title: Diet client v0 +authors: + - Harsh Patel (Tinydancer) + - Anoushk Kharangate (Tinydancer) +category: Standard +type: Core +status: Draft +created: 2023-04-26 +--- + +## Summary +Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this (SIMD)[https://github.com/solana-foundation/solana-improvement-documents/pull/10] + +## Motivation +For light clients to be possible on Solana, there is a need to access ledger data in the form of shreds for Data Availability +sampling to verify ledger data. In addition to verifying the ledger, we also need to verify the consensus for a particular block. +Solana doesn't have the concept of blockheaders like Ethereum, so we added a new data structure in the solana-transaction-status +crate to address this issue. +## Alternatives Considered +Currently there are no other alternatives + + +## New Terminology + +BlockHeader + +## Detailed Design + +We introduce two RPC calls: +- #### getShreds + ``` + pub async fn get_shreds( + &self, + slot: Slot, + shred_indices: Vec, + config: Option, + ) -> Result + ``` +This call would allow for data availability sampling, this is specifically added to the `rpc.rs` file as an additional method to +`JsonRpcRequestProcessor` where we pass in the slot, the indices of the required shreds and the config which contains the +CommitmentConfig of the block. + +- #### getBlockHeaders +``` +pub async fn get_block_headers( + &self, + slot: Slot, + config: Option>, + ) -> Result +``` + +``` +pub struct BlockHeader { + pub vote_signature: Vec>, + pub validator_identity: Vec>, + pub validator_stake: Vec>, +} +``` +This function will return a BlockHeader, a data structure storing a list of: + - Signatures of validators who voted on that block + - The public keys or 'identities' of the validators who voted on that block. + - The stake amounts of each of those validators. + +Explain the feature as if it was already implemented and you're explaining it +to another Solana core contributor. The generally means: + +- Explain the proposed change and how it works +- Where the feature fits in to the runtime, core, or relevant sub-system +- How this feature was/could be implemented +- Interaction with other features +- Edge cases + +## Impact + +How will the implemented proposal impacts dapp developers, validators, and core contributors? + +## Security Considerations + +What security implications/considerations come with implementing this feature? +Are there any implementation-specific guidance or pitfalls? + +## Drawbacks *(Optional)* + +Why should we not do this? + +## Backwards Compatibility *(Optional)* + +Does the feature introduce any breaking changes? All incompatibilities and +consequences should be listed. From f2e19494b279aed19abf56d97c540492dba3d79b Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 26 Apr 2023 18:54:39 +0530 Subject: [PATCH 02/67] Update 0023-Diet-client-v0.md --- proposals/0023-Diet-client-v0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-Diet-client-v0.md index a3dc14e3c..89a44d0b2 100644 --- a/proposals/0023-Diet-client-v0.md +++ b/proposals/0023-Diet-client-v0.md @@ -11,7 +11,7 @@ created: 2023-04-26 --- ## Summary -Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this (SIMD)[https://github.com/solana-foundation/solana-improvement-documents/pull/10] +Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) ## Motivation For light clients to be possible on Solana, there is a need to access ledger data in the form of shreds for Data Availability From c1143c7b6ff278d08a2f8a1a72335bb654810dbb Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 26 Apr 2023 18:58:46 +0530 Subject: [PATCH 03/67] Update 0023-Diet-client-v0.md --- proposals/0023-Diet-client-v0.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-Diet-client-v0.md index 89a44d0b2..8678d816b 100644 --- a/proposals/0023-Diet-client-v0.md +++ b/proposals/0023-Diet-client-v0.md @@ -10,6 +10,7 @@ status: Draft created: 2023-04-26 --- +#WIP ## Summary Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) From 1b6577b7f8b2b1135070ab74cc17d08309c73f71 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 26 Apr 2023 18:59:35 +0530 Subject: [PATCH 04/67] Update 0023-Diet-client-v0.md --- proposals/0023-Diet-client-v0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-Diet-client-v0.md index 8678d816b..fa184d297 100644 --- a/proposals/0023-Diet-client-v0.md +++ b/proposals/0023-Diet-client-v0.md @@ -10,7 +10,7 @@ status: Draft created: 2023-04-26 --- -#WIP +# WIP ## Summary Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) From 1534b5b0b8c03916cd84b22fdfa9a2c789d1de52 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 26 Apr 2023 19:55:38 +0530 Subject: [PATCH 05/67] Update 0023-Diet-client-v0.md --- proposals/0023-Diet-client-v0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-Diet-client-v0.md index fa184d297..28af6ba2a 100644 --- a/proposals/0023-Diet-client-v0.md +++ b/proposals/0023-Diet-client-v0.md @@ -60,9 +60,9 @@ pub struct BlockHeader { } ``` This function will return a BlockHeader, a data structure storing a list of: - - Signatures of validators who voted on that block - - The public keys or 'identities' of the validators who voted on that block. - - The stake amounts of each of those validators. + - Signatures of validators who voted on that block + - The public keys or 'identities' of the validators who voted on that block. + - The stake amounts of each of those validators. Explain the feature as if it was already implemented and you're explaining it to another Solana core contributor. The generally means: From 4021f25e9d7113c6ddc5a56aebd068a71800ca0f Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Thu, 27 Apr 2023 13:40:50 +0530 Subject: [PATCH 06/67] Update 0023-Diet-client-v0.md --- proposals/0023-Diet-client-v0.md | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-Diet-client-v0.md index 28af6ba2a..650f40d08 100644 --- a/proposals/0023-Diet-client-v0.md +++ b/proposals/0023-Diet-client-v0.md @@ -18,9 +18,10 @@ Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for d For light clients to be possible on Solana, there is a need to access ledger data in the form of shreds for Data Availability sampling to verify ledger data. In addition to verifying the ledger, we also need to verify the consensus for a particular block. Solana doesn't have the concept of blockheaders like Ethereum, so we added a new data structure in the solana-transaction-status -crate to address this issue. +crate to address this issue. + ## Alternatives Considered -Currently there are no other alternatives +Currently there are no other alternatives to retrieving ## New Terminology @@ -41,7 +42,8 @@ We introduce two RPC calls: ``` This call would allow for data availability sampling, this is specifically added to the `rpc.rs` file as an additional method to `JsonRpcRequestProcessor` where we pass in the slot, the indices of the required shreds and the config which contains the -CommitmentConfig of the block. +CommitmentConfig of the block. Apart from this we also made the shred data structures accessible to other crates from +`pub(crate)` => `pub`. - #### getBlockHeaders ``` @@ -64,14 +66,6 @@ This function will return a BlockHeader, a data structure storing a list of: - The public keys or 'identities' of the validators who voted on that block. - The stake amounts of each of those validators. -Explain the feature as if it was already implemented and you're explaining it -to another Solana core contributor. The generally means: - -- Explain the proposed change and how it works -- Where the feature fits in to the runtime, core, or relevant sub-system -- How this feature was/could be implemented -- Interaction with other features -- Edge cases ## Impact From c555a78851044d7e2cc19027b03615815c2fbb2b Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Mon, 1 May 2023 13:18:09 +0530 Subject: [PATCH 07/67] Update and rename 0023-Diet-client-v0.md to 0023-diet-client-v0.md --- proposals/{0023-Diet-client-v0.md => 0023-diet-client-v0.md} | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) rename proposals/{0023-Diet-client-v0.md => 0023-diet-client-v0.md} (82%) diff --git a/proposals/0023-Diet-client-v0.md b/proposals/0023-diet-client-v0.md similarity index 82% rename from proposals/0023-Diet-client-v0.md rename to proposals/0023-diet-client-v0.md index 650f40d08..553b42c78 100644 --- a/proposals/0023-Diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -21,7 +21,9 @@ Solana doesn't have the concept of blockheaders like Ethereum, so we added a new crate to address this issue. ## Alternatives Considered -Currently there are no other alternatives to retrieving +Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: +- Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. +- Validators aren't obligated to propagate gossip changes and have reason to do so as it reduces egress costs. ## New Terminology From c9a5ad784c49c5e4b7540acedc0ae2d0faf5f862 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Mon, 1 May 2023 13:22:20 +0530 Subject: [PATCH 08/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 1 - 1 file changed, 1 deletion(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 553b42c78..7d5566f20 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -10,7 +10,6 @@ status: Draft created: 2023-04-26 --- -# WIP ## Summary Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) From 248fb164c9a266cc4ce4db33372add5c71c6e2b5 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Mon, 1 May 2023 13:45:46 +0530 Subject: [PATCH 09/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 7d5566f20..8a3730292 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -80,8 +80,3 @@ Are there any implementation-specific guidance or pitfalls? ## Drawbacks *(Optional)* Why should we not do this? - -## Backwards Compatibility *(Optional)* - -Does the feature introduce any breaking changes? All incompatibilities and -consequences should be listed. From dcde05fede00497b0bcbab12dd3647ef8d671b38 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Wed, 3 May 2023 13:20:48 +0530 Subject: [PATCH 10/67] update --- proposals/0023-diet-client-v0.md | 31 +++++++++---------------------- 1 file changed, 9 insertions(+), 22 deletions(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 8a3730292..f4320b310 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -1,6 +1,6 @@ --- simd: '0023' -title: Diet client v0 +title: Diet Client V0(Consensus Client) authors: - Harsh Patel (Tinydancer) - Anoushk Kharangate (Tinydancer) @@ -11,13 +11,15 @@ created: 2023-04-26 --- ## Summary -Add two new RPC calls `getShreds()` and `getBlockHeaders()` to add support for diet clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) +Add a new RPC call and `getBlockHeaders()` to add support for consenus verifying clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) ## Motivation -For light clients to be possible on Solana, there is a need to access ledger data in the form of shreds for Data Availability -sampling to verify ledger data. In addition to verifying the ledger, we also need to verify the consensus for a particular block. -Solana doesn't have the concept of blockheaders like Ethereum, so we added a new data structure in the solana-transaction-status -crate to address this issue. +For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from mailicious nodes. + +This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. + +This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less propable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet clietn implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. + ## Alternatives Considered Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: @@ -27,25 +29,10 @@ Another solution is to use the gossip plane to read votes out of the CRDS optimi ## New Terminology -BlockHeader +BlockHeader: A structure containing all vote identities, vote signatures and stake amounts that has voted on a block. ## Detailed Design -We introduce two RPC calls: -- #### getShreds - ``` - pub async fn get_shreds( - &self, - slot: Slot, - shred_indices: Vec, - config: Option, - ) -> Result - ``` -This call would allow for data availability sampling, this is specifically added to the `rpc.rs` file as an additional method to -`JsonRpcRequestProcessor` where we pass in the slot, the indices of the required shreds and the config which contains the -CommitmentConfig of the block. Apart from this we also made the shred data structures accessible to other crates from -`pub(crate)` => `pub`. - - #### getBlockHeaders ``` pub async fn get_block_headers( From 42be8195bb1729916e0fc87aacb97e1b6e87ffc3 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Sat, 6 May 2023 11:37:48 +0530 Subject: [PATCH 11/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index f4320b310..8e112bafd 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -24,7 +24,7 @@ This ensures that at-least the user doesn't have to trust the RPC service that i ## Alternatives Considered Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: - Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. -- Validators aren't obligated to propagate gossip changes and have reason to do so as it reduces egress costs. +- Validators aren't obligated to propagate gossip changes and have reason to not do so as it reduces egress costs. ## New Terminology From 18b5bb27c3629827750c65fdf161fe8a566de960 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Sat, 6 May 2023 12:17:01 +0530 Subject: [PATCH 12/67] update --- proposals/0023-diet-client-v0.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 8e112bafd..74cd40326 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -32,6 +32,16 @@ Another solution is to use the gossip plane to read votes out of the CRDS optimi BlockHeader: A structure containing all vote identities, vote signatures and stake amounts that has voted on a block. ## Detailed Design +The protocol interaction will be as follows: +- User makes a transaction using their wallet in slot N. +- The client then reads the validator set from gossip and requests BlockHeader for slot N + 1, N + 2, N + 3 +- It first validates whether the validator identies match the set from gossip. +- Then proceeds to validate the vote signatures. +- The client also has to sync the epoch stake history from genesis from the entrypoint light client. +- Next it checks if the stake weights in the block header are within reasonable boundaries of the epoch history. +- If so, it checks if the stake weights add up to > 67% of the total stake +- If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. + - #### getBlockHeaders ``` From 289ab7c275aa159cf3e91a2794d8e5f039268ad5 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Sat, 6 May 2023 12:20:22 +0530 Subject: [PATCH 13/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 74cd40326..cd127c6d3 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -14,11 +14,11 @@ created: 2023-04-26 Add a new RPC call and `getBlockHeaders()` to add support for consenus verifying clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) ## Motivation -For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from mailicious nodes. +For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. -This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less propable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet clietn implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. +This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less probable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. ## Alternatives Considered From 1d78fbbf39902e89169afb8250e21cf621740a66 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Sat, 6 May 2023 12:22:26 +0530 Subject: [PATCH 14/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index cd127c6d3..0fffb9079 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -35,7 +35,7 @@ BlockHeader: A structure containing all vote identities, vote signatures and sta The protocol interaction will be as follows: - User makes a transaction using their wallet in slot N. - The client then reads the validator set from gossip and requests BlockHeader for slot N + 1, N + 2, N + 3 -- It first validates whether the validator identies match the set from gossip. +- It first validates whether the validator identities match the set from gossip. - Then proceeds to validate the vote signatures. - The client also has to sync the epoch stake history from genesis from the entrypoint light client. - Next it checks if the stake weights in the block header are within reasonable boundaries of the epoch history. From 9bc9c9e3894779cf135f10ae8f36e77fe2227970 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Sat, 6 May 2023 12:33:23 +0530 Subject: [PATCH 15/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 0fffb9079..9f358aeab 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -11,7 +11,7 @@ created: 2023-04-26 --- ## Summary -Add a new RPC call and `getBlockHeaders()` to add support for consenus verifying clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) +Add a new RPC call and `getBlockHeaders()` to add support for consensus verifying clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) ## Motivation For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. From 7a8bcc855398829851b955d132f3dd205fa8c900 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Sat, 6 May 2023 21:29:35 +0530 Subject: [PATCH 16/67] Update 0023-diet-client-v0.md --- proposals/0023-diet-client-v0.md | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index 9f358aeab..cb1ec3972 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -26,6 +26,8 @@ Another solution is to use the gossip plane to read votes out of the CRDS optimi - Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. - Validators aren't obligated to propagate gossip changes and have reason to not do so as it reduces egress costs. +We could also just directly make these calls from the client itself but its much faster and more convenient to do it on server side. + ## New Terminology @@ -33,17 +35,16 @@ BlockHeader: A structure containing all vote identities, vote signatures and sta ## Detailed Design The protocol interaction will be as follows: -- User makes a transaction using their wallet in slot N. -- The client then reads the validator set from gossip and requests BlockHeader for slot N + 1, N + 2, N + 3 -- It first validates whether the validator identities match the set from gossip. -- Then proceeds to validate the vote signatures. -- The client also has to sync the epoch stake history from genesis from the entrypoint light client. -- Next it checks if the stake weights in the block header are within reasonable boundaries of the epoch history. -- If so, it checks if the stake weights add up to > 67% of the total stake -- If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. +1. User makes a transaction using their wallet in slot N. +2. The client then reads the validator set from gossip and requests BlockHeader for slot N + 1, N + 2, N + 3 +3. It first validates whether the validator identities match the set from gossip. +4. Then proceeds to validate the vote signatures. +5. The client also has to sync the epoch stake history from genesis from the entrypoint light client(eventually can be requested from multiple other light clients). +6. Next it checks if the stake weights in the block header are within reasonable boundaries of the epoch history. More specifically the stake difference between epoch N and N-1 is not > 20 % of epoch N - 1. +8. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. -- #### getBlockHeaders +#### New RPC Methods ``` pub async fn get_block_headers( &self, @@ -59,7 +60,7 @@ pub struct BlockHeader { pub validator_stake: Vec>, } ``` -This function will return a BlockHeader, a data structure storing a list of: +This method will return a BlockHeader, a data structure storing a list of: - Signatures of validators who voted on that block - The public keys or 'identities' of the validators who voted on that block. - The stake amounts of each of those validators. @@ -67,13 +68,9 @@ This function will return a BlockHeader, a data structure storing a list of: ## Impact -How will the implemented proposal impacts dapp developers, validators, and core contributors? +This proposal will improve the overall security and decentralisation of the Solana network allowing users to access the blockchain in a trust +minimised way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. ## Security Considerations -What security implications/considerations come with implementing this feature? -Are there any implementation-specific guidance or pitfalls? - -## Drawbacks *(Optional)* - -Why should we not do this? +None From 08b657e0d00af63122ccd6cc936180540fc47829 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Tue, 16 May 2023 13:31:27 +0530 Subject: [PATCH 17/67] stake logic --- proposals/0023-diet-client-v0.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md index cb1ec3972..156af4057 100644 --- a/proposals/0023-diet-client-v0.md +++ b/proposals/0023-diet-client-v0.md @@ -11,7 +11,8 @@ created: 2023-04-26 --- ## Summary -Add a new RPC call and `getBlockHeaders()` to add support for consensus verifying clients as first described in this [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) +Users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. The user +will be running a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) ## Motivation For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. @@ -36,13 +37,17 @@ BlockHeader: A structure containing all vote identities, vote signatures and sta ## Detailed Design The protocol interaction will be as follows: 1. User makes a transaction using their wallet in slot N. -2. The client then reads the validator set from gossip and requests BlockHeader for slot N + 1, N + 2, N + 3 +2. The client then reads the validator set from gossip and requests BlockHeader for slot N to (N+32) given the 32 block depth finality. 3. It first validates whether the validator identities match the set from gossip. 4. Then proceeds to validate the vote signatures. -5. The client also has to sync the epoch stake history from genesis from the entrypoint light client(eventually can be requested from multiple other light clients). -6. Next it checks if the stake weights in the block header are within reasonable boundaries of the epoch history. More specifically the stake difference between epoch N and N-1 is not > 20 % of epoch N - 1. +5. The client also has to sync the epoch stake history from genesis from the entrypoint light clients eventually can be requested from multiple other light clients). +6. Next it checks if the stake weights in the block header are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. 8. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. +#### Types of Light Clients +1. **Entrypoint / Hub Clients** - These will be holding the snapshot with the stake history from genesis to current slot and be periodically syncing with the latest epoch. + +3. **Ultra Light Clients** - These will be clients running in wallets on mobile or browser that will TOFU(Trust On First Use) the stake from the entrypoint clients and verify if the user transactions are voted on by the supermajority. #### New RPC Methods ``` @@ -73,4 +78,5 @@ minimised way unlike traditionally where users had to fully trust their RPC prov ## Security Considerations -None +### Trust Assumptions +The light client makes certain trust assumptions that reduce reliance on RPC but these assumptions have tradeoffs. The ultra light clients verify if the supermajority has voted on the block, this is could be an issue if the supermajority itself is corrupt and is trying to vote on an invalid transaction. The future iterations of the light client that include data availability sampling and fraud proving should address this issue reduce the trust assumption to single honest node. We also rely on a entrypoint light client that has the snapshot and serves the stake history which is also a point of trust. From 1ef37b03a89b5aa33206e03137f12e34c0fc7cca Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Wed, 24 May 2023 16:00:05 -0400 Subject: [PATCH 18/67] init --- proposals/00xx-tx-verification.md | 141 ++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 proposals/00xx-tx-verification.md diff --git a/proposals/00xx-tx-verification.md b/proposals/00xx-tx-verification.md new file mode 100644 index 000000000..6751e68c0 --- /dev/null +++ b/proposals/00xx-tx-verification.md @@ -0,0 +1,141 @@ +--- +simd: '00XX' +title: Transaction Verification RPC Method +authors: + - Harsh Patel (Tinydancer) + - Anoushk Kharangate (Tinydancer) + - x19 (Tinydancer) +category: Standard +type: Core +status: Draft +created: 2023-05-24 +--- + +## Summary + +Add a new RPC method which returns data which proves that a transaction is included in a specific slot. + +## Motivation + +The first step for Solana light/diet clients is to verify a transaction was processed in a specific slot. This requires a hash path from a transaction's signature to a slot's bankhash as proof (as noted in a [previous diet client SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10)). Since validators vote on bankhashes, once we have this proof, we can also verify a transaction has received a supermajority of votes. Current RPC methods don't provide enough information so we need a new RPC method for this. + +## Alternatives Considered + +None + +## New Terminology + +None + +## Detailed Design + +The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return the following struct: + +```rust +// new RPC struct to verify tx inclusion +pub struct TransactionProof { + // used to verify hash([start_blockhash, entires]) = bank_hash + // also verify the tx_signature of interest is included in an Entry + pub entries: Vec, + pub start_blockhash: Hash, + // used to reconstruct `bank_hash` + pub parent_hash: Hash, + pub accounts_delta_hash: Hash, + pub signature_count_buf: [u8; 8], +} +``` + +All the variables are accessible from the rpc's blockstore and bank_forks variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: + +```rust +// new RPC method +pub async fn get_transaction_proof(&self, slot: Slot) -> Result { + // first retrieve all the entries + let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) + // require all the shreds + assert!(is_full) + + let bank_forks = self.bank_forks.read().unwrap(); + let bank = bank_forks.get(slot); + + // get variables used to compute bank hash + let parent_hash = bank.parent_hash(); + let accounts_delta_hash = bank + .rc + .accounts + .accounts_db + .calculate_accounts_delta_hash(slot).0; + let mut signature_count_buf = [0u8; 8]; + LittleEndian::write_u64(&mut signature_count_buf[..], bank.signature_count()); + + // get the start_hash for the slot (will be the last entry's hash from slot-1) + let start_blockhash = self.blockstore.get_slot_entries_with_shred_info(slot-1, 0, false).last().hash; + + Ok(TransactionProof{ + entries, + start_blockhash, + parent_hash, + accounts_delta_hash, + signature_count_buf, + }) +} +``` + +On the client side, verifying a transaction is included in a bankhash is as follows: +- find the slot in which the transaction has been included +- get the `TransactionProof` struct from the RPC for that slot +- validate the `entries` are valid PoH hashes starting with the hash `start_blockhash` +- validate that the transaction signature is included in an Entry +- reconstruct the expected bankhash using the other variables in the struct and the final entry hash + +Below is client pseudocode for verifying a transaction: + +```rust +let tx_sig = "..."; // tx of interest +let slot = 19; // slot which includes the tx + +// new RPC call +let tx_proof: TransactionProof = get_tx_proof(slot, endpoint); + +// verify the entires are valid +let verified = tx_proof.entries.verify(&tx_proof.start_blockhash); +assert!(verified); + +// find and verify the tx sig +let mut start_hash = &tx_proof.start_blockhash; +for entry in entries.iter() { + // find Entry which includes tx sig + let tx_is_in = entry.transactions.iter().any(|tx| { + tx.signatures.contains(&tx_sig) + }); + if tx_is_in { + // verify Entry includes tx + let hash = next_hash(start_hash, entry.num_hashes, &entry.transactions); + assert!(hash == entry.hash); + break; + } + start_hash = &entry.hash; +} + +// recompute the bank hash +let last_blockhash = entries.last().unwrap().hash; +let bankhash = hashv(&[ + tx_proof.parent_hash.as_ref(), + tx_proof.accounts_delta_hash.as_ref(), + tx_proof.signature_count_buf.as_ref(), + last_blockhash.as_ref() +]); +``` +Once we computed the expected bankhash which includes the transaction, we can parse vote transactions which vote on that bankhash and assert that a supermajority (>= 2/3 of stake) has voted on that bank. + +## Future Work + +While there are improvements and additional features that can be made (as mentioned in [this proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification)), including using merkle trees to compute the blockhash (instead of the current sequential implementation), transaction status codes, and validator set verification, we chose to keep this SIMD self-contained and only add a new RPC method with no changes to the protocol. Optimizations will be left to future SIMDs which will build off of this one. + +## Impact + +This proposal will improve the overall security and decentralisation of the Solana network allowing users to verify their transactions in a trustless way, unlike traditionally where users had to fully trust their RPC providers. Since its only a new RPC method, Dapp developers don't have to make any special changes, as wallets and can easily integrate the verification to be compatible with any transaction. + +## Security Considerations + +While this SIMD greatly increases the trustlessness on a single RPC, the light client will still need to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid (when using a merkle tree structure for the entries). We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving. From bc6c30494012a3676152c3ae97dbca8968130d61 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Wed, 24 May 2023 16:02:43 -0400 Subject: [PATCH 19/67] fix --- proposals/00xx-tx-verification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/00xx-tx-verification.md b/proposals/00xx-tx-verification.md index 6751e68c0..cb67eeafb 100644 --- a/proposals/00xx-tx-verification.md +++ b/proposals/00xx-tx-verification.md @@ -13,7 +13,7 @@ created: 2023-05-24 ## Summary -Add a new RPC method which returns data which proves that a transaction is included in a specific slot. +Add a new RPC method that returns data to prove a transaction is included in a specific slot. ## Motivation From 938dec63933c138b099a7ff5939b7b3fb382f22f Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Wed, 24 May 2023 16:03:52 -0400 Subject: [PATCH 20/67] up --- proposals/00xx-tx-verification.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/00xx-tx-verification.md b/proposals/00xx-tx-verification.md index cb67eeafb..e419a58f9 100644 --- a/proposals/00xx-tx-verification.md +++ b/proposals/00xx-tx-verification.md @@ -32,6 +32,8 @@ None The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return the following struct: ```rust +pub async fn get_transaction_proof(&self, slot: Slot) -> Result; + // new RPC struct to verify tx inclusion pub struct TransactionProof { // used to verify hash([start_blockhash, entires]) = bank_hash From 79d58791d26ecac11209b575f14288499c98b865 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Wed, 24 May 2023 16:45:20 -0400 Subject: [PATCH 21/67] update --- proposals/00xx-tx-verification.md | 36 +++++++++++++++++-------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/proposals/00xx-tx-verification.md b/proposals/00xx-tx-verification.md index e419a58f9..f0346cfbb 100644 --- a/proposals/00xx-tx-verification.md +++ b/proposals/00xx-tx-verification.md @@ -13,11 +13,11 @@ created: 2023-05-24 ## Summary -Add a new RPC method that returns data to prove a transaction is included in a specific slot. +Add a new RPC method that returns data to prove a transaction (tx) is included in a specific slot. ## Motivation -The first step for Solana light/diet clients is to verify a transaction was processed in a specific slot. This requires a hash path from a transaction's signature to a slot's bankhash as proof (as noted in a [previous diet client SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10)). Since validators vote on bankhashes, once we have this proof, we can also verify a transaction has received a supermajority of votes. Current RPC methods don't provide enough information so we need a new RPC method for this. +The first step for Solana light/diet clients is to verify a transaction was processed in a specific slot. This requires a hash path from a transaction's signature to a slot's bankhash as proof (as noted in a [previous diet client SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10)). Since validators sign votes on bankhashes, once we have this proof, we can also verify a transaction has received a supermajority of votes. Current RPC methods don't provide enough information so we need a new RPC method for this. ## Alternatives Considered @@ -29,9 +29,10 @@ None ## Detailed Design -The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return the following struct: +The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct: ```rust +// new RPC method pub async fn get_transaction_proof(&self, slot: Slot) -> Result; // new RPC struct to verify tx inclusion @@ -50,11 +51,10 @@ pub struct TransactionProof { All the variables are accessible from the rpc's blockstore and bank_forks variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: ```rust -// new RPC method pub async fn get_transaction_proof(&self, slot: Slot) -> Result { // first retrieve all the entries let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) - // require all the shreds + // require all of the entries assert!(is_full) let bank_forks = self.bank_forks.read().unwrap(); @@ -83,28 +83,29 @@ pub async fn get_transaction_proof(&self, slot: Slot) -> Result= 2/3 of stake) has voted on that bank. + +Once we computed the expected bankhash, we can parse vote transactions which vote on that bankhash, and assert that a supermajority (>= 2/3 of stake) has voted on it. ## Future Work @@ -140,4 +144,4 @@ This proposal will improve the overall security and decentralisation of the Sola ## Security Considerations -While this SIMD greatly increases the trustlessness on a single RPC, the light client will still need to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid (when using a merkle tree structure for the entries). We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving. +While this SIMD greatly increases the trustlessness on a single RPC, the light client will still need to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid. We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving. From c5a2a38c422922c05b153d442c984cb03c774c55 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Thu, 25 May 2023 21:44:36 +0530 Subject: [PATCH 22/67] Update and rename 0023-diet-client-v0.md to 0023-consensus-and-transaction-inclusion-verification.md --- ...-and-transaction-inclusion-verification.md | 169 ++++++++++++++++++ proposals/0023-diet-client-v0.md | 82 --------- 2 files changed, 169 insertions(+), 82 deletions(-) create mode 100644 proposals/0023-consensus-and-transaction-inclusion-verification.md delete mode 100644 proposals/0023-diet-client-v0.md diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md new file mode 100644 index 000000000..36c318baf --- /dev/null +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -0,0 +1,169 @@ +--- +simd: '0023' +title: Consensus and Transaction Inclusion Verification +authors: + - Harsh Patel (Tinydancer) + - Anoushk Kharangate (Tinydancer) + - x19 (Tinydancer) +category: Standard +type: Core +status: Draft +created: 2023-05-25 +--- + +## Summary +Users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. The user +will be running a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) + +## Motivation +For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. + +This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. + +This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less probable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. + + +## Alternatives Considered +Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: +- Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. +- Validators aren't obligated to propagate gossip changes and have reason to not do so as it reduces egress costs. + +We could also just directly make these calls from the client itself but its much faster and more convenient to do it on server side. + + +## New Terminology + +TransactionProof: A structure containing necessary information to verify if a transaction was included in the bank hash of a slot. + +## Detailed Design +The protocol interaction will be as follows: +1. User makes a transaction using their wallet in slot N. +2. The client then reads the validator set from gossip and requests vote signatures and stake commitments for slot N to (N+32) given the 32 block depth finality. +3. It first validates whether the validator identities match the set from gossip. +4. Then proceeds to validate the vote signatures. +5. The client also has to sync the epoch stake history from genesis from the entrypoint light clients eventually can be requested from multiple other light clients). +6. Next it checks if the stake weights returned are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. +7. Next it will request the transaction proof using the slot and transaction signature. It will then perform the following checks: + - Verify the entries by checking that hashing the start hash of the slot `num_hash` times results in the same hash as the entry. + - Check if the transaction signature is included in any of the slots entries. + - Check if transaction is included in the poh hash by calculating `hash(prev_hash,hash(transaction_signatures))` and if it matches the entry hash. +8. Reconstruct the bank hash with blockhash(entry hash), parent_hash, accounts_delta_hash and signature_count and check if all votes voted on this bank hash by parsing them. +9. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. + +#### Types of Light Clients +1. **Entrypoint / Hub Clients** - These will be holding the snapshot with the stake history from genesis to current slot and be periodically syncing with the latest epoch. + +3. **Ultra Light Clients** - These will be clients running in wallets on mobile or browser that will TOFU(Trust On First Use) the stake from the entrypoint clients and verify if the user transactions are voted on by the supermajority. + +#### New RPC Methods +The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct +```rs +// new RPC method +pub async fn get_transaction_proof(&self, slot: Slot) -> Result; + +// new RPC struct to verify tx inclusion +pub struct TransactionProof { + pub entries: Vec, + pub start_blockhash: Hash, + pub parent_hash: Hash, + pub accounts_delta_hash: Hash, + pub signature_count_buf: [u8; 8], +} +``` +All the variables are accessible from the rpc's blockstore and bank_forks variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: +```rs +pub async fn get_transaction_proof(&self, slot: Slot) -> Result { + // first retrieve all the entries + let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) + // require all of the entries + assert!(is_full) + + let bank_forks = self.bank_forks.read().unwrap(); + let bank = bank_forks.get(slot); + + // get variables used to compute bank hash + let parent_hash = bank.parent_hash(); + let accounts_delta_hash = bank + .rc + .accounts + .accounts_db + .calculate_accounts_delta_hash(slot).0; + let mut signature_count_buf = [0u8; 8]; + LittleEndian::write_u64(&mut signature_count_buf[..], bank.signature_count()); + + // get the start_hash for the slot (will be the last entry's hash from slot-1) + let start_blockhash = self.blockstore.get_slot_entries_with_shred_info(slot-1, 0, false).last().hash; + + Ok(TransactionProof{ + entries, + start_blockhash, + parent_hash, + accounts_delta_hash, + signature_count_buf, + }) +} +``` +Using the `get_transaction_proof` RPC call, a client can verify a transaction with the following steps: +- first, for a given transaction signature, find the slot which its included in +- call the `get_transaction_proof` RPC method with that slot as input +- verify the `entries` are valid PoH hashes starting with the hash `start_blockhash` +- verify that the transaction signature is included in one of the `entries` +- reconstruct the expected bankhash using the other variables in the struct and the final entry's hash + +Below is client pseudocode for verifying a transaction: +```rust +let tx_sig = "..."; // tx signature of interest +let slot = 19; // slot which includes the tx + +// call the new RPC +let tx_proof: TransactionProof = get_tx_proof(slot, endpoint); + +// verify the entires are valid +let verified = tx_proof.entries.verify(&tx_proof.start_blockhash); +assert!(verified); + +// verify that the transaction signature is included in one of the `entries` +let mut start_hash = &tx_proof.start_blockhash; +let mut was_verified = false; +for entry in entries.iter() { + // find Entry which includes tx sig + let tx_is_in = entry.transactions.iter().any(|tx| { + tx.signatures.contains(&tx_sig) + }); + if tx_is_in { + // verify Entry includes tx + let hash = next_hash(start_hash, entry.num_hashes, &entry.transactions); + assert!(hash == entry.hash); + was_verified = true; + break; + } + start_hash = &entry.hash; +} +assert!(was_verified); + +// recompute the bank hash +let last_blockhash = entries.last().unwrap().hash; +let bankhash = hashv(&[ + tx_proof.parent_hash.as_ref(), + tx_proof.accounts_delta_hash.as_ref(), + tx_proof.signature_count_buf.as_ref(), + last_blockhash.as_ref() +]); +``` +Once we computed the expected bankhash, we can parse vote transactions which vote on that bankhash, and assert that a supermajority (>= 2/3 of stake) has voted on it. + +## Impact + +This proposal will improve the overall security and decentralisation of the Solana network allowing users to access the blockchain in a trust +minimised way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. + +## Security Considerations + +### Trust Assumptions +The light client makes certain trust assumptions that reduce reliance on RPC but these assumptions have tradeoffs. The ultra light clients verify if the supermajority has voted on the block, this is could be an issue if the supermajority itself is corrupt and is trying to vote on an invalid transaction. + +## Future Work +The future iterations of the light client that include data availability sampling and fraud proving should address this issue reduce the trust assumption to single honest node. We also rely on a entrypoint light client that has the snapshot and serves the stake history which is also a point of trust. + +Additionally, while there are improvements and additional features that can be made (as mentioned in [this proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification)), including using merkle trees to compute the blockhash (instead of the current sequential implementation), transaction status codes, and validator set verification, we chose to keep this SIMD self-contained and only add a new RPC method with no changes to the protocol. Optimizations will be left to future SIMDs which will build off of this one. + diff --git a/proposals/0023-diet-client-v0.md b/proposals/0023-diet-client-v0.md deleted file mode 100644 index 156af4057..000000000 --- a/proposals/0023-diet-client-v0.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -simd: '0023' -title: Diet Client V0(Consensus Client) -authors: - - Harsh Patel (Tinydancer) - - Anoushk Kharangate (Tinydancer) -category: Standard -type: Core -status: Draft -created: 2023-04-26 ---- - -## Summary -Users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. The user -will be running a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) - -## Motivation -For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. - -This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. - -This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less probable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. - - -## Alternatives Considered -Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: -- Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. -- Validators aren't obligated to propagate gossip changes and have reason to not do so as it reduces egress costs. - -We could also just directly make these calls from the client itself but its much faster and more convenient to do it on server side. - - -## New Terminology - -BlockHeader: A structure containing all vote identities, vote signatures and stake amounts that has voted on a block. - -## Detailed Design -The protocol interaction will be as follows: -1. User makes a transaction using their wallet in slot N. -2. The client then reads the validator set from gossip and requests BlockHeader for slot N to (N+32) given the 32 block depth finality. -3. It first validates whether the validator identities match the set from gossip. -4. Then proceeds to validate the vote signatures. -5. The client also has to sync the epoch stake history from genesis from the entrypoint light clients eventually can be requested from multiple other light clients). -6. Next it checks if the stake weights in the block header are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. -8. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. - -#### Types of Light Clients -1. **Entrypoint / Hub Clients** - These will be holding the snapshot with the stake history from genesis to current slot and be periodically syncing with the latest epoch. - -3. **Ultra Light Clients** - These will be clients running in wallets on mobile or browser that will TOFU(Trust On First Use) the stake from the entrypoint clients and verify if the user transactions are voted on by the supermajority. - -#### New RPC Methods -``` -pub async fn get_block_headers( - &self, - slot: Slot, - config: Option>, - ) -> Result -``` - -``` -pub struct BlockHeader { - pub vote_signature: Vec>, - pub validator_identity: Vec>, - pub validator_stake: Vec>, -} -``` -This method will return a BlockHeader, a data structure storing a list of: - - Signatures of validators who voted on that block - - The public keys or 'identities' of the validators who voted on that block. - - The stake amounts of each of those validators. - - -## Impact - -This proposal will improve the overall security and decentralisation of the Solana network allowing users to access the blockchain in a trust -minimised way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. - -## Security Considerations - -### Trust Assumptions -The light client makes certain trust assumptions that reduce reliance on RPC but these assumptions have tradeoffs. The ultra light clients verify if the supermajority has voted on the block, this is could be an issue if the supermajority itself is corrupt and is trying to vote on an invalid transaction. The future iterations of the light client that include data availability sampling and fraud proving should address this issue reduce the trust assumption to single honest node. We also rely on a entrypoint light client that has the snapshot and serves the stake history which is also a point of trust. From 8ee393ee39ed39c01dc99679569df8707c8c2a2d Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 13:50:51 -0400 Subject: [PATCH 23/67] update --- ...023-consensus-and-transaction-inclusion-verification.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 36c318baf..5d090731a 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -12,10 +12,13 @@ created: 2023-05-25 --- ## Summary -Users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. The user -will be running a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) + +Add a new RPC method that returns a proof that a transaction (tx) is included in a specific slot. + +With this new RPC method, users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. This is the first step in implementing a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10). ## Motivation + For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. From 4bb3bc210fe373353588c6dea0720f0fb3b54e97 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 14:04:53 -0400 Subject: [PATCH 24/67] update --- ...023-consensus-and-transaction-inclusion-verification.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 5d090731a..2afe081e1 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -19,12 +19,11 @@ With this new RPC method, users can verify that supermajority has voted on the s ## Motivation -For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. +Currently, for a user to validate whether their transaction was included in a block, it needs to trust the confirmation from an RPC provider. This is a glaring attack vector for malicious RPC actors that could lie to users if it's in their own interest. -This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. - -This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less probable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. +To combat this, a user must run a full node to process the entire ledger and verify the blocks and votes themselves. The downside of this is that its very costly to run a full node which makes inaccessible to everyday users. +One solution to this problem is to use diet clients to verify transaction confirmations without fully trusting the RPC. This SIMD is the first step towards implementing diet clients for Solana and provides a way for users to verify their transaction is included in a block and that the block has recieved a supermajority of votes. ## Alternatives Considered Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: From a64f6ed83aca98316b2aff6a2e21d118bb825332 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 14:07:10 -0400 Subject: [PATCH 25/67] update --- ...0023-consensus-and-transaction-inclusion-verification.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 2afe081e1..b20fbbf3a 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -26,12 +26,8 @@ To combat this, a user must run a full node to process the entire ledger and ver One solution to this problem is to use diet clients to verify transaction confirmations without fully trusting the RPC. This SIMD is the first step towards implementing diet clients for Solana and provides a way for users to verify their transaction is included in a block and that the block has recieved a supermajority of votes. ## Alternatives Considered -Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: -- Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. -- Validators aren't obligated to propagate gossip changes and have reason to not do so as it reduces egress costs. - -We could also just directly make these calls from the client itself but its much faster and more convenient to do it on server side. +None ## New Terminology From 0682f070f8503a591a1596f2c6a3767ca39edf9f Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 14:11:16 -0400 Subject: [PATCH 26/67] update --- ...-and-transaction-inclusion-verification.md | 22 +------------------ 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index b20fbbf3a..83d01ee02 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -31,27 +31,7 @@ None ## New Terminology -TransactionProof: A structure containing necessary information to verify if a transaction was included in the bank hash of a slot. - -## Detailed Design -The protocol interaction will be as follows: -1. User makes a transaction using their wallet in slot N. -2. The client then reads the validator set from gossip and requests vote signatures and stake commitments for slot N to (N+32) given the 32 block depth finality. -3. It first validates whether the validator identities match the set from gossip. -4. Then proceeds to validate the vote signatures. -5. The client also has to sync the epoch stake history from genesis from the entrypoint light clients eventually can be requested from multiple other light clients). -6. Next it checks if the stake weights returned are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. -7. Next it will request the transaction proof using the slot and transaction signature. It will then perform the following checks: - - Verify the entries by checking that hashing the start hash of the slot `num_hash` times results in the same hash as the entry. - - Check if the transaction signature is included in any of the slots entries. - - Check if transaction is included in the poh hash by calculating `hash(prev_hash,hash(transaction_signatures))` and if it matches the entry hash. -8. Reconstruct the bank hash with blockhash(entry hash), parent_hash, accounts_delta_hash and signature_count and check if all votes voted on this bank hash by parsing them. -9. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. - -#### Types of Light Clients -1. **Entrypoint / Hub Clients** - These will be holding the snapshot with the stake history from genesis to current slot and be periodically syncing with the latest epoch. - -3. **Ultra Light Clients** - These will be clients running in wallets on mobile or browser that will TOFU(Trust On First Use) the stake from the entrypoint clients and verify if the user transactions are voted on by the supermajority. +`TransactionProof`: A structure containing necessary information to verify if a transaction was included in the bankhash of a slot. #### New RPC Methods The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct From 13c9b4450c290d7b8a8d960432f3bdb8800bc75c Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 14:22:15 -0400 Subject: [PATCH 27/67] fix --- ...023-consensus-and-transaction-inclusion-verification.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 83d01ee02..d712e4d71 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -138,10 +138,9 @@ minimised way unlike traditionally where users had to fully trust their RPC prov ## Security Considerations ### Trust Assumptions -The light client makes certain trust assumptions that reduce reliance on RPC but these assumptions have tradeoffs. The ultra light clients verify if the supermajority has voted on the block, this is could be an issue if the supermajority itself is corrupt and is trying to vote on an invalid transaction. -## Future Work -The future iterations of the light client that include data availability sampling and fraud proving should address this issue reduce the trust assumption to single honest node. We also rely on a entrypoint light client that has the snapshot and serves the stake history which is also a point of trust. +While this SIMD greatly increases the user's trustlessness required to an RPC, the light client will still needs to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid (incase the supermajority is corrupt). We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving which will only require a single honest full node. -Additionally, while there are improvements and additional features that can be made (as mentioned in [this proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification)), including using merkle trees to compute the blockhash (instead of the current sequential implementation), transaction status codes, and validator set verification, we chose to keep this SIMD self-contained and only add a new RPC method with no changes to the protocol. Optimizations will be left to future SIMDs which will build off of this one. +## Future Work +While there are improvements and additional features that can be made (as mentioned in [this proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification)), including using merkle trees to compute the blockhash (instead of the current sequential implementation), transaction status codes, and validator set verification, we chose to keep this SIMD self-contained and only add a new RPC method with no changes to the protocol. Optimizations will be left to future SIMDs which will build off of this one. From 3f59b532a0f9e4d26a2602ef0ddc6236783ee059 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 14:26:41 -0400 Subject: [PATCH 28/67] spell check --- ...-and-transaction-inclusion-verification.md | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index d712e4d71..8bd66611e 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -13,17 +13,17 @@ created: 2023-05-25 ## Summary -Add a new RPC method that returns a proof that a transaction (tx) is included in a specific slot. +Add a new RPC method that returns proof that a transaction (tx) is included in a specific slot. -With this new RPC method, users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. This is the first step in implementing a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10). +With this new RPC method, users can verify that a supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. This is the first step in implementing a consensus-verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10). ## Motivation -Currently, for a user to validate whether their transaction was included in a block, it needs to trust the confirmation from an RPC provider. This is a glaring attack vector for malicious RPC actors that could lie to users if it's in their own interest. +Currently, for a user to validate whether their transaction was included in a block, it needs to trust the confirmation from an RPC provider. This is a glaring attack vector for malicious RPC actors that could lie to users if it's in their interest. -To combat this, a user must run a full node to process the entire ledger and verify the blocks and votes themselves. The downside of this is that its very costly to run a full node which makes inaccessible to everyday users. +To combat this, a user must run a full node to process the entire ledger and verify the blocks and votes themselves. The downside of this is that it's very costly to run a full node which makes it inaccessible to everyday users. -One solution to this problem is to use diet clients to verify transaction confirmations without fully trusting the RPC. This SIMD is the first step towards implementing diet clients for Solana and provides a way for users to verify their transaction is included in a block and that the block has recieved a supermajority of votes. +One solution to this problem is to use diet clients to verify transaction confirmations without fully trusting the RPC. This SIMD is the first step towards implementing diet clients for Solana and provides a way for users to verify their transaction is included in a block and that the block has received a supermajority of votes. ## Alternatives Considered @@ -31,7 +31,7 @@ None ## New Terminology -`TransactionProof`: A structure containing necessary information to verify if a transaction was included in the bankhash of a slot. +`TransactionProof`: A structure containing the necessary information to verify if a transaction was included in the bankhash of a slot. #### New RPC Methods The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct @@ -82,13 +82,13 @@ pub async fn get_transaction_proof(&self, slot: Slot) -> Result Date: Thu, 25 May 2023 14:27:12 -0400 Subject: [PATCH 29/67] remove old simd --- proposals/00xx-tx-verification.md | 147 ------------------------------ 1 file changed, 147 deletions(-) delete mode 100644 proposals/00xx-tx-verification.md diff --git a/proposals/00xx-tx-verification.md b/proposals/00xx-tx-verification.md deleted file mode 100644 index f0346cfbb..000000000 --- a/proposals/00xx-tx-verification.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -simd: '00XX' -title: Transaction Verification RPC Method -authors: - - Harsh Patel (Tinydancer) - - Anoushk Kharangate (Tinydancer) - - x19 (Tinydancer) -category: Standard -type: Core -status: Draft -created: 2023-05-24 ---- - -## Summary - -Add a new RPC method that returns data to prove a transaction (tx) is included in a specific slot. - -## Motivation - -The first step for Solana light/diet clients is to verify a transaction was processed in a specific slot. This requires a hash path from a transaction's signature to a slot's bankhash as proof (as noted in a [previous diet client SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10)). Since validators sign votes on bankhashes, once we have this proof, we can also verify a transaction has received a supermajority of votes. Current RPC methods don't provide enough information so we need a new RPC method for this. - -## Alternatives Considered - -None - -## New Terminology - -None - -## Detailed Design - -The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct: - -```rust -// new RPC method -pub async fn get_transaction_proof(&self, slot: Slot) -> Result; - -// new RPC struct to verify tx inclusion -pub struct TransactionProof { - // used to verify hash([start_blockhash, entires]) = bank_hash - // also verify the tx_signature of interest is included in an Entry - pub entries: Vec, - pub start_blockhash: Hash, - // used to reconstruct `bank_hash` - pub parent_hash: Hash, - pub accounts_delta_hash: Hash, - pub signature_count_buf: [u8; 8], -} -``` - -All the variables are accessible from the rpc's blockstore and bank_forks variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: - -```rust -pub async fn get_transaction_proof(&self, slot: Slot) -> Result { - // first retrieve all the entries - let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) - // require all of the entries - assert!(is_full) - - let bank_forks = self.bank_forks.read().unwrap(); - let bank = bank_forks.get(slot); - - // get variables used to compute bank hash - let parent_hash = bank.parent_hash(); - let accounts_delta_hash = bank - .rc - .accounts - .accounts_db - .calculate_accounts_delta_hash(slot).0; - let mut signature_count_buf = [0u8; 8]; - LittleEndian::write_u64(&mut signature_count_buf[..], bank.signature_count()); - - // get the start_hash for the slot (will be the last entry's hash from slot-1) - let start_blockhash = self.blockstore.get_slot_entries_with_shred_info(slot-1, 0, false).last().hash; - - Ok(TransactionProof{ - entries, - start_blockhash, - parent_hash, - accounts_delta_hash, - signature_count_buf, - }) -} -``` - -Using the `get_transaction_proof` RPC call, a client can verify a transaction with the following steps: -- first, for a given transaction signature, find the slot which its included in -- call the `get_transaction_proof` RPC method with that slot as input -- verify the `entries` are valid PoH hashes starting with the hash `start_blockhash` -- verify that the transaction signature is included in one of the `entries` -- reconstruct the expected bankhash using the other variables in the struct and the final entry's hash - -Below is client pseudocode for verifying a transaction: - -```rust -let tx_sig = "..."; // tx signature of interest -let slot = 19; // slot which includes the tx - -// call the new RPC -let tx_proof: TransactionProof = get_tx_proof(slot, endpoint); - -// verify the entires are valid -let verified = tx_proof.entries.verify(&tx_proof.start_blockhash); -assert!(verified); - -// verify that the transaction signature is included in one of the `entries` -let mut start_hash = &tx_proof.start_blockhash; -let mut was_verified = false; -for entry in entries.iter() { - // find Entry which includes tx sig - let tx_is_in = entry.transactions.iter().any(|tx| { - tx.signatures.contains(&tx_sig) - }); - if tx_is_in { - // verify Entry includes tx - let hash = next_hash(start_hash, entry.num_hashes, &entry.transactions); - assert!(hash == entry.hash); - was_verified = true; - break; - } - start_hash = &entry.hash; -} -assert!(was_verified); - -// recompute the bank hash -let last_blockhash = entries.last().unwrap().hash; -let bankhash = hashv(&[ - tx_proof.parent_hash.as_ref(), - tx_proof.accounts_delta_hash.as_ref(), - tx_proof.signature_count_buf.as_ref(), - last_blockhash.as_ref() -]); -``` - -Once we computed the expected bankhash, we can parse vote transactions which vote on that bankhash, and assert that a supermajority (>= 2/3 of stake) has voted on it. - -## Future Work - -While there are improvements and additional features that can be made (as mentioned in [this proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification)), including using merkle trees to compute the blockhash (instead of the current sequential implementation), transaction status codes, and validator set verification, we chose to keep this SIMD self-contained and only add a new RPC method with no changes to the protocol. Optimizations will be left to future SIMDs which will build off of this one. - -## Impact - -This proposal will improve the overall security and decentralisation of the Solana network allowing users to verify their transactions in a trustless way, unlike traditionally where users had to fully trust their RPC providers. Since its only a new RPC method, Dapp developers don't have to make any special changes, as wallets and can easily integrate the verification to be compatible with any transaction. - -## Security Considerations - -While this SIMD greatly increases the trustlessness on a single RPC, the light client will still need to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid. We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving. From aaaf9363c15a18c865ce51269ac7f2eda0fa2ae6 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 22:49:43 -0400 Subject: [PATCH 30/67] add merkle proof info --- ...-and-transaction-inclusion-verification.md | 101 ++++++++++++++---- 1 file changed, 82 insertions(+), 19 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 8bd66611e..83f45a520 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -33,29 +33,93 @@ None `TransactionProof`: A structure containing the necessary information to verify if a transaction was included in the bankhash of a slot. +`EntryProof`: An enum representing proof information about an entry of a slot. The enum will either contain a `MerkleEntry` which includes a merkle proof that a transaction is included in its entry hash, or a `PartialEntry` which only contains a hash of its transactions. + #### New RPC Methods -The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct + +The new RPC method would be called `get_transaction_proof` which would take a slot number and a transaction signature as input and return a `TransactionProof` struct + ```rs // new RPC method -pub async fn get_transaction_proof(&self, slot: Slot) -> Result; +pub async fn get_transaction_proof(&self, slot: Slot, signature: Signature) -> Result; // new RPC struct to verify tx inclusion pub struct TransactionProof { - pub entries: Vec, + pub entries: Vec, pub start_blockhash: Hash, pub parent_hash: Hash, pub accounts_delta_hash: Hash, pub signature_count_buf: [u8; 8], } + +// entry proof information +pub enum EntryProof { + MerkleEntry(MerkleEntry), + PartialEntry(PartialEntry), +} + +// only for the entry which contains the transaction of interest +pub struct MerkleEntry { + pub hash: Hash, + pub num_hashes: u64, + pub proof: Proof, // uses solana-merkle-tree's Proof as a merkle proof +} + +pub struct PartialEntry { + pub num_hashes: u64, + pub hash: Hash, + pub transaction_hash: Option, +} ``` + All the variables are accessible from the rpc's blockstore and bank_forks variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: + ```rs -pub async fn get_transaction_proof(&self, slot: Slot) -> Result { +pub async fn get_transaction_proof(&self, slot: Slot, signature: Signature) -> Result { // first retrieve all the entries let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) // require all of the entries assert!(is_full) + // parse the entires to only include the required information for the proof + let mut proof_entries = Vec::with_capacity(entries.len()); + for entry in entries.iter() { + let contains_sig = entry.transactions.iter().any(|tx| { + tx.signatures.contains(&signature) + }); + + let proof_entry = if contains_sig { + // create merkle proof for the entry which contains the tx of interest + let signatures: Vec<_> = entry.transactions + .iter() + .flat_map(|tx| tx.signatures.iter()) + .collect(); + + let merkle_tree = MerkleTree::new(&signatures); + let proof: Proof = merkle_tree.create_proof(&signature).unwrap(); + + EntryProof::MerkleEntry(MerkleEntry { + hash: entry.hash, + num_hashes: entry.num_hashes, + proof, + }) + } else { + // only include the required tx hash of other entries + let tx_hash = if !entry.transactions.is_empty() { + Some(hash_transactions(&entry.transactions)) + } else { + None + }; + + EntryProof::PartialEntry(PartialEntry { + hash: entry.hash, + num_hashes: entry.num_hashes, + transaction_hash: tx_hash, + }) + }; + proof_entries.push(proof_entry); + } + let bank_forks = self.bank_forks.read().unwrap(); let bank = bank_forks.get(slot); @@ -73,7 +137,7 @@ pub async fn get_transaction_proof(&self, slot: Slot) -> Result Result { + // verify merkle proof + let verified = x.proof.verify(tx_sig); + assert!(verified); + + was_verified = true; + break; + }, + _ => {} + }; } assert!(was_verified); From 1afc14b36d6879ee75c6a8a33c82b4bdb6c2e4c3 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 23:04:19 -0400 Subject: [PATCH 31/67] add more merkle info --- .../0023-consensus-and-transaction-inclusion-verification.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 83f45a520..85962f2bd 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -33,7 +33,9 @@ None `TransactionProof`: A structure containing the necessary information to verify if a transaction was included in the bankhash of a slot. -`EntryProof`: An enum representing proof information about an entry of a slot. The enum will either contain a `MerkleEntry` which includes a merkle proof that a transaction is included in its entry hash, or a `PartialEntry` which only contains a hash of its transactions. +`EntryProof`: An enum representing proof information that a transaction is included in an entry of a slot. The enum will either contain a `MerkleEntry` which includes a merkle proof that a transaction is included in its entry hash, or a `PartialEntry` which contains the root hash of its transactions. + +*Note:* Compared to sending the full list of `Entry` structs, this approach includes only the necessary information to validate 1) the hashpath from a transaction signature to an `Entry` and 2) the hashpath from the array of entries to the blockhash. #### New RPC Methods From 617bee7796844feedbd03e386de2095d6eec369f Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Thu, 25 May 2023 23:07:04 -0400 Subject: [PATCH 32/67] up --- .../0023-consensus-and-transaction-inclusion-verification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 85962f2bd..8d9e9ddde 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -162,7 +162,7 @@ let slot = 19; // slot which includes the tx // call the new RPC let tx_proof: TransactionProof = get_tx_proof(slot, endpoint); -// verify the entires are valid +// verify the entires form a valid hashpath let verified = tx_proof.entries.verify(&tx_proof.start_blockhash); assert!(verified); From 6f0eb5849bfff3cda8100726ff3a603ee724a1ff Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Fri, 26 May 2023 11:37:46 +0530 Subject: [PATCH 33/67] update --- ...onsensus-and-transaction-inclusion-verification.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 36c318baf..5636ee2fb 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -12,15 +12,16 @@ created: 2023-05-25 --- ## Summary -Users can verify that supermajority has voted on the slot that their transaction was included in without fully trusting the RPC provider. The user -will be running a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) + +This SIMD describes the overall design and changes required that allows users to verify that supermajority has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. This is the first step in implementing a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) ## Motivation -For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. -This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block. +For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this, mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. + +This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block and that their transaction was included in. -This ensures that at-least the user doesn't have to trust the RPC service that is centralised and can rather trust the supermajority of the network which is less probable to be corrupt than a malicious RPC. However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. +However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. ## Alternatives Considered From 1da4621a3a908776e3d4ca3217a8405b19115af0 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Fri, 26 May 2023 11:46:20 +0530 Subject: [PATCH 34/67] format --- .../0023-consensus-and-transaction-inclusion-verification.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 5636ee2fb..e0b6788a3 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -45,10 +45,10 @@ The protocol interaction will be as follows: 5. The client also has to sync the epoch stake history from genesis from the entrypoint light clients eventually can be requested from multiple other light clients). 6. Next it checks if the stake weights returned are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. 7. Next it will request the transaction proof using the slot and transaction signature. It will then perform the following checks: - - Verify the entries by checking that hashing the start hash of the slot `num_hash` times results in the same hash as the entry. + - Verify the `entries` by checking that hashing the start hash of the slot `num_hash` times results in the same hash as the entry. - Check if the transaction signature is included in any of the slots entries. - Check if transaction is included in the poh hash by calculating `hash(prev_hash,hash(transaction_signatures))` and if it matches the entry hash. -8. Reconstruct the bank hash with blockhash(entry hash), parent_hash, accounts_delta_hash and signature_count and check if all votes voted on this bank hash by parsing them. +8. Reconstruct the bankhash with `blockhash(entry hash)`, `parent_hash`, `accounts_delta_hash` and `signature_count` and check if all votes voted on this bankhash by parsing them. 9. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. #### Types of Light Clients From 400c5275be2da307ee65e4156946d54fbd138e61 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Fri, 26 May 2023 13:22:28 +0530 Subject: [PATCH 35/67] update --- ...-and-transaction-inclusion-verification.md | 58 ++++++++++++++++--- 1 file changed, 50 insertions(+), 8 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index e0b6788a3..1662c799e 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -8,7 +8,7 @@ authors: category: Standard type: Core status: Draft -created: 2023-05-25 +created: 2023-05-26 --- ## Summary @@ -36,6 +36,9 @@ We could also just directly make these calls from the client itself but its much TransactionProof: A structure containing necessary information to verify if a transaction was included in the bank hash of a slot. +`EntryProof`: An enum representing proof information that a transaction is included in an entry of a slot. The enum will either contain a `MerkleEntry` which includes a merkle proof that a transaction is included in its entry hash, or a `PartialEntry` which contains the root hash of its transactions. + +*Note:* Compared to sending the full list of `Entry` structs, this approach includes only the necessary information to validate 1) the hashpath from a transaction signature to an `Entry` and 2) the hashpath from the array of entries to the blockhash. ## Detailed Design The protocol interaction will be as follows: 1. User makes a transaction using their wallet in slot N. @@ -46,7 +49,7 @@ The protocol interaction will be as follows: 6. Next it checks if the stake weights returned are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. 7. Next it will request the transaction proof using the slot and transaction signature. It will then perform the following checks: - Verify the `entries` by checking that hashing the start hash of the slot `num_hash` times results in the same hash as the entry. - - Check if the transaction signature is included in any of the slots entries. + - Check if the transaction signature is included in any of the slots entries and the merkle proof is valid. - Check if transaction is included in the poh hash by calculating `hash(prev_hash,hash(transaction_signatures))` and if it matches the entry hash. 8. Reconstruct the bankhash with `blockhash(entry hash)`, `parent_hash`, `accounts_delta_hash` and `signature_count` and check if all votes voted on this bankhash by parsing them. 9. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. @@ -71,14 +74,53 @@ pub struct TransactionProof { pub signature_count_buf: [u8; 8], } ``` -All the variables are accessible from the rpc's blockstore and bank_forks variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: +All the variables are accessible from the rpc's blockstore and `bank_forks` variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: ```rs -pub async fn get_transaction_proof(&self, slot: Slot) -> Result { +pub async fn get_transaction_proof(&self, slot: Slot, signature: Signature) -> Result { // first retrieve all the entries let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) // require all of the entries assert!(is_full) + // parse the entires to only include the required information for the proof + let mut proof_entries = Vec::with_capacity(entries.len()); + for entry in entries.iter() { + let contains_sig = entry.transactions.iter().any(|tx| { + tx.signatures.contains(&signature) + }); + + let proof_entry = if contains_sig { + // create merkle proof for the entry which contains the tx of interest + let signatures: Vec<_> = entry.transactions + .iter() + .flat_map(|tx| tx.signatures.iter()) + .collect(); + + let merkle_tree = MerkleTree::new(&signatures); + let proof: Proof = merkle_tree.create_proof(&signature).unwrap(); + + EntryProof::MerkleEntry(MerkleEntry { + hash: entry.hash, + num_hashes: entry.num_hashes, + proof, + }) + } else { + // only include the required tx hash of other entries + let tx_hash = if !entry.transactions.is_empty() { + Some(hash_transactions(&entry.transactions)) + } else { + None + }; + + EntryProof::PartialEntry(PartialEntry { + hash: entry.hash, + num_hashes: entry.num_hashes, + transaction_hash: tx_hash, + }) + }; + proof_entries.push(proof_entry); + } + let bank_forks = self.bank_forks.read().unwrap(); let bank = bank_forks.get(slot); @@ -96,7 +138,7 @@ pub async fn get_transaction_proof(&self, slot: Slot) -> Result Result Date: Tue, 30 May 2023 18:29:06 +0530 Subject: [PATCH 36/67] Update 0023-consensus-and-transaction-inclusion-verification.md --- ...-and-transaction-inclusion-verification.md | 228 ++++++++---------- 1 file changed, 98 insertions(+), 130 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index 1662c799e..d6c6958ac 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -8,123 +8,110 @@ authors: category: Standard type: Core status: Draft -created: 2023-05-26 +created: 2023-05-30 --- ## Summary -This SIMD describes the overall design and changes required that allows users to verify that supermajority has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. This is the first step in implementing a consenus verifying client as first described in [SIMD](https://github.com/solana-foundation/solana-improvement-documents/pull/10) +This SIMD describes the overall design and changes required that allows users to verify that supermajority has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. + +This includes two main changes: +1) Adding a new RPC method which provides a proof that a transaction has been included in a slot and +2) Modifying the blockhash to be computed as a Merkle Tree and include transaction statuses + +This SIMD is the first step in implementing a consensus verifying client as first described in [SIMD #10](https://github.com/solana-foundation/solana-improvement-documents/pull/10) and a majority of the changes mentioned in the accepted [Simple Payment and State Verification +proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification). ## Motivation -For a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. To combat this, mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of that being very high cost to run a full node making it less accessible to everyday users, in effect exposing users to potential attacks from malicious nodes. +Currently, for a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. -This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. The SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if supermajority stake actually signed off on a block and that their transaction was included in. +To combat this, mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of this is that it's very costly to run a full node which makes it inaccessible to everyday users,exposing users to potential attacks from malicious nodes. -However it is not impossible, hence the full diet client implementation discusses further steps to counter that and this is only the consensus verifying stage of the client. +This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. This SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if a supermajority of the total stake has voted for a block which includes their transaction. +However, this is only the consensus verifying stage of the client, and with only these changes, the RPC provider can still trick users, hence we also discuss future work that will be implemented in future SIMDs to provide a fully trustless setup. ## Alternatives Considered -Another solution is to use the gossip plane to read votes out of the CRDS optimistically and then confirm by verifying inclusion with the blockhash. The advantage being that no changes are required to the validator to read the votes. However this has a couple of drawbacks: -- Votes have a high probability of getting dropped after being inserted into the CRDS as sometimes validators can vote on older slots after voting on the newer slot. -- Validators aren't obligated to propagate gossip changes and have reason to not do so as it reduces egress costs. -We could also just directly make these calls from the client itself but its much faster and more convenient to do it on server side. +None ## New Terminology TransactionProof: A structure containing necessary information to verify if a transaction was included in the bank hash of a slot. -`EntryProof`: An enum representing proof information that a transaction is included in an entry of a slot. The enum will either contain a `MerkleEntry` which includes a merkle proof that a transaction is included in its entry hash, or a `PartialEntry` which contains the root hash of its transactions. +```rs +// new RPC struct to verify tx inclusion +pub struct TransactionProof { + pub proof: Vec, + + pub parent_hash: Hash, + pub accounts_delta_hash: Hash, + pub signature_count_buf: [u8; 8], +} +``` + +The proof variable will provide a Merkle hashpath from the transaction to the blockhash; which is then used with the other variables to compute the bankhash. -*Note:* Compared to sending the full list of `Entry` structs, this approach includes only the necessary information to validate 1) the hashpath from a transaction signature to an `Entry` and 2) the hashpath from the array of entries to the blockhash. ## Detailed Design + The protocol interaction will be as follows: -1. User makes a transaction using their wallet in slot N. -2. The client then reads the validator set from gossip and requests vote signatures and stake commitments for slot N to (N+32) given the 32 block depth finality. -3. It first validates whether the validator identities match the set from gossip. -4. Then proceeds to validate the vote signatures. -5. The client also has to sync the epoch stake history from genesis from the entrypoint light clients eventually can be requested from multiple other light clients). -6. Next it checks if the stake weights returned are valid with the local stake history that wasy synced from entrypoint and if the stake is >= 67% of the total stake. -7. Next it will request the transaction proof using the slot and transaction signature. It will then perform the following checks: - - Verify the `entries` by checking that hashing the start hash of the slot `num_hash` times results in the same hash as the entry. - - Check if the transaction signature is included in any of the slots entries and the merkle proof is valid. - - Check if transaction is included in the poh hash by calculating `hash(prev_hash,hash(transaction_signatures))` and if it matches the entry hash. -8. Reconstruct the bankhash with `blockhash(entry hash)`, `parent_hash`, `accounts_delta_hash` and `signature_count` and check if all votes voted on this bankhash by parsing them. -9. If all these checks are valid the slot can be marked as confirmed under a supermajority trust assumption. - -#### Types of Light Clients -1. **Entrypoint / Hub Clients** - These will be holding the snapshot with the stake history from genesis to current slot and be periodically syncing with the latest epoch. - -3. **Ultra Light Clients** - These will be clients running in wallets on mobile or browser that will TOFU(Trust On First Use) the stake from the entrypoint clients and verify if the user transactions are voted on by the supermajority. +- A user sends a transaction and it lands in slot N +- The user requests proof that the transaction is included in slot N using the new RPC method +- The user verifies the proof: + - Verifies the entries form a valid hashpath starting with the `start_blockhash` hash and ending with the last entry (which is the blockhash) + - Verifies a MerkleEntry exists and contains a hash path from the transaction signature and its status to the entry’s hash + - Then computes the expected bankhash using the last entries hash, `parent_hash`, `accounts_delta_hash` and `signature_count` +- The user retrieves the epoch’s current validator set and stake amounts from a trusted source (making this step trustless is future work) +- The user requests blocks from slots > N (up to 32 slots past N given the 32 block depth finality) +- The user parses vote transactions from the blocks, verifying their signature, and computing the sum of stake which voted on the bankhash they computed in step 3 +- If the sum of stake is greater than or equal to 2/3 of the total stake then their transaction has been finalized under a supermajority trust assumption (making this assumption require only a single honest validator is also future work) + + +#### Modifying the Blockhash + +We also propose modifying the blockhash computation to +1) Compute the blockhash using a Merkle Tree of entries and +2) Include the status (either succeeding or failing) of each transaction + +To produce the blockhash for a slot, the current implementation hashes Entries in a sequential way which requires a O(N) proof size to provide a hashpath from a transaction to a blockhash. Implementing change 1) would allow for a more efficient O(log(N)) proof size. The current Entry implementation already hashes transaction signatures using a Merkle Tree to get the Entry’s hash, so this change would only modify how the Entry’s hashes are hashed together to get a blockhash. + +For change 2), the blockhash is currently computed using only transaction signatures, and does not include transaction statuses which means we are unable to prove if a transaction has succeeded or failed. Implementing 2) would enable verifying a transactions status, as mentioned in the [accepted proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification#transaction-merkle) and in a [previous github issue](https://github.com/solana-labs/solana/issues/7053)). + +Fig #1 shows an example hashpath from a transaction and its signature to a bankhash with both of the proposed changes implemented. +
+ Figure showing the hashpath to the transaction +
Fig #1
+
+ #### New RPC Methods -The new RPC method would be called `get_transaction_proof` which would take a slot number as input and return a `TransactionProof` struct -```rs -// new RPC method -pub async fn get_transaction_proof(&self, slot: Slot) -> Result; -// new RPC struct to verify tx inclusion -pub struct TransactionProof { - pub entries: Vec, - pub start_blockhash: Hash, - pub parent_hash: Hash, - pub accounts_delta_hash: Hash, - pub signature_count_buf: [u8; 8], -} +The new RPC method would be called `get_transaction_proof` which would take a transaction signature as input and return a `TransactionProof` struct + +```rs +// new RPC method +pub async fn get_transaction_proof(&self, signature: Signature) -> Result; ``` -All the variables are accessible from the rpc's blockstore and `bank_forks` variables, so no changes to the rest of the codebase will be required. The following is psuedocode of the RPC method: + +Below is psuedocode of the RPC method: + ```rs -pub async fn get_transaction_proof(&self, slot: Slot, signature: Signature) -> Result { - // first retrieve all the entries - let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) - // require all of the entries +pub async fn get_transaction_proof(&self, signature: Signature) -> Result { + // first retrieve all the entries + let slot = self.get_slot_of_signature(&signature); + let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) + // require all of the entries assert!(is_full) - // parse the entires to only include the required information for the proof - let mut proof_entries = Vec::with_capacity(entries.len()); - for entry in entries.iter() { - let contains_sig = entry.transactions.iter().any(|tx| { - tx.signatures.contains(&signature) - }); - - let proof_entry = if contains_sig { - // create merkle proof for the entry which contains the tx of interest - let signatures: Vec<_> = entry.transactions - .iter() - .flat_map(|tx| tx.signatures.iter()) - .collect(); - - let merkle_tree = MerkleTree::new(&signatures); - let proof: Proof = merkle_tree.create_proof(&signature).unwrap(); - - EntryProof::MerkleEntry(MerkleEntry { - hash: entry.hash, - num_hashes: entry.num_hashes, - proof, - }) - } else { - // only include the required tx hash of other entries - let tx_hash = if !entry.transactions.is_empty() { - Some(hash_transactions(&entry.transactions)) - } else { - None - }; - - EntryProof::PartialEntry(PartialEntry { - hash: entry.hash, - num_hashes: entry.num_hashes, - transaction_hash: tx_hash, - }) - }; - proof_entries.push(proof_entry); - } + // compute the Merkle hashpath from the signature and status to the blockhash + let proof = entries.get_merkle_proof(signature); + // get variables used to compute the bankhash let bank_forks = self.bank_forks.read().unwrap(); let bank = bank_forks.get(slot); - // get variables used to compute bank hash let parent_hash = bank.parent_hash(); let accounts_delta_hash = bank .rc @@ -134,79 +121,60 @@ pub async fn get_transaction_proof(&self, slot: Slot, signature: Signature) -> R let mut signature_count_buf = [0u8; 8]; LittleEndian::write_u64(&mut signature_count_buf[..], bank.signature_count()); - // get the start_hash for the slot (will be the last entry's hash from slot-1) - let start_blockhash = self.blockstore.get_slot_entries_with_shred_info(slot-1, 0, false).last().hash; - - Ok(TransactionProof{ - entries: proof_entries, - start_blockhash, - parent_hash, - accounts_delta_hash, + Ok(TransactionProof{ + proof, + parent_hash, + accounts_delta_hash, signature_count_buf, }) } ``` -Using the `get_transaction_proof` RPC call, a client can verify a transaction with the following steps: -- first, for a given transaction signature, find the slot which it is included in -- call the `get_transaction_proof` RPC method with that slot and transaction signature as input -- verify the `entries` are valid PoH hashes starting with the hash `start_blockhash` -- verify that the transaction signature is included in one of the `entries` and the merkle proof is valid -- reconstruct the expected bankhash using the other variables in the struct and the final entry's hash Below is client pseudocode for verifying a transaction: -```rust + +```rust let tx_sig = "..."; // tx signature of interest let slot = 19; // slot which includes the tx // call the new RPC -let tx_proof: TransactionProof = get_tx_proof(slot, endpoint); +let tx_proof: TransactionProof = get_tx_proof(&tx_sig, endpoint); -// verify the entires are valid -let verified = tx_proof.entries.verify(&tx_proof.start_blockhash); +// verify the transaction signature proof is valid and status is success +let leaf = hash_leaf!([tx_sig, TxStatus::Success]); +let verified = tx_proof.proof.verify_path(&leaf); assert!(verified); -// verify that the transaction signature is included in one of the `entries` -let mut start_hash = &tx_proof.start_blockhash; -let mut was_verified = false; -for entry in entries.iter() { - // find Entry which includes tx sig - let tx_is_in = entry.transactions.iter().any(|tx| { - tx.signatures.contains(&tx_sig) - }); - if tx_is_in { - // verify Entry includes tx - let hash = next_hash(start_hash, entry.num_hashes, &entry.transactions); - assert!(hash == entry.hash); - was_verified = true; - break; - } - start_hash = &entry.hash; -} -assert!(was_verified); +// compute the blockhash +let last_blockhash = tx_proof.proof.get_root(); -// recompute the bank hash -let last_blockhash = entries.last().unwrap().hash; +// compute the expected bankhash let bankhash = hashv(&[ tx_proof.parent_hash.as_ref(), tx_proof.accounts_delta_hash.as_ref(), - tx_proof.signature_count_buf.as_ref(), + tx_proof.signature_count_buf.as_ref(), last_blockhash.as_ref() ]); + +// parse vote transactions and stake amounts on expected bankhash +let (voted_stake_amount, total_stake_amount) = parse_votes_from_blocks(slot, bankhash) + +// validate supermajority voted for expected bankhash +let supermajority_verified = 3 * voted_stake_amount >= 2 * total_stake_amount; +assert!(supermajority_verified) ``` -Once we computed the expected bankhash, we can parse vote transactions which vote on that bankhash, and assert that a supermajority (>= 2/3 of stake) has voted on it. + ## Impact This proposal will improve the overall security and decentralisation of the Solana network allowing users to access the blockchain in a trust minimised way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. -## Security Considerations +## Impact -### Trust Assumptions -The light client makes certain trust assumptions that reduce reliance on RPC but these assumptions have tradeoffs. The ultra light clients verify if the supermajority has voted on the block, this is could be an issue if the supermajority itself is corrupt and is trying to vote on an invalid transaction. +This proposal will improve the overall security and decentralization of the Solana network allowing users to access the blockchain in a trust minimized way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. -## Future Work -The future iterations of the light client that include data availability sampling and fraud proving should address this issue reduce the trust assumption to single honest node. We also rely on a entrypoint light client that has the snapshot and serves the stake history which is also a point of trust. +## Security Considerations -Additionally, while there are improvements and additional features that can be made (as mentioned in [this proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification)), including using merkle trees to compute the blockhash (instead of the current sequential implementation), transaction status codes, and validator set verification, we chose to keep this SIMD self-contained and only add a new RPC method with no changes to the protocol. Optimizations will be left to future SIMDs which will build off of this one. +### Trust Assumptions and Future Work +While this SIMD greatly reduces the user's trust in an RPC, the light client will still need to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid (in case the supermajority is corrupt). We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving which will only require a single honest full node. From 0248900b0c74b6c523a083a36a2c00198e465115 Mon Sep 17 00:00:00 2001 From: 0xNineteen <0x39015319@gmail.com> Date: Tue, 30 May 2023 09:12:43 -0400 Subject: [PATCH 37/67] update --- ...-and-transaction-inclusion-verification.md | 33 +++++++------------ 1 file changed, 11 insertions(+), 22 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index d6c6958ac..01987d2e4 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -13,7 +13,7 @@ created: 2023-05-30 ## Summary -This SIMD describes the overall design and changes required that allows users to verify that supermajority has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. +This SIMD describes the overall design and changes required that allows users to verify that a supermajority of the validators has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. This includes two main changes: 1) Adding a new RPC method which provides a proof that a transaction has been included in a slot and @@ -26,17 +26,14 @@ proposal](https://docs.solana.com/proposals/simple-payment-and-state-verificatio Currently, for a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. -To combat this, mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of this is that it's very costly to run a full node which makes it inaccessible to everyday users,exposing users to potential attacks from malicious nodes. +To combat this, mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of this is that it's very costly to run a full node which makes it inaccessible to everyday users,exposing users to potential attacks from malicious nodes. This is where diet clients come in, users run the client to verify the confirmation of their transaction without trusting the RPC. -This is where diet clients come in, users run the client to verify confirmation of their transaction without trusting the RPC. This SIMD is the first step towards implementing the diet client by proposing a small change to the rpc service that allows the client to validate if a supermajority of the total stake has voted for a block which includes their transaction. - -However, this is only the consensus verifying stage of the client, and with only these changes, the RPC provider can still trick users, hence we also discuss future work that will be implemented in future SIMDs to provide a fully trustless setup. +However, this is only the consensus verifying stage of the client, and with only these changes, the RPC provider can still trick users, hence we also discuss future work that will be implemented in future SIMDs to provide a fully trustless setup. ## Alternatives Considered None - ## New Terminology TransactionProof: A structure containing necessary information to verify if a transaction was included in the bank hash of a slot. @@ -44,8 +41,7 @@ TransactionProof: A structure containing necessary information to verify if a tr ```rs // new RPC struct to verify tx inclusion pub struct TransactionProof { - pub proof: Vec, - + pub proof: Vec, pub parent_hash: Hash, pub accounts_delta_hash: Hash, pub signature_count_buf: [u8; 8], @@ -59,10 +55,9 @@ The proof variable will provide a Merkle hashpath from the transaction to the bl The protocol interaction will be as follows: - A user sends a transaction and it lands in slot N - The user requests proof that the transaction is included in slot N using the new RPC method -- The user verifies the proof: - - Verifies the entries form a valid hashpath starting with the `start_blockhash` hash and ending with the last entry (which is the blockhash) - - Verifies a MerkleEntry exists and contains a hash path from the transaction signature and its status to the entry’s hash - - Then computes the expected bankhash using the last entries hash, `parent_hash`, `accounts_delta_hash` and `signature_count` +- The user verifies that the proof includes the transaction signature and a success status +- The user constructs the merkle tree to derive the root hash +- The user computes the expected bankhash using the root hash, `parent_hash`, `accounts_delta_hash` and `signature_count` - The user retrieves the epoch’s current validator set and stake amounts from a trusted source (making this step trustless is future work) - The user requests blocks from slots > N (up to 32 slots past N given the 32 block depth finality) - The user parses vote transactions from the blocks, verifying their signature, and computing the sum of stake which voted on the bankhash they computed in step 3 @@ -71,9 +66,9 @@ The protocol interaction will be as follows: #### Modifying the Blockhash -We also propose modifying the blockhash computation to +We propose modifying the blockhash computation to 1) Compute the blockhash using a Merkle Tree of entries and -2) Include the status (either succeeding or failing) of each transaction +2) Include the status (either succeeding or failing) of each transaction in each Entry leaf To produce the blockhash for a slot, the current implementation hashes Entries in a sequential way which requires a O(N) proof size to provide a hashpath from a transaction to a blockhash. Implementing change 1) would allow for a more efficient O(log(N)) proof size. The current Entry implementation already hashes transaction signatures using a Merkle Tree to get the Entry’s hash, so this change would only modify how the Entry’s hashes are hashed together to get a blockhash. @@ -88,7 +83,7 @@ Fig #1 shows an example hashpath from a transaction and its signature to a bankh #### New RPC Methods -The new RPC method would be called `get_transaction_proof` which would take a transaction signature as input and return a `TransactionProof` struct +We also need a new RPC method to provide proofs to clients. This method would be called `get_transaction_proof` which would take a transaction signature as input and return a `TransactionProof` struct ```rs // new RPC method @@ -106,7 +101,7 @@ pub async fn get_transaction_proof(&self, signature: Signature) -> Result= 2 * total_stake_amount; assert!(supermajority_verified) ``` - -## Impact - -This proposal will improve the overall security and decentralisation of the Solana network allowing users to access the blockchain in a trust -minimised way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. - ## Impact This proposal will improve the overall security and decentralization of the Solana network allowing users to access the blockchain in a trust minimized way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. From e44a30377c24cdeb9bda1f5a7f213cc285951dfd Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Tue, 30 May 2023 18:44:39 +0530 Subject: [PATCH 38/67] Update 0023-consensus-and-transaction-inclusion-verification.md --- .../0023-consensus-and-transaction-inclusion-verification.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0023-consensus-and-transaction-inclusion-verification.md index d6c6958ac..c408ca1fc 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0023-consensus-and-transaction-inclusion-verification.md @@ -169,10 +169,6 @@ assert!(supermajority_verified) This proposal will improve the overall security and decentralisation of the Solana network allowing users to access the blockchain in a trust minimised way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. -## Impact - -This proposal will improve the overall security and decentralization of the Solana network allowing users to access the blockchain in a trust minimized way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. - ## Security Considerations ### Trust Assumptions and Future Work From 887c05312ded246601637f12900be7b33bec0e56 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Tue, 30 May 2023 18:57:58 +0530 Subject: [PATCH 39/67] Update and rename 0023-consensus-and-transaction-inclusion-verification.md to 0052-consensus-and-transaction-inclusion-verification.md --- ...=> 0052-consensus-and-transaction-inclusion-verification.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{0023-consensus-and-transaction-inclusion-verification.md => 0052-consensus-and-transaction-inclusion-verification.md} (99%) diff --git a/proposals/0023-consensus-and-transaction-inclusion-verification.md b/proposals/0052-consensus-and-transaction-inclusion-verification.md similarity index 99% rename from proposals/0023-consensus-and-transaction-inclusion-verification.md rename to proposals/0052-consensus-and-transaction-inclusion-verification.md index 01987d2e4..81891a361 100644 --- a/proposals/0023-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0052-consensus-and-transaction-inclusion-verification.md @@ -1,5 +1,5 @@ --- -simd: '0023' +simd: '0052' title: Consensus and Transaction Inclusion Verification authors: - Harsh Patel (Tinydancer) From 9d70e390b2b6cb4040b8a25a04d13b12d962f2dc Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 3 Jun 2023 22:01:24 +0530 Subject: [PATCH 40/67] fix lint --- ...-and-transaction-inclusion-verification.md | 60 ++++++++++++------- 1 file changed, 39 insertions(+), 21 deletions(-) diff --git a/proposals/0052-consensus-and-transaction-inclusion-verification.md b/proposals/0052-consensus-and-transaction-inclusion-verification.md index 81891a361..36abb1bb8 100644 --- a/proposals/0052-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0052-consensus-and-transaction-inclusion-verification.md @@ -13,22 +13,38 @@ created: 2023-05-30 ## Summary -This SIMD describes the overall design and changes required that allows users to verify that a supermajority of the validators has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. +This SIMD describes the overall design and changes required that allows users to +verify that a supermajority of the validators has voted on the slot that their +transaction was included in the block without fully trusting the RPC provider. -This includes two main changes: -1) Adding a new RPC method which provides a proof that a transaction has been included in a slot and -2) Modifying the blockhash to be computed as a Merkle Tree and include transaction statuses +This includes two main changes: -This SIMD is the first step in implementing a consensus verifying client as first described in [SIMD #10](https://github.com/solana-foundation/solana-improvement-documents/pull/10) and a majority of the changes mentioned in the accepted [Simple Payment and State Verification -proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification). +1) Adding a new RPC method which provides a proof that a transaction has been + included in a slot +2) Modifying the blockhash to be computed as a Merkle Tree and + includes transaction statuses + +This SIMD is the first step in implementing a consensus verifying client as first +described in [SIMD #10](https://github.com/solana-foundation/solana-improvement-documents/pull/10) +and a majority of the changes mentioned in +the accepted [Simple Payment and State Verification Proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification). ## Motivation -Currently, for a user to validate whether their transaction is valid and included in a block it needs to trust the confirmation from the RPC. This has been a glaring attack vector for malicious actors that could lie to users if it's in their own interest. +Currently, for a user to validate whether their transaction is valid and included +in a block it needs to trust the confirmation from the RPC. This has been a glaring +attack vector for malicious actors that could lie to users if it's in their own interest. -To combat this, mature entities like exchanges run full nodes that process the entire ledger and can verify entire blocks. The downside of this is that it's very costly to run a full node which makes it inaccessible to everyday users,exposing users to potential attacks from malicious nodes. This is where diet clients come in, users run the client to verify the confirmation of their transaction without trusting the RPC. +To combat this, mature entities like exchanges run full nodes that process the +entire ledger and can verify entire blocks. The downside of this is that it's +very costly to run a full node which makes it inaccessible to everyday users, +exposing users to potential attacks from malicious nodes. +This is where diet clients come in, users run the client to verify +the confirmation of their transaction without trusting the RPC. -However, this is only the consensus verifying stage of the client, and with only these changes, the RPC provider can still trick users, hence we also discuss future work that will be implemented in future SIMDs to provide a fully trustless setup. +However, this is only the consensus verifying stage of the client, and with only +these changes, the RPC provider can still trick users, hence we also discuss future +work that will be implemented in futureSIMDs to provide a fully trustless setup. ## Alternatives Considered @@ -36,7 +52,8 @@ None ## New Terminology -TransactionProof: A structure containing necessary information to verify if a transaction was included in the bank hash of a slot. +TransactionProof: A structure containing necessary information to verify if a +transaction was included in the bank hash of a slot. ```rs // new RPC struct to verify tx inclusion @@ -48,26 +65,28 @@ pub struct TransactionProof { } ``` -The proof variable will provide a Merkle hashpath from the transaction to the blockhash; which is then used with the other variables to compute the bankhash. +The proof variable will provide a Merkle hashpath from the transaction to the blockhash; +which is then used with the other variables to compute the bankhash. ## Detailed Design The protocol interaction will be as follows: + - A user sends a transaction and it lands in slot N -- The user requests proof that the transaction is included in slot N using the new RPC method -- The user verifies that the proof includes the transaction signature and a success status +- The user requests proof that the transaction is included in slot N using the new RPC method +- The user verifies that the proof includes the transaction signature and a success status - The user constructs the merkle tree to derive the root hash - The user computes the expected bankhash using the root hash, `parent_hash`, `accounts_delta_hash` and `signature_count` -- The user retrieves the epoch’s current validator set and stake amounts from a trusted source (making this step trustless is future work) -- The user requests blocks from slots > N (up to 32 slots past N given the 32 block depth finality) -- The user parses vote transactions from the blocks, verifying their signature, and computing the sum of stake which voted on the bankhash they computed in step 3 -- If the sum of stake is greater than or equal to 2/3 of the total stake then their transaction has been finalized under a supermajority trust assumption (making this assumption require only a single honest validator is also future work) +- The user retrieves the epoch’s current validator set and stake amounts from a trusted source (making this step trustless is future work) +- The user requests blocks from slots > N (up to 32 slots past N given the 32 block depth finality) +- The user parses vote transactions from the blocks, verifying their signature, and computing the sum of stake which voted on the bankhash they computed in step 3 +- If the sum of stake is greater than or equal to 2/3 of the total stake then their transaction has been finalized under a supermajority trust assumption (making this assumption require only a single honest validator is also future work) +### Modifying the Blockhash -#### Modifying the Blockhash +We propose modifying the blockhash computation to: -We propose modifying the blockhash computation to -1) Compute the blockhash using a Merkle Tree of entries and +1) Compute the blockhash using a Merkle Tree of entries 2) Include the status (either succeeding or failing) of each transaction in each Entry leaf To produce the blockhash for a slot, the current implementation hashes Entries in a sequential way which requires a O(N) proof size to provide a hashpath from a transaction to a blockhash. Implementing change 1) would allow for a more efficient O(log(N)) proof size. The current Entry implementation already hashes transaction signatures using a Merkle Tree to get the Entry’s hash, so this change would only modify how the Entry’s hashes are hashed together to get a blockhash. @@ -80,7 +99,6 @@ Fig #1 shows an example hashpath from a transaction and its signature to a bankh
Fig #1
- #### New RPC Methods We also need a new RPC method to provide proofs to clients. This method would be called `get_transaction_proof` which would take a transaction signature as input and return a `TransactionProof` struct From 0023e7b2efa5260f4e5944978a52772d798a86f6 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 3 Jun 2023 22:09:15 +0530 Subject: [PATCH 41/67] fix lint --- ...-and-transaction-inclusion-verification.md | 69 ++++++++++++++----- 1 file changed, 51 insertions(+), 18 deletions(-) diff --git a/proposals/0052-consensus-and-transaction-inclusion-verification.md b/proposals/0052-consensus-and-transaction-inclusion-verification.md index 36abb1bb8..c9990cd66 100644 --- a/proposals/0052-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0052-consensus-and-transaction-inclusion-verification.md @@ -73,27 +73,47 @@ which is then used with the other variables to compute the bankhash. The protocol interaction will be as follows: - A user sends a transaction and it lands in slot N -- The user requests proof that the transaction is included in slot N using the new RPC method -- The user verifies that the proof includes the transaction signature and a success status +- The user requests proof that the transaction is included in slot N using + the new RPC method +- The user verifies that the proof includes the transaction signature + and a success status - The user constructs the merkle tree to derive the root hash -- The user computes the expected bankhash using the root hash, `parent_hash`, `accounts_delta_hash` and `signature_count` -- The user retrieves the epoch’s current validator set and stake amounts from a trusted source (making this step trustless is future work) -- The user requests blocks from slots > N (up to 32 slots past N given the 32 block depth finality) -- The user parses vote transactions from the blocks, verifying their signature, and computing the sum of stake which voted on the bankhash they computed in step 3 -- If the sum of stake is greater than or equal to 2/3 of the total stake then their transaction has been finalized under a supermajority trust assumption (making this assumption require only a single honest validator is also future work) +- The user computes the expected bankhash using the root hash, `parent_hash`, + `accounts_delta_hash` and `signature_count` +- The user retrieves the epoch’s current validator set and stake amounts from + a trusted source (making this step trustless is future work) +- The user requests blocks from slots > N (up to 32 slots past N given the + 32 block depth finality) +- The user parses vote transactions from the blocks, verifying their signature, + and computing the sum of stake which voted on + the bankhash they computed in step 3 +- If the sum of stake is greater than or equal to 2/3 of the total stake then + their transaction has been finalized under a supermajority trust assumption + (making this assumption require only a single honest validator is also future work) ### Modifying the Blockhash We propose modifying the blockhash computation to: 1) Compute the blockhash using a Merkle Tree of entries -2) Include the status (either succeeding or failing) of each transaction in each Entry leaf - -To produce the blockhash for a slot, the current implementation hashes Entries in a sequential way which requires a O(N) proof size to provide a hashpath from a transaction to a blockhash. Implementing change 1) would allow for a more efficient O(log(N)) proof size. The current Entry implementation already hashes transaction signatures using a Merkle Tree to get the Entry’s hash, so this change would only modify how the Entry’s hashes are hashed together to get a blockhash. - -For change 2), the blockhash is currently computed using only transaction signatures, and does not include transaction statuses which means we are unable to prove if a transaction has succeeded or failed. Implementing 2) would enable verifying a transactions status, as mentioned in the [accepted proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification#transaction-merkle) and in a [previous github issue](https://github.com/solana-labs/solana/issues/7053)). - -Fig #1 shows an example hashpath from a transaction and its signature to a bankhash with both of the proposed changes implemented. +2) Include the status (either succeeding or failing) of each transaction + in each Entry leaf + +To produce the blockhash for a slot, the current implementation hashes Entries in +a sequential way which requires a O(N) proof size to provide a hashpath from a +transaction to a blockhash. Implementing change 1) would allow for a more efficient +O(log(N)) proof size. The current Entry implementation already hashes transaction +signatures using a Merkle Tree to get the Entry’s hash, so this change would only +modify how the Entry’s hashes are hashed together to get a blockhash. + +For change 2), the blockhash is currently computed using only transaction signatures, +and does not include transaction statuses which means we are unable to prove if +a transaction has succeeded or failed. Implementing 2) would enable verifying +a transactions status, as mentioned in the [accepted proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification#transaction-merkle) +and in a [previous github issue](https://github.com/solana-labs/solana/issues/7053)). + +Fig #1 shows an example hashpath from a transaction and its signature to a +bankhash with both of the proposed changes implemented.
Figure showing the hashpath to the transaction
Fig #1
@@ -101,7 +121,9 @@ Fig #1 shows an example hashpath from a transaction and its signature to a bankh #### New RPC Methods -We also need a new RPC method to provide proofs to clients. This method would be called `get_transaction_proof` which would take a transaction signature as input and return a `TransactionProof` struct +We also need a new RPC method to provide proofs to clients. This method would be +called `get_transaction_proof` which would take a transaction signature as input +and return a `TransactionProof` struct ```rs // new RPC method @@ -114,7 +136,8 @@ Below is psuedocode of the RPC method: pub async fn get_transaction_proof(&self, signature: Signature) -> Result { // first retrieve all the entries let slot = self.get_slot_of_signature(&signature); - let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) + let (entries, _, is_full) = + self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) // require all of the entries assert!(is_full) @@ -178,10 +201,20 @@ assert!(supermajority_verified) ## Impact -This proposal will improve the overall security and decentralization of the Solana network allowing users to access the blockchain in a trust minimized way unlike traditionally where users had to fully trust their RPC providers. Dapp developers don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. +This proposal will improve the overall security and decentralization of the Solana +network allowing users to access the blockchain in a trust minimized way unlike +traditionally where users had to fully trust their RPC providers. Dapp developers +don't have to make any changes as wallets can easily integrate the client making +it compatible with any dapp. ## Security Considerations ### Trust Assumptions and Future Work -While this SIMD greatly reduces the user's trust in an RPC, the light client will still need to make certain trust assumptions. This includes finding a trusted source for the validator set per epoch (including their pubkeys and stake weights) and trusting that all transactions are valid (in case the supermajority is corrupt). We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving which will only require a single honest full node. +While this SIMD greatly reduces the user's trust in an RPC, the light client will + still need to make certain trust assumptions. This includes finding a trusted + source for the validator set per epoch (including their pubkeys and stake weights) + and trusting that all transactions are valid (in case the supermajority is corrupt). + We plan to solve these problems in future SIMDs to provide a full trustless setup + including data availability sampling and fraud proving which will only require a + single honest full node. From c50f2b6fc1602b9ffd40f493ed5c68c6a4f966ac Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 3 Jun 2023 22:15:45 +0530 Subject: [PATCH 42/67] fix lint --- ...onsensus-and-transaction-inclusion-verification.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/proposals/0052-consensus-and-transaction-inclusion-verification.md b/proposals/0052-consensus-and-transaction-inclusion-verification.md index c9990cd66..b4757a2d3 100644 --- a/proposals/0052-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0052-consensus-and-transaction-inclusion-verification.md @@ -114,10 +114,7 @@ and in a [previous github issue](https://github.com/solana-labs/solana/issues/70 Fig #1 shows an example hashpath from a transaction and its signature to a bankhash with both of the proposed changes implemented. -
- Figure showing the hashpath to the transaction -
Fig #1
-
+![Fig #1](https://github.com/tinydancer-io/solana-improvement-documents/assets/32778608/5370950d-e27b-4c1b-9f04-6e9164789e65) #### New RPC Methods @@ -133,16 +130,16 @@ pub async fn get_transaction_proof(&self, signature: Signature) -> Result Result { +async fn get_transaction_proof(&self, s: Signature) -> Result { // first retrieve all the entries - let slot = self.get_slot_of_signature(&signature); + let slot = self.get_slot_of_signature(&s); let (entries, _, is_full) = self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) // require all of the entries assert!(is_full) // compute the Merkle hashpath from the signature and status to the blockhash - let proof = entries.get_merkle_proof(&signature); + let proof = entries.get_merkle_proof(&s); // get variables used to compute the bankhash let bank_forks = self.bank_forks.read().unwrap(); From 489f2f15776a48c122c189bf7ea1981f1a6c8d07 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 3 Jun 2023 22:18:44 +0530 Subject: [PATCH 43/67] fix simd lint --- ....md => 0052-consensus-and-transaction-proof-verification.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{0052-consensus-and-transaction-inclusion-verification.md => 0052-consensus-and-transaction-proof-verification.md} (99%) diff --git a/proposals/0052-consensus-and-transaction-inclusion-verification.md b/proposals/0052-consensus-and-transaction-proof-verification.md similarity index 99% rename from proposals/0052-consensus-and-transaction-inclusion-verification.md rename to proposals/0052-consensus-and-transaction-proof-verification.md index b4757a2d3..2db30cd93 100644 --- a/proposals/0052-consensus-and-transaction-inclusion-verification.md +++ b/proposals/0052-consensus-and-transaction-proof-verification.md @@ -1,6 +1,6 @@ --- simd: '0052' -title: Consensus and Transaction Inclusion Verification +title: Consensus and Transaction Proof Verification authors: - Harsh Patel (Tinydancer) - Anoushk Kharangate (Tinydancer) From a4aeb7f3a55531c573e3edbbe808b544beb9dcf1 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sun, 4 Jun 2023 17:33:03 +0530 Subject: [PATCH 44/67] fix simd lint --- .../0052-consensus-and-transaction-proof-verification.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/0052-consensus-and-transaction-proof-verification.md b/proposals/0052-consensus-and-transaction-proof-verification.md index 2db30cd93..cf8335f4b 100644 --- a/proposals/0052-consensus-and-transaction-proof-verification.md +++ b/proposals/0052-consensus-and-transaction-proof-verification.md @@ -8,7 +8,8 @@ authors: category: Standard type: Core status: Draft -created: 2023-05-30 +created: 2023-05-30 +feature: (fill in with feature tracking issues once accepted) --- ## Summary @@ -215,3 +216,5 @@ While this SIMD greatly reduces the user's trust in an RPC, the light client wil We plan to solve these problems in future SIMDs to provide a full trustless setup including data availability sampling and fraud proving which will only require a single honest full node. + +## Backwards Compatibility *(Optional)* From 19d9bbe4318dbcc2bb0c1cb1f0d9fe59e1a4b55e Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Sun, 25 Jun 2023 14:40:25 +0530 Subject: [PATCH 45/67] Create 00xx-add-transaction-receipts-in-crds.md initial draft --- .../00xx-add-transaction-receipts-in-crds.md | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 proposals/00xx-add-transaction-receipts-in-crds.md diff --git a/proposals/00xx-add-transaction-receipts-in-crds.md b/proposals/00xx-add-transaction-receipts-in-crds.md new file mode 100644 index 000000000..5a5f8a306 --- /dev/null +++ b/proposals/00xx-add-transaction-receipts-in-crds.md @@ -0,0 +1,58 @@ +--- +simd: 'XXXX' +title: Add Transaction Receipts to CRDS in Gossip +authors: + - Anoushk Kharangate (Tinydancer) + - Harsh Patel (Tinydancer) +category: Standard +type: Core/Networking +status: Draft +created: (fill me in with today's date, 2023-06-25) +--- + +## Summary + +A brief summary of what the feature is. + +## Motivation + +Why are we doing this? What use cases does it support? What is the expected +outcome? + +## Alternatives Considered + +What alternative designs were considered and what pros/cons does this feature +have relative to them? + +## New Terminology + +Is there any new terminology introduced with this proposal? + +## Detailed Design + +Explain the feature as if it was already implemented and you're explaining it +to another Solana core contributor. The generally means: + +- Explain the proposed change and how it works +- Where the feature fits in to the runtime, core, or relevant sub-system +- How this feature was/could be implemented +- Interaction with other features +- Edge cases + +## Impact + +How will the implemented proposal impacts dapp developers, validators, and core contributors? + +## Security Considerations + +What security implications/considerations come with implementing this feature? +Are there any implementation-specific guidance or pitfalls? + +## Drawbacks *(Optional)* + +Why should we not do this? + +## Backwards Compatibility *(Optional)* + +Does the feature introduce any breaking changes? All incompatibilities and +consequences should be listed. From 6a0b744114cdafeec85d17d3e3d93c8b75557b64 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Mon, 26 Jun 2023 14:22:52 +0530 Subject: [PATCH 46/67] Revert "Create 00xx-add-transaction-receipts-in-crds.md" This reverts commit 19d9bbe4318dbcc2bb0c1cb1f0d9fe59e1a4b55e. --- .../00xx-add-transaction-receipts-in-crds.md | 58 ------------------- 1 file changed, 58 deletions(-) delete mode 100644 proposals/00xx-add-transaction-receipts-in-crds.md diff --git a/proposals/00xx-add-transaction-receipts-in-crds.md b/proposals/00xx-add-transaction-receipts-in-crds.md deleted file mode 100644 index 5a5f8a306..000000000 --- a/proposals/00xx-add-transaction-receipts-in-crds.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -simd: 'XXXX' -title: Add Transaction Receipts to CRDS in Gossip -authors: - - Anoushk Kharangate (Tinydancer) - - Harsh Patel (Tinydancer) -category: Standard -type: Core/Networking -status: Draft -created: (fill me in with today's date, 2023-06-25) ---- - -## Summary - -A brief summary of what the feature is. - -## Motivation - -Why are we doing this? What use cases does it support? What is the expected -outcome? - -## Alternatives Considered - -What alternative designs were considered and what pros/cons does this feature -have relative to them? - -## New Terminology - -Is there any new terminology introduced with this proposal? - -## Detailed Design - -Explain the feature as if it was already implemented and you're explaining it -to another Solana core contributor. The generally means: - -- Explain the proposed change and how it works -- Where the feature fits in to the runtime, core, or relevant sub-system -- How this feature was/could be implemented -- Interaction with other features -- Edge cases - -## Impact - -How will the implemented proposal impacts dapp developers, validators, and core contributors? - -## Security Considerations - -What security implications/considerations come with implementing this feature? -Are there any implementation-specific guidance or pitfalls? - -## Drawbacks *(Optional)* - -Why should we not do this? - -## Backwards Compatibility *(Optional)* - -Does the feature introduce any breaking changes? All incompatibilities and -consequences should be listed. From 16bd03b1e41755bb2f46e3ad9fbb1fd6242c3d68 Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Tue, 18 Jul 2023 19:24:56 +0530 Subject: [PATCH 47/67] Create 0058-transaction-receipts --- proposals/0058-transaction-receipts | 138 ++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 proposals/0058-transaction-receipts diff --git a/proposals/0058-transaction-receipts b/proposals/0058-transaction-receipts new file mode 100644 index 000000000..61e860518 --- /dev/null +++ b/proposals/0058-transaction-receipts @@ -0,0 +1,138 @@ +--- +simd: '0064' +title: Transaction Receipts +authors: + - Anoushk Kharangate (Tinydancer) + - Harsh Patel (Tinydancer) + - Richard Patel (Jump Firedancer) +category: Standard +type: Core +status: Draft +created: 2023-06-20 +--- + +# SIMD-0058: Transaction Receipts + +## Summary + +This proposal introduces two new concepts to the Solana runtime. + +- Receipts, a deterministic encoding of state changes induced by a transaction; +- The receipt tree, a commitment scheme over all transaction receipts in a slot. + +## Motivation + +One of the fundamental requirements of a Solana client is access to the status +of a confirmed transaction. This is due to the fact that transactions are not +guaranteed to be confirmed once submitted, and may fail once executed. Virtually +all Solana clients therefore use a subscription-based or polling mechanism to +inquire whether a transaction was successfully executed. + +The only standard mechanism to retrieve transaction statuses remotely is via the +RPC protocol. However, the RPC protocol only serves replay information of the +RPC service itself. It does not provide information whether the validator +network at large has derived the same information. This may allow an RPC +provider to send incorrect information to a client, such as marking a failed +transaction as successful. + +Solana validator nodes replay all transactions and thus have access to +transaction status information. To improve security, clients should verify that +transaction status information received via RPC matches the validator network at +large. + +This proposal introduces a new “transaction receipt” data structure, which +contains a subset of the information available via RPC. The derivation and +serialization functions for receipts are defined to be deterministic to prevent +malleability. + +To succinctly detect receipt mismatches, this proposal further introduces a +commitment scheme based on a binary hash tree that is constructed once per slot. + +## Design Goals + +1. Receipts should be deterministic. + Given a transaction T and ledger history leading up to it, serializing the + receipt generated for T should result in the same byte vector for all nodes + in the network. + *Rationale:* Determinism is required for cluster-wide consensus. + +2. Receipts should not be required during block construction. + *Rationale:* Future upgrades propose tolerating asynchronous replay during + block construction. In other words, validators should be allowed to produce + and distribute a block before replaying said block. It is impossible to + introduce such a tolerance if receipts are mandatory components of blocks. + +3. Construction of receipt commitments should initially be optional to + validators. + *Rationale:* Enforcing construction of receipt commitments (e.g. + by slashing validators that don’t) introduces additional security + considerations. The failure domain of additional receipt logic should be + isolated in the initial rollout to allow for timely activation. + +## Alternatives Considered + +### Using TransactionStatusMeta + +An alternative to introducing a new receipt type is reusing the transaction +status data as it appears in RPC ([TransactionStatusMeta]). This would reduce +complexity for clients. + +However, *TransactionStatusMeta* has no strict definition and is thus malleable +in violation of design goal 1. Technical reasons are as follows: + +- Available fields vary between node releases. +- Log data is truncated based on node configuration. +- The [TransactionResult] type includes error codes, which are + implementation-defined (Thus breaks multi-client compatibility). + +This would make a commitment scheme impractical as-is. Addressing these concerns +is a breaking change. + + [TransactionStatusMeta]: https://docs.rs/solana-transaction-status/1.16.1/solana_transaction_status/struct.TransactionStatusMeta.html + [TransactionResult]: https://docs.rs/solana-sdk/1.16.1/solana_sdk/transaction/type.Result.html + +### Bank Hash + +An alternative to introducing a new commitment scheme is reusing the bank hash. + +When executing transactions, Solana validators only indirectly commit to the +state changes via the bank hash. Namely, the bank hash commits to all changed +accounts after replaying a slot using a binary hash tree. However, it does not +commit to intermediate states during replay. + +Redefining the bank hash to use a construction with transaction-level +granularity is a breaking change. Because construction of the bank hash is +practically mandatory for validators, it also violates design goal 3. + +### Proof-of-History / Block Hash + +Another alternative to introducing a new commitment scheme is reusing the +proof-of-history (PoH) hash chain. This was proposed in +[SIMD-0052](https://github.com/solana-foundation/solana-improvement-documents/pull/52). + +The PoH chain currently commits to the signatures of all transactions added to +the ledger. The consensus layer then periodically votes on the last state of the +PoH chain for each block (block hash). Expanding the PoH hash is the least +complex option as of today but is consequential for future upgrades. + +Such a change would be incompatible with design goal 2 (and by extension, goal +3) because it redefines the PoH hash to additionally commit to execution +results, instead of only ledger content. Block producers are then forced to +synchronously replay while appending PoH ticks. + +Furthermore, it significantly changes behavior in the event of an execution +disagreement (e.g. due to a difference in execution behavior between +validators). Mixing execution results into PoH forces execution disagreements to +result in a chain split. + +## Detailed Design + +### Transaction Receipt Specification +The transaction receipt must contain the following information +related to the transaction: +- Signature +- Execution Status +- Truncated Logs + +### Receipt Tree Specification + From 0a48f4da49c840d3b747e175a8da134b2764889f Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Tue, 18 Jul 2023 19:25:07 +0530 Subject: [PATCH 48/67] Rename 0058-transaction-receipts to 0064-transaction-receipts --- .../{0058-transaction-receipts => 0064-transaction-receipts} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{0058-transaction-receipts => 0064-transaction-receipts} (100%) diff --git a/proposals/0058-transaction-receipts b/proposals/0064-transaction-receipts similarity index 100% rename from proposals/0058-transaction-receipts rename to proposals/0064-transaction-receipts From 941a25ab92bd73aca80edbe76bb01bc41b87924a Mon Sep 17 00:00:00 2001 From: Anoushk Kharangate <32778608+anoushk1234@users.noreply.github.com> Date: Tue, 18 Jul 2023 19:25:41 +0530 Subject: [PATCH 49/67] Delete 0064-transaction-receipts --- proposals/0064-transaction-receipts | 138 ---------------------------- 1 file changed, 138 deletions(-) delete mode 100644 proposals/0064-transaction-receipts diff --git a/proposals/0064-transaction-receipts b/proposals/0064-transaction-receipts deleted file mode 100644 index 61e860518..000000000 --- a/proposals/0064-transaction-receipts +++ /dev/null @@ -1,138 +0,0 @@ ---- -simd: '0064' -title: Transaction Receipts -authors: - - Anoushk Kharangate (Tinydancer) - - Harsh Patel (Tinydancer) - - Richard Patel (Jump Firedancer) -category: Standard -type: Core -status: Draft -created: 2023-06-20 ---- - -# SIMD-0058: Transaction Receipts - -## Summary - -This proposal introduces two new concepts to the Solana runtime. - -- Receipts, a deterministic encoding of state changes induced by a transaction; -- The receipt tree, a commitment scheme over all transaction receipts in a slot. - -## Motivation - -One of the fundamental requirements of a Solana client is access to the status -of a confirmed transaction. This is due to the fact that transactions are not -guaranteed to be confirmed once submitted, and may fail once executed. Virtually -all Solana clients therefore use a subscription-based or polling mechanism to -inquire whether a transaction was successfully executed. - -The only standard mechanism to retrieve transaction statuses remotely is via the -RPC protocol. However, the RPC protocol only serves replay information of the -RPC service itself. It does not provide information whether the validator -network at large has derived the same information. This may allow an RPC -provider to send incorrect information to a client, such as marking a failed -transaction as successful. - -Solana validator nodes replay all transactions and thus have access to -transaction status information. To improve security, clients should verify that -transaction status information received via RPC matches the validator network at -large. - -This proposal introduces a new “transaction receipt” data structure, which -contains a subset of the information available via RPC. The derivation and -serialization functions for receipts are defined to be deterministic to prevent -malleability. - -To succinctly detect receipt mismatches, this proposal further introduces a -commitment scheme based on a binary hash tree that is constructed once per slot. - -## Design Goals - -1. Receipts should be deterministic. - Given a transaction T and ledger history leading up to it, serializing the - receipt generated for T should result in the same byte vector for all nodes - in the network. - *Rationale:* Determinism is required for cluster-wide consensus. - -2. Receipts should not be required during block construction. - *Rationale:* Future upgrades propose tolerating asynchronous replay during - block construction. In other words, validators should be allowed to produce - and distribute a block before replaying said block. It is impossible to - introduce such a tolerance if receipts are mandatory components of blocks. - -3. Construction of receipt commitments should initially be optional to - validators. - *Rationale:* Enforcing construction of receipt commitments (e.g. - by slashing validators that don’t) introduces additional security - considerations. The failure domain of additional receipt logic should be - isolated in the initial rollout to allow for timely activation. - -## Alternatives Considered - -### Using TransactionStatusMeta - -An alternative to introducing a new receipt type is reusing the transaction -status data as it appears in RPC ([TransactionStatusMeta]). This would reduce -complexity for clients. - -However, *TransactionStatusMeta* has no strict definition and is thus malleable -in violation of design goal 1. Technical reasons are as follows: - -- Available fields vary between node releases. -- Log data is truncated based on node configuration. -- The [TransactionResult] type includes error codes, which are - implementation-defined (Thus breaks multi-client compatibility). - -This would make a commitment scheme impractical as-is. Addressing these concerns -is a breaking change. - - [TransactionStatusMeta]: https://docs.rs/solana-transaction-status/1.16.1/solana_transaction_status/struct.TransactionStatusMeta.html - [TransactionResult]: https://docs.rs/solana-sdk/1.16.1/solana_sdk/transaction/type.Result.html - -### Bank Hash - -An alternative to introducing a new commitment scheme is reusing the bank hash. - -When executing transactions, Solana validators only indirectly commit to the -state changes via the bank hash. Namely, the bank hash commits to all changed -accounts after replaying a slot using a binary hash tree. However, it does not -commit to intermediate states during replay. - -Redefining the bank hash to use a construction with transaction-level -granularity is a breaking change. Because construction of the bank hash is -practically mandatory for validators, it also violates design goal 3. - -### Proof-of-History / Block Hash - -Another alternative to introducing a new commitment scheme is reusing the -proof-of-history (PoH) hash chain. This was proposed in -[SIMD-0052](https://github.com/solana-foundation/solana-improvement-documents/pull/52). - -The PoH chain currently commits to the signatures of all transactions added to -the ledger. The consensus layer then periodically votes on the last state of the -PoH chain for each block (block hash). Expanding the PoH hash is the least -complex option as of today but is consequential for future upgrades. - -Such a change would be incompatible with design goal 2 (and by extension, goal -3) because it redefines the PoH hash to additionally commit to execution -results, instead of only ledger content. Block producers are then forced to -synchronously replay while appending PoH ticks. - -Furthermore, it significantly changes behavior in the event of an execution -disagreement (e.g. due to a difference in execution behavior between -validators). Mixing execution results into PoH forces execution disagreements to -result in a chain split. - -## Detailed Design - -### Transaction Receipt Specification -The transaction receipt must contain the following information -related to the transaction: -- Signature -- Execution Status -- Truncated Logs - -### Receipt Tree Specification - From c74d8b4f5c8eca3a58a8e006cbadcd76760ed4bc Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 5 Aug 2023 15:53:51 +0530 Subject: [PATCH 50/67] update simd with receipts --- ...nsus-and-transaction-proof-verification.md | 221 ++++++------------ 1 file changed, 67 insertions(+), 154 deletions(-) diff --git a/proposals/0052-consensus-and-transaction-proof-verification.md b/proposals/0052-consensus-and-transaction-proof-verification.md index cf8335f4b..8a14559b9 100644 --- a/proposals/0052-consensus-and-transaction-proof-verification.md +++ b/proposals/0052-consensus-and-transaction-proof-verification.md @@ -1,10 +1,10 @@ --- simd: '0052' -title: Consensus and Transaction Proof Verification +title: Adding receipt root to Bankhash authors: - - Harsh Patel (Tinydancer) - Anoushk Kharangate (Tinydancer) - - x19 (Tinydancer) + - Harsh Patel (Tinydancer) + - Richard Patel (Jump Firedancer) category: Standard type: Core status: Draft @@ -14,21 +14,14 @@ feature: (fill in with feature tracking issues once accepted) ## Summary -This SIMD describes the overall design and changes required that allows users to -verify that a supermajority of the validators has voted on the slot that their +This SIMD describes the changes required to allow users to +verify that a supermajority of the stake has voted on the slot that their transaction was included in the block without fully trusting the RPC provider. -This includes two main changes: - -1) Adding a new RPC method which provides a proof that a transaction has been - included in a slot -2) Modifying the blockhash to be computed as a Merkle Tree and - includes transaction statuses +The main change includes: -This SIMD is the first step in implementing a consensus verifying client as first -described in [SIMD #10](https://github.com/solana-foundation/solana-improvement-documents/pull/10) -and a majority of the changes mentioned in -the accepted [Simple Payment and State Verification Proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification). +- Modifying the bankhash to add a Receipt Root of the receipt merkle tree that + includes transaction signatures and statuses. ## Motivation @@ -43,160 +36,72 @@ exposing users to potential attacks from malicious nodes. This is where diet clients come in, users run the client to verify the confirmation of their transaction without trusting the RPC. -However, this is only the consensus verifying stage of the client, and with only -these changes, the RPC provider can still trick users, hence we also discuss future -work that will be implemented in futureSIMDs to provide a fully trustless setup. - ## Alternatives Considered None ## New Terminology -TransactionProof: A structure containing necessary information to verify if a -transaction was included in the bank hash of a slot. - -```rs -// new RPC struct to verify tx inclusion -pub struct TransactionProof { - pub proof: Vec, - pub parent_hash: Hash, - pub accounts_delta_hash: Hash, - pub signature_count_buf: [u8; 8], -} -``` +Receipt: A structure containing transaction signature and its execution status. -The proof variable will provide a Merkle hashpath from the transaction to the blockhash; -which is then used with the other variables to compute the bankhash. +Receipt Root: The root hash of a binary merkle tree of Receipts. ## Detailed Design -The protocol interaction will be as follows: - -- A user sends a transaction and it lands in slot N -- The user requests proof that the transaction is included in slot N using - the new RPC method -- The user verifies that the proof includes the transaction signature - and a success status -- The user constructs the merkle tree to derive the root hash -- The user computes the expected bankhash using the root hash, `parent_hash`, - `accounts_delta_hash` and `signature_count` -- The user retrieves the epoch’s current validator set and stake amounts from - a trusted source (making this step trustless is future work) -- The user requests blocks from slots > N (up to 32 slots past N given the - 32 block depth finality) -- The user parses vote transactions from the blocks, verifying their signature, - and computing the sum of stake which voted on - the bankhash they computed in step 3 -- If the sum of stake is greater than or equal to 2/3 of the total stake then - their transaction has been finalized under a supermajority trust assumption - (making this assumption require only a single honest validator is also future work) - -### Modifying the Blockhash - -We propose modifying the blockhash computation to: - -1) Compute the blockhash using a Merkle Tree of entries -2) Include the status (either succeeding or failing) of each transaction - in each Entry leaf - -To produce the blockhash for a slot, the current implementation hashes Entries in -a sequential way which requires a O(N) proof size to provide a hashpath from a -transaction to a blockhash. Implementing change 1) would allow for a more efficient -O(log(N)) proof size. The current Entry implementation already hashes transaction -signatures using a Merkle Tree to get the Entry’s hash, so this change would only -modify how the Entry’s hashes are hashed together to get a blockhash. - -For change 2), the blockhash is currently computed using only transaction signatures, -and does not include transaction statuses which means we are unable to prove if -a transaction has succeeded or failed. Implementing 2) would enable verifying -a transactions status, as mentioned in the [accepted proposal](https://docs.solana.com/proposals/simple-payment-and-state-verification#transaction-merkle) -and in a [previous github issue](https://github.com/solana-labs/solana/issues/7053)). - -Fig #1 shows an example hashpath from a transaction and its signature to a -bankhash with both of the proposed changes implemented. -![Fig #1](https://github.com/tinydancer-io/solana-improvement-documents/assets/32778608/5370950d-e27b-4c1b-9f04-6e9164789e65) - -#### New RPC Methods - -We also need a new RPC method to provide proofs to clients. This method would be -called `get_transaction_proof` which would take a transaction signature as input -and return a `TransactionProof` struct - -```rs -// new RPC method -pub async fn get_transaction_proof(&self, signature: Signature) -> Result; -``` +### Modifying the Bankhash -Below is psuedocode of the RPC method: - -```rs -async fn get_transaction_proof(&self, s: Signature) -> Result { - // first retrieve all the entries - let slot = self.get_slot_of_signature(&s); - let (entries, _, is_full) = - self.blockstore.get_slot_entries_with_shred_info(slot, 0, false) - // require all of the entries - assert!(is_full) - - // compute the Merkle hashpath from the signature and status to the blockhash - let proof = entries.get_merkle_proof(&s); - - // get variables used to compute the bankhash - let bank_forks = self.bank_forks.read().unwrap(); - let bank = bank_forks.get(slot); - - let parent_hash = bank.parent_hash(); - let accounts_delta_hash = bank - .rc - .accounts - .accounts_db - .calculate_accounts_delta_hash(slot).0; - let mut signature_count_buf = [0u8; 8]; - LittleEndian::write_u64(&mut signature_count_buf[..], bank.signature_count()); - - Ok(TransactionProof{ - proof, - parent_hash, - accounts_delta_hash, - signature_count_buf, - }) -} -``` +We propose two new changes: -Below is client pseudocode for verifying a transaction: +1) The receipt data structure and the receipt merkle tree which is formally + defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) ```rust -let tx_sig = "..."; // tx signature of interest -let slot = 19; // slot which includes the tx - -// call the new RPC -let tx_proof: TransactionProof = get_tx_proof(&tx_sig, endpoint); - -// verify the transaction signature proof is valid and status is success -let leaf = hash_leaf!([tx_sig, TxStatus::Success]); -let verified = tx_proof.proof.verify_path(&leaf); -assert!(verified); - -// compute the blockhash -let last_blockhash = tx_proof.proof.get_root(); - -// compute the expected bankhash -let bankhash = hashv(&[ - tx_proof.parent_hash.as_ref(), - tx_proof.accounts_delta_hash.as_ref(), - tx_proof.signature_count_buf.as_ref(), - last_blockhash.as_ref() -]); - -// parse vote transactions and stake amounts on expected bankhash -let (voted_stake_amount, total_stake_amount) = parse_votes_from_blocks(slot, bankhash) - -// validate supermajority voted for expected bankhash -let supermajority_verified = 3 * voted_stake_amount >= 2 * total_stake_amount; -assert!(supermajority_verified) + pub struct Receipt { + pub signature: [u8; 64], + pub status: u8, + } +``` + +2) Add a transaction receipt root to the bankhash calculation where the receipt + root is the root of the merkle tree of receipts. This root would be a sha256 + hash constructed as a final result of the binary merkle tree of receipts. + Specifically it will be a 32 byte array. The receipt root would be added to + the bankhash as follows: + +``` rust + let mut hash = hashv(&[ + self.parent_hash.as_ref(), + accounts_delta_hash.0.as_ref(), + receipt_root, + &signature_count_buf, + self.last_blockhash().as_ref(), + ]); ``` +Note: The second change would initially be feature gated with a flag and can +be activated once we have enough stake on the network with this version of the client. + +#### Benchmarks + +We have performed benchmarks comparing two merkle tree implementations, +the benchmark was done on 1 million leaves, each leaf consisted of a 64 byte +signature and a single byte status. + +1) Solana Labs Merkle Tree: This is the pure rust implementation that is currently + used by the Solana Labs client. + More details [Solana labs repository](https://github.com/solana-labs/solana/tree/master/merkle-tree) +2) Firedancer binary merkle tree (bmtree): Implemented in C and uses firedancer's + optimised SHA-256 implementation as it's hashing algorithm. However the benchmarks + were performed using its rust FFI bindings. + More details: [Firedancer](https://github.com/firedancer-io/firedancer/tree/main/src/ballet/bmtree) + ![Benchamrk Results](https://github.com/tinydancer-io/solana-improvement-documents/assets/50767810/6c8d0013-1d62-4c7b-8264-4ec71ea28d7c) + +More details with an attached flamegraph can be found in our [repository](https://github.com/tinydancer-io/merkle-bench). + +Despite the performance penalty of FFI we can still see that fd_bmtree32 is ~61% +faster than the pure rust implementation which makes it fast enough to not impact +consensus with decent headroom. + ## Impact This proposal will improve the overall security and decentralization of the Solana @@ -205,6 +110,11 @@ traditionally where users had to fully trust their RPC providers. Dapp developer don't have to make any changes as wallets can easily integrate the client making it compatible with any dapp. +The proposal would also be compatible with the future protocol updates like +Bankless leaders since the tree construction would be done async by buffering +transaction statuses. Bankless leaders won't need replay before propagating +the block. + ## Security Considerations ### Trust Assumptions and Future Work @@ -217,4 +127,7 @@ While this SIMD greatly reduces the user's trust in an RPC, the light client wil including data availability sampling and fraud proving which will only require a single honest full node. -## Backwards Compatibility *(Optional)* +## Backwards Compatibility + +The change is not backwards compatible due to which it would require +a feature flag activation strategy. From 8a0eea5b4e27d9c0a2cd92d6dec621902924ce93 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 5 Aug 2023 16:06:47 +0530 Subject: [PATCH 51/67] fix --- ...> 0052-adding-receipt-root-to-bankhash.md} | 33 +++++++++---------- 1 file changed, 16 insertions(+), 17 deletions(-) rename proposals/{0052-consensus-and-transaction-proof-verification.md => 0052-adding-receipt-root-to-bankhash.md} (93%) diff --git a/proposals/0052-consensus-and-transaction-proof-verification.md b/proposals/0052-adding-receipt-root-to-bankhash.md similarity index 93% rename from proposals/0052-consensus-and-transaction-proof-verification.md rename to proposals/0052-adding-receipt-root-to-bankhash.md index 8a14559b9..4b3f776aa 100644 --- a/proposals/0052-consensus-and-transaction-proof-verification.md +++ b/proposals/0052-adding-receipt-root-to-bankhash.md @@ -1,10 +1,9 @@ --- simd: '0052' -title: Adding receipt root to Bankhash +title: Adding Receipt Root to Bankhash authors: - Anoushk Kharangate (Tinydancer) - Harsh Patel (Tinydancer) - - Richard Patel (Jump Firedancer) category: Standard type: Core status: Draft @@ -55,12 +54,12 @@ We propose two new changes: 1) The receipt data structure and the receipt merkle tree which is formally defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) -```rust - pub struct Receipt { - pub signature: [u8; 64], - pub status: u8, - } -``` + ```rust + pub struct Receipt { + pub signature: [u8; 64], + pub status: u8, + } + ``` 2) Add a transaction receipt root to the bankhash calculation where the receipt root is the root of the merkle tree of receipts. This root would be a sha256 @@ -68,15 +67,15 @@ We propose two new changes: Specifically it will be a 32 byte array. The receipt root would be added to the bankhash as follows: -``` rust - let mut hash = hashv(&[ - self.parent_hash.as_ref(), - accounts_delta_hash.0.as_ref(), - receipt_root, - &signature_count_buf, - self.last_blockhash().as_ref(), - ]); -``` + ``` rust + let mut hash = hashv(&[ + self.parent_hash.as_ref(), + accounts_delta_hash.0.as_ref(), + receipt_root, + &signature_count_buf, + self.last_blockhash().as_ref(), + ]); + ``` Note: The second change would initially be feature gated with a flag and can be activated once we have enough stake on the network with this version of the client. From 7205af1232231b96006b6ee0df6b4e5a988f37a9 Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Sat, 5 Aug 2023 17:43:39 +0530 Subject: [PATCH 52/67] fix --- ...root-to-bankhash.md => 0052-add-receipt-root-to-bankhash.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{0052-adding-receipt-root-to-bankhash.md => 0052-add-receipt-root-to-bankhash.md} (99%) diff --git a/proposals/0052-adding-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md similarity index 99% rename from proposals/0052-adding-receipt-root-to-bankhash.md rename to proposals/0052-add-receipt-root-to-bankhash.md index 4b3f776aa..212cbea2d 100644 --- a/proposals/0052-adding-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -1,6 +1,6 @@ --- simd: '0052' -title: Adding Receipt Root to Bankhash +title: Add Receipt Root to Bankhash authors: - Anoushk Kharangate (Tinydancer) - Harsh Patel (Tinydancer) From 2fee5ff7c3f3e1450443aef83e2548a8a5a9829c Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Mon, 7 Aug 2023 13:51:24 +0530 Subject: [PATCH 53/67] update author --- proposals/0052-add-receipt-root-to-bankhash.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 212cbea2d..9f5431780 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -4,6 +4,7 @@ title: Add Receipt Root to Bankhash authors: - Anoushk Kharangate (Tinydancer) - Harsh Patel (Tinydancer) + - x19 category: Standard type: Core status: Draft From e40077ba54defdaa6211dbc433c64f8e29f0f4ad Mon Sep 17 00:00:00 2001 From: anoushk1234 Date: Mon, 21 Aug 2023 12:40:24 +0530 Subject: [PATCH 54/67] fix --- .../0052-add-receipt-root-to-bankhash.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 9f5431780..edc94f0b0 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -50,10 +50,10 @@ Receipt Root: The root hash of a binary merkle tree of Receipts. ### Modifying the Bankhash -We propose two new changes: +We propose the following change: -1) The receipt data structure and the receipt merkle tree which is formally - defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) +Using the receipt data structure and the receipt merkle tree which is formally +defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) ```rust pub struct Receipt { @@ -62,19 +62,19 @@ We propose two new changes: } ``` -2) Add a transaction receipt root to the bankhash calculation where the receipt - root is the root of the merkle tree of receipts. This root would be a sha256 - hash constructed as a final result of the binary merkle tree of receipts. - Specifically it will be a 32 byte array. The receipt root would be added to - the bankhash as follows: +We add a transaction receipt root to the bankhash calculation where the receipt +root is the root of the merkle tree of receipts. This root would be a sha256 +hash constructed as a final result of the binary merkle tree of receipts. +Specifically it will be a 32 byte array. The receipt root would be added to +the bankhash as follows: ``` rust let mut hash = hashv(&[ - self.parent_hash.as_ref(), - accounts_delta_hash.0.as_ref(), + parent_hash, + accounts_delta_hash, receipt_root, - &signature_count_buf, - self.last_blockhash().as_ref(), + signature_count_buf, + last_blockhash, ]); ``` From 931506f7925be49fe7d235cd4880725aba3a67af Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Sun, 22 Oct 2023 18:17:34 +0530 Subject: [PATCH 55/67] simd 52 changes wip --- .../0052-add-receipt-root-to-bankhash.md | 41 +++---------------- 1 file changed, 5 insertions(+), 36 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index edc94f0b0..352723c85 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -20,8 +20,8 @@ transaction was included in the block without fully trusting the RPC provider. The main change includes: -- Modifying the bankhash to add a Receipt Root of the receipt merkle tree that - includes transaction signatures and statuses. +- Modifying the Bankhash to add a transaction receipt root of the transaction receipt merkle tree. + This root would be a 32 byte SHA-256 hash. ## Motivation @@ -42,9 +42,9 @@ None ## New Terminology -Receipt: A structure containing transaction signature and its execution status. +Receipt: A structure containing a 64 byte version, a transaction message hash and its execution status. -Receipt Root: The root hash of a binary merkle tree of Receipts. +Receipt Root: The root hash of a binary merkle tree of transaction receipts. ## Detailed Design @@ -55,15 +55,8 @@ We propose the following change: Using the receipt data structure and the receipt merkle tree which is formally defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) - ```rust - pub struct Receipt { - pub signature: [u8; 64], - pub status: u8, - } - ``` - We add a transaction receipt root to the bankhash calculation where the receipt -root is the root of the merkle tree of receipts. This root would be a sha256 +root is the root of the merkle tree of transaction receipts. This root would be a sha256 hash constructed as a final result of the binary merkle tree of receipts. Specifically it will be a 32 byte array. The receipt root would be added to the bankhash as follows: @@ -78,30 +71,6 @@ the bankhash as follows: ]); ``` -Note: The second change would initially be feature gated with a flag and can -be activated once we have enough stake on the network with this version of the client. - -#### Benchmarks - -We have performed benchmarks comparing two merkle tree implementations, -the benchmark was done on 1 million leaves, each leaf consisted of a 64 byte -signature and a single byte status. - -1) Solana Labs Merkle Tree: This is the pure rust implementation that is currently - used by the Solana Labs client. - More details [Solana labs repository](https://github.com/solana-labs/solana/tree/master/merkle-tree) -2) Firedancer binary merkle tree (bmtree): Implemented in C and uses firedancer's - optimised SHA-256 implementation as it's hashing algorithm. However the benchmarks - were performed using its rust FFI bindings. - More details: [Firedancer](https://github.com/firedancer-io/firedancer/tree/main/src/ballet/bmtree) - ![Benchamrk Results](https://github.com/tinydancer-io/solana-improvement-documents/assets/50767810/6c8d0013-1d62-4c7b-8264-4ec71ea28d7c) - -More details with an attached flamegraph can be found in our [repository](https://github.com/tinydancer-io/merkle-bench). - -Despite the performance penalty of FFI we can still see that fd_bmtree32 is ~61% -faster than the pure rust implementation which makes it fast enough to not impact -consensus with decent headroom. - ## Impact This proposal will improve the overall security and decentralization of the Solana From dfc6b83979285fed6cf47940ba2e95335c87b0c1 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Sun, 22 Oct 2023 18:23:29 +0530 Subject: [PATCH 56/67] simd 52 changes wip --- proposals/0052-add-receipt-root-to-bankhash.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 352723c85..924e3419e 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -20,8 +20,8 @@ transaction was included in the block without fully trusting the RPC provider. The main change includes: -- Modifying the Bankhash to add a transaction receipt root of the transaction receipt merkle tree. - This root would be a 32 byte SHA-256 hash. +- Modifying the Bankhash to add a transaction receipt root of the transaction + receipt merkle tree. This root would be a 32 byte SHA-256 hash. ## Motivation @@ -42,7 +42,8 @@ None ## New Terminology -Receipt: A structure containing a 64 byte version, a transaction message hash and its execution status. +Receipt: A structure containing a 64 byte version, a transaction message hash +and its execution status. Receipt Root: The root hash of a binary merkle tree of transaction receipts. @@ -56,10 +57,10 @@ Using the receipt data structure and the receipt merkle tree which is formally defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) We add a transaction receipt root to the bankhash calculation where the receipt -root is the root of the merkle tree of transaction receipts. This root would be a sha256 -hash constructed as a final result of the binary merkle tree of receipts. -Specifically it will be a 32 byte array. The receipt root would be added to -the bankhash as follows: +root is the root of the merkle tree of transaction receipts. +This root would be a sha256 hash constructed as a final result of the +binary merkle tree of receipts. Specifically it will be a 32 byte array. +The receipt root would be added to the bankhash as follows: ``` rust let mut hash = hashv(&[ From d8ca3ddd5d37b694401990303381f1cfb06bc955 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Tue, 14 Nov 2023 16:03:37 +0530 Subject: [PATCH 57/67] modify summary --- .../0052-add-receipt-root-to-bankhash.md | 20 +++++++++---------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 924e3419e..b3bd16a4e 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -14,14 +14,13 @@ feature: (fill in with feature tracking issues once accepted) ## Summary -This SIMD describes the changes required to allow users to -verify that a supermajority of the stake has voted on the slot that their -transaction was included in the block without fully trusting the RPC provider. +This proposal describes the inclusion of a transaction receipt tree root which +is a 32 byte SHA-256 hash to the current Bankhash construction. This will enable +a way to verify transaction inclusion by a merkle proof which has a path from +the transaction receipt to the transaction receipt tree root since the network +votes on the bankhash for consensus. -The main change includes: - -- Modifying the Bankhash to add a transaction receipt root of the transaction - receipt merkle tree. This root would be a 32 byte SHA-256 hash. +The receipt tree construction is formally described in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) ## Motivation @@ -36,16 +35,15 @@ exposing users to potential attacks from malicious nodes. This is where diet clients come in, users run the client to verify the confirmation of their transaction without trusting the RPC. + + ## Alternatives Considered None ## New Terminology -Receipt: A structure containing a 64 byte version, a transaction message hash -and its execution status. - -Receipt Root: The root hash of a binary merkle tree of transaction receipts. +Transaction Receipt Root: The merkle root hash of a binary merkle tree of TransactionReceiptData as formally describled in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64). ## Detailed Design From 0614f5c263628e0c31d16e27e1ca2c14501c7082 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Tue, 14 Nov 2023 16:07:05 +0530 Subject: [PATCH 58/67] fix lint --- proposals/0052-add-receipt-root-to-bankhash.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index b3bd16a4e..14cfdb02f 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -14,11 +14,11 @@ feature: (fill in with feature tracking issues once accepted) ## Summary -This proposal describes the inclusion of a transaction receipt tree root which -is a 32 byte SHA-256 hash to the current Bankhash construction. This will enable -a way to verify transaction inclusion by a merkle proof which has a path from -the transaction receipt to the transaction receipt tree root since the network -votes on the bankhash for consensus. +This proposal describes the inclusion of a transaction receipt tree root, +which is a 32 byte SHA-256 hash, to the current Bankhash construction. +This will enable a way to verify transaction inclusion by a merkle proof +which has a path from the transaction receipt to the transaction receipt +tree root since the network votes on the bankhash for consensus. The receipt tree construction is formally described in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) From 82d017d11241de4b55650407dfbbdcdf5cd1ea3d Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Tue, 14 Nov 2023 16:08:42 +0530 Subject: [PATCH 59/67] fix lint --- proposals/0052-add-receipt-root-to-bankhash.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 14cfdb02f..aeb29e1d7 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -43,7 +43,8 @@ None ## New Terminology -Transaction Receipt Root: The merkle root hash of a binary merkle tree of TransactionReceiptData as formally describled in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64). +Transaction Receipt Root: The merkle root hash of a binary merkle tree of +TransactionReceiptData as formally describled in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64). ## Detailed Design From 1068a23e2637fe4c31b6a297552a74df9bd6fc33 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 00:25:32 +0530 Subject: [PATCH 60/67] fixes --- .../0052-add-receipt-root-to-bankhash.md | 28 +++++++------------ 1 file changed, 10 insertions(+), 18 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index aeb29e1d7..f8fd4fc5b 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -20,22 +20,17 @@ This will enable a way to verify transaction inclusion by a merkle proof which has a path from the transaction receipt to the transaction receipt tree root since the network votes on the bankhash for consensus. -The receipt tree construction is formally described in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) - ## Motivation -Currently, for a user to validate whether their transaction is valid and included -in a block it needs to trust the confirmation from the RPC. This has been a glaring -attack vector for malicious actors that could lie to users if it's in their own interest. - -To combat this, mature entities like exchanges run full nodes that process the -entire ledger and can verify entire blocks. The downside of this is that it's -very costly to run a full node which makes it inaccessible to everyday users, -exposing users to potential attacks from malicious nodes. -This is where diet clients come in, users run the client to verify -the confirmation of their transaction without trusting the RPC. - - +The transaction receipt tree as formalized in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) +proposes a commitment scheme to construct a binary merkle tree +of transaction receipts which contain information about the +execution status of the transaction. The root of this tree +needs to have network wide consensus per slot. Hence we propose +adding the transaction receipt root to Bankhash since it's voted +on by the staked nodes who participate in consensus. This will +help Solana clients verify transaction inclusion and supermajority +consensus on the transaction hence enabling consensus light clients. ## Alternatives Considered @@ -44,7 +39,7 @@ None ## New Terminology Transaction Receipt Root: The merkle root hash of a binary merkle tree of -TransactionReceiptData as formally describled in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64). +TransactionReceiptData described in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64). ## Detailed Design @@ -52,9 +47,6 @@ TransactionReceiptData as formally describled in [SIMD-0064](https://github.com/ We propose the following change: -Using the receipt data structure and the receipt merkle tree which is formally -defined in this [SIMD]([https://github.com/tinydancer-io/solana-improvement-documents](https://github.com/tinydancer-io/solana-improvement-documents/blob/transaction-receipt/proposals/0064-transaction-receipt.md)) - We add a transaction receipt root to the bankhash calculation where the receipt root is the root of the merkle tree of transaction receipts. This root would be a sha256 hash constructed as a final result of the From 52124c6b556b5dc77ee5cbdfd3dcb86059cc790e Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 00:29:12 +0530 Subject: [PATCH 61/67] fix lint --- proposals/0052-add-receipt-root-to-bankhash.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index f8fd4fc5b..b0dde3e2b 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -22,7 +22,8 @@ tree root since the network votes on the bankhash for consensus. ## Motivation -The transaction receipt tree as formalized in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) +The transaction receipt tree as formalized in +[SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) proposes a commitment scheme to construct a binary merkle tree of transaction receipts which contain information about the execution status of the transaction. The root of this tree From 0d825d09eaad16b6feb8d9d6709dba44edffa919 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 00:31:15 +0530 Subject: [PATCH 62/67] fix --- proposals/0052-add-receipt-root-to-bankhash.md | 1 - 1 file changed, 1 deletion(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index b0dde3e2b..074404128 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -21,7 +21,6 @@ which has a path from the transaction receipt to the transaction receipt tree root since the network votes on the bankhash for consensus. ## Motivation - The transaction receipt tree as formalized in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) proposes a commitment scheme to construct a binary merkle tree From bef2cb24ff475470930e11896fb9073cea7a78a8 Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 00:32:49 +0530 Subject: [PATCH 63/67] update --- proposals/0052-add-receipt-root-to-bankhash.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 074404128..7b0a425c0 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -21,8 +21,7 @@ which has a path from the transaction receipt to the transaction receipt tree root since the network votes on the bankhash for consensus. ## Motivation -The transaction receipt tree as formalized in -[SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) +The transaction receipt tree as formalized in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) proposes a commitment scheme to construct a binary merkle tree of transaction receipts which contain information about the execution status of the transaction. The root of this tree From 200611edb834745ae0e38eced8c02f16f65c47bb Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 00:35:31 +0530 Subject: [PATCH 64/67] fix --- proposals/0052-add-receipt-root-to-bankhash.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 7b0a425c0..10ab70446 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -21,10 +21,9 @@ which has a path from the transaction receipt to the transaction receipt tree root since the network votes on the bankhash for consensus. ## Motivation -The transaction receipt tree as formalized in [SIMD-0064](https://github.com/solana-foundation/solana-improvement-documents/pull/64) -proposes a commitment scheme to construct a binary merkle tree -of transaction receipts which contain information about the -execution status of the transaction. The root of this tree +The transaction receipt tree proposes a commitment scheme to construct +a binary merkle tree of transaction receipts which contain information +about the execution status of the transaction. The root of this tree needs to have network wide consensus per slot. Hence we propose adding the transaction receipt root to Bankhash since it's voted on by the staked nodes who participate in consensus. This will From b7b7b00b9049435b19ee87fbf12fcbc1f670d68d Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 00:46:18 +0530 Subject: [PATCH 65/67] update --- proposals/0052-add-receipt-root-to-bankhash.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 10ab70446..3afd18037 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -21,6 +21,7 @@ which has a path from the transaction receipt to the transaction receipt tree root since the network votes on the bankhash for consensus. ## Motivation + The transaction receipt tree proposes a commitment scheme to construct a binary merkle tree of transaction receipts which contain information about the execution status of the transaction. The root of this tree From 2f0e2931df9aa2d114eb790f95c740df76c8351d Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Wed, 15 Nov 2023 13:28:01 +0530 Subject: [PATCH 66/67] update --- proposals/0052-add-receipt-root-to-bankhash.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 3afd18037..2c964ee05 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -44,7 +44,9 @@ TransactionReceiptData described in [SIMD-0064](https://github.com/solana-founda ### Modifying the Bankhash -We propose the following change: +The bankhash is computed in the `hash_internal_state() -> Hash` which is +implicitly called by `bank.freeze()`. The bank is frozen after all the +transactions of a particular slot have been executed and committed. We add a transaction receipt root to the bankhash calculation where the receipt root is the root of the merkle tree of transaction receipts. From e425a1a3d17aac865e618078d7629cda8eefd66d Mon Sep 17 00:00:00 2001 From: harsh4786 <50767810+harsh4786@users.noreply.github.com> Date: Fri, 17 Nov 2023 23:45:00 +0530 Subject: [PATCH 67/67] fix --- proposals/0052-add-receipt-root-to-bankhash.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0052-add-receipt-root-to-bankhash.md b/proposals/0052-add-receipt-root-to-bankhash.md index 2c964ee05..84c60bb3e 100644 --- a/proposals/0052-add-receipt-root-to-bankhash.md +++ b/proposals/0052-add-receipt-root-to-bankhash.md @@ -58,9 +58,9 @@ The receipt root would be added to the bankhash as follows: let mut hash = hashv(&[ parent_hash, accounts_delta_hash, - receipt_root, signature_count_buf, last_blockhash, + receipt_root, ]); ```