Turbine for Duplicate Block Prevention #71
Conversation
carllin
changed the title
carllin
force-pushed
the
duplicate_block_prevention
branch
3 times, most recently
from
67b0147 to
cabe3fb
Compare
carllin
force-pushed
the
duplicate_block_prevention
branch
3 times, most recently
from
ddf88d8 to
4d62fe8
Compare
carllin
force-pushed
the
duplicate_block_prevention
branch
from
4d62fe8 to
e6d12a9
Compare
carllin
changed the title
| to the FEC set to which it belongs. This means that given an FEC set of minimum | ||
| 32 shreds, a leader cannot create an entirely new FEC set by just modifying the | ||
| last shred, because the `witness` in that last shred disambiguates which FEC set | ||
| it belongs to. |
There was a problem hiding this comment.
This is not very friendly to first-time readers, would be good to add SIMD or other doc here which describes the Merkle shred change introduced.
There was a problem hiding this comment.
makes sense i'll link it
|
|
||
| 1. Lock down shred propagation so that validators only accept shred `X` if it | ||
| arrives from the correct ancestor in the turbine tree for that shred `X`. There | ||
| are a few downstream effects of this: |
There was a problem hiding this comment.
We can lock down in our validator implementation. But if team X implements and allows their own sideline shred forwarder, how much of the assumption here is broken?
|
|
||
| - In repair, a validator `V` can no longer repair shred `X` from anybody other | ||
| than the singular ancestor `Y` that was responsible for delivering shred `X` to | ||
| `V` in the turbine tree. |
There was a problem hiding this comment.
You mean a validator can only repair from its single parent on the Turbine tree, not grandparents?
There was a problem hiding this comment.
We should put that into the doc, something like "from anybody other than the parent (not even from grandparents) ...", ancestors include parents and grandparents, I think.
|
|
||
| then if validator `V`'s ancestor `Y` for shred `X` is down, then shred `X` is | ||
| unrecoverable. Without being able to repair a backup erasure shred, this would | ||
| mean validator `X` could never recover this block |
There was a problem hiding this comment.
Not sure whether this block belong to any bullet above, and what "then if" refers to. And not clear what "this" refers to in "this would".
| that shred `S`' for the same block, it will propagate the witness of both of | ||
| those shreds so that everyone in the turbine tree sees the duplicate proof. This | ||
| makes it harder for leaders to split the network into groups that see a block is | ||
| duplicate and groups that don't. |
There was a problem hiding this comment.
Is it guaranteed the Turbine tree is always the same if two validators with the same pubkey are physically apart (e.g. setting up hot-standby in US-Europe).
There was a problem hiding this comment.
To make the doc self-contained, we should probably list the properties of Turbine we depend on in the doc as well.
| ### Duplicate block resolution | ||
|
|
||
| Against a powerful adversary, the preventative measures outlined above can be | ||
| circumvented. Namely an adversary that controls a large percentage of stake and |
There was a problem hiding this comment.
Maybe mention roughly how large a percentage here, I assume >33%?
| In the worse case we can assume that the adversary controls 33% of the network | ||
| stake. By utilizing this stake, they can attack honest nodes by creating network | ||
| partitions. In a turbine setup with offline nodes and malicious stake | ||
| communicating through side channel, simulations show a 1% propagation to honest |
There was a problem hiding this comment.
What does "1% propagation to honest nodes" mean?
| communicating through side channel, simulations show a 1% propagation to honest | ||
| nodes given at least 15% honest nodes are in the partition. [1] | ||
|
|
||
| Median stake recovered with 33% malicious, 10K trials |
There was a problem hiding this comment.
Maybe explain what "percentage online", "equal stake", "mainnet stake" mean briefly. It would be good for SIMD to be self-contained by roughly describing how the tests were structured.
|
|
||
| ## Impact | ||
|
|
||
| The network will be more resilient against duplicate blocks |
There was a problem hiding this comment.
Maybe we need to estimate memory/network impact for storing up to 5 duplicate forks?
There was a problem hiding this comment.
definitely, will be interesting to run inv tests with 5 partitions on turbine, but a connected gossip & repair. I believe that will allow us to propagate 5 blocks for each slot.
we can compare that to a cluster running only on repair to get a rough idea.
There was a problem hiding this comment.
I've heard mentions of storing up to 5 duplicate forks, but what does this actually mean? Does it mean plumbing blockstore to hold up to 5 versions of the same block and keying everything based on Slot Hash?
I don't see it mentioned anywhere in this SIMD, and it's not clear why it would actually be necessary as part of this proposal
There was a problem hiding this comment.
that was the original idea, however there has been no solid consensus on whether we need to implement such a change. Originally I had a section in this SIMD with that design carllin@807b5ee#diff-d1443f19931349d37d7a29462e1c96d99f6bd1a4d7b08757dd6360425ae15076L95, but since it is still uncertain I removed it.
I think the scope of this SIMD can be purely on efforts to prevent the propagation of duplicate blocks, and if necessary a later SIMD can speak about the new resolution efforts.
|
|
||
| ## Backwards Compatibility | ||
|
|
||
| Not applicable |
There was a problem hiding this comment.
What if there are other mods which didn't implement this SIMD?
There was a problem hiding this comment.
Also I think it'd be nice to have a "rollout plan". Are all the changes we intend to make compatible to the current status so we don't need any feature gate/adoption etc?
There was a problem hiding this comment.
generally anything about duplicate block prevention is local. If you circumvent the turbine tree or run some other shred related mods there's a chance that you can observe duplicate blocks while the rest of the network doesn't.
Since we set the vote threshold at 34%, we'd require 34% of the cluster to be non-compliant with this proposal in order to have a cluster wide impact. At that point compliant validators will see the duplicate blocks that they were prevented from seeing, and will need to use the strategies outlined in duplicate block resolution.
|
splitting this into 2 simds. the scope of this simd is only for best effort duplicate block prevention. |
| mean validator `X` could never recover this block | ||
|
|
||
| 2. If a validator received shred `S` for a block, and then another version of | ||
| that shred `S`' for the same block, it will propagate the witness of both of |
There was a problem hiding this comment.
After some offline discussion with behzad I think this is an unfeasible strategy in turbine. Sending payloads fragmented over more than one packet introduces a lot of overhead. It seems unwise to introduce this latency in turbine where performance is critical.
However for exact (slot, shred_index, type) duplicate proofs, this will continue to work out of the box like today.
| `V` in the turbine tree. | ||
| - Validators need to be able to repair erasure shreds, whereas they can only | ||
| repair data shreds today. This is because now the set of repair peers is locked, | ||
|
|
| mean validator `X` could never recover this block | ||
|
|
||
| 2. If a validator received shred `S` for a block, and then another version of | ||
| that shred `S`' for the same block, it will propagate the witness of both of |
There was a problem hiding this comment.
Would be good to define what 'witness' means here
| see a duplicate proof when the other canonical version of the shred is | ||
| broadcasted. | ||
|
|
||
| 3. The last FEC set is unique in that it can have less than 32 data shreds. |
There was a problem hiding this comment.
Can update this to indicate the new strategy of ensuring fully packed FEC sets
| arrives from the correct ancestor in the turbine tree for that shred `X`. There | ||
| are a few downstream effects of this: | ||
|
|
||
| - In repair, a validator `V` can no longer repair shred `X` from anybody other |
There was a problem hiding this comment.
This is the only part of the proposal that gives me major heartburn.
Ignoring duplicate blocks for a second, we see quite a few cases where leader transmission to root of turbine tree drops for some period of time and we see lots of shreds drop in a row. In this case, we would need the various roots to request repair from the leader, then their children to request repair from them, etc. until everyone can repair the block. Seems like major latency in getting the block into blockstore so we can replay.
In other words, if you drop a shred near the top of the turbine tree, good luck getting your block confirmed. Obviously I haven't actually collected data to see if my assumptions are true, but I'll remain cautiously pessimistic for now.
One thing that might help is to enable retransmission of repaired shreds.
|
closing this for now and plan to submit / reopen with a more complete design once we nail it down. |
No description provided.