Develop 3.0 lastgaspring #945
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allow ring to have metadata indicating it is lastgasp - so that subscribers to ring changes cna ignore the last gasp ring defined just before the close. Prompt for a tick before the close of the riak_core_claimant - to make sure there is always a tick before the last gasp ring is created.
The riak_core_vnode_manager has a mangement tick that may occur during shutdown - that is whilst dependencies are closing, and cor eis not yet closed, the riak_core_vnode_manager could receive and act on a management tick. When a node is exiting, this cna cause the vnode_manager to start the vnodes required in the fresh ring that was generated as the "last gasp" before closing (so that at on restarting it will be an empty and detatched vnode). This change prevents that restart - and avoids the associated log noise when those started vnodes fail to start due to the shutdown process.
to make it easier should it ever need to be unset via remote_console should something go wrong. Also provides function to unset tainted ring re #744
This was referenced Oct 31, 2019
The set_ring/2 function will update the glocal ring - so in the previous version there was a short window where a process could see the lastgaps ring without the lastgasp status. Processes can check the ring before the ring_manager has returned. This should close that window, but still persist without the lastgasp status
ThomasArts
approved these changes
Nov 5, 2019
albsch
pushed a commit
to riak-core-lite/riak_core_lite
that referenced
this pull request
Jul 13, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a node leaves a cluster before it closes it will change the ring to a fresh ring (which knows only of the local node). This is necessary to ensure that restarting the node does not lead to any attempt to rejoin.
There is an issue though, that the close of riak is not immediate. It is, as one would expect, an orderly shutdown of the dependencies, before riak_core is the final application to close.
During this closing process, there are multiple processes which are monitoring the ring and may react to the change in the ring. At this stage, as the shutdown is not complete these processes may take unnecessary action, and as the node is still connected to he cluster at an erlang level - this could leak into the rest of the cluster.
This was causing significant problems for
riak_ensemble, where ensembles would be changed through the cluster incorrectly. In some cases there would then be a version mismatch that meant that this was never corrected back.This PR introduces 'lastgasp' metadata to the ring. If the ring is being changed to reflect shutdown on departure this metadata is set after the ring has been persisted (so it is not seen on restart).
There are related PRs then due in
riak_kvandriak_replwith ring_handlers amended to check thislastgaspstatus before reacting to a ring change.These changes have stabilised ensemble riak_test tests (without actually changing
riak_ensemble), and also reduce unexpected crashes during the shutdown processes caused by the false starting of elections/vnodes.