LeaderElection is a library for electing a leader with a pluggable backend (a
RaceDecider).
The RaceDecider implementation needs to provide a conditional-write or
transactional-write mechanism to ensure that only one write succeeds per
election-cycle, but otherwise has very minimal requirements.
Currently, this implementation only has two implementations:
- Google Cloud Storage: intended for easy production use while running in GCP, using a GCS object for locking.
- Memory: intended for tests.
The Config struct contains callbacks and tunables for the leader election
"campaign".
Each process that would like to acquire leadership must register callbacks for
all three of OnElected, OnOusting and LeaderChanged, as well as specify
unique LeaderID and HostPorts (the latter two are used for communication, so
some use-cases may be able to ignore them)
The TermLength is the length of time that a leader acquires leadership for,
and the length of any extension. This duration should be long enough to get
useful work done, but short enough that you won't have a problem if the leader
dies and no one takes over the remainder of the lease term. The Config.Acquire
method takes care of extending the lease twice per term to reduce the likelihood
of spuriously losing the lock.
MaxClockSkew specifies the corrections added to and subtracted from sleeps
and leases to account for a lack of perfect synchronization among the clocks of
all candidates.
ConnectionParams is a generic byte-payload side-channel. The legrpc package
uses it for the GRPC ServiceConfig, but other use-cases may stash other
serialized data there (may be nil).
Clock is an instance of a clocks.Clock, which should be nil outside of
tests, in which case it uses clocks.DefaultClock().
The OnElected callback takes two arguments which indicate the state of the
leadership lock with different degrees of certainty.
The ctx argument is a context derived from the one passed to Acquire that
will be canceled upon losing the leadership role. This is an explicit
cancellation by the goroutine handling lease renewals and acquisition, and as
such is subject to normal thread-scheduling delay caveats (particularly relevant
when operating with heavy CPU-contention).
To address the thread-scheduling-delay issues plaguing use of ctx, the
second argument to OnElected is a *TimeView containing the current
expiration time. This pointer tracks an atomic value which is updated every time
the lease is extended. Before taking any action that requires holding
leadership, one should always check that the time returned by t.Get() is in
the future. TimeView has a ValueInFuture() convenience-method to facilitate
such checks for quick operations against the correct clock.
The gcs RaceDecider is the only currently usable implementation. It
requires a Google Cloud Storage client, a bucket and an object.
In tests, one can use the memory RaceDecider, as that implementation
is trivially fast and lacks any external dependencies, but doesn't work outside
a single process.
Leader election is useful on its own only for a subset of use cases. Many times
it is necessary for other processes to observe a leader election to send
requests to the correct process. As such, one can define a WatchConfig and
call Watch() on it. The callback will be called sequentially for every
lease-extension and acquisition, thus allowing an observer to track the
expiration of the current leader's leadership term.