Observe state changes in a Raft node

[diptanu]

HI @drmingdrmer, what's the best way to get notified when the raft node's server state changes? I want the leader node to run some tasks, and stop them when it loses leadership. Is there a good api to get a stream of state changes? I saw that there is a wait method which waits for a specific state, but in that case if I am waiting only for leader I am wondering how I can know that the leadership is lost after a node becomes a leader prior to that.

[github-actions]

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

[drmingdrmer]

To monitor the state of a Raft node in OpenRaft, it's recommended to subscribe to updates in the RaftMetrics. The following code snippet provides an example of how to wait for changes in RaftMetrics until a leader is elected:

https://github.com/drmingdrmer/databend/blob/313b40ed3ff1cfe001bc6e2f806773d9001861d0/src/meta/service/src/meta_service/meta_node.rs#L1149-L1173

        let mut rx = self.raft.metrics();

        loop {
            if let Some(l) = rx.borrow().current_leader {
                return Ok(Some(l));
            }

            rx.changed().await?;
        }

This approach allows you to react to changes in the cluster's leadership efficiently. For more detailed information, refer to the specific section of the code in the RaftMetrics::state.

[diptanu]

@drmingdrmer I tried this out, looks like the receiver of raft metrics unblocks every few millisecond. Is there any way to subscribe to only leadership change notifications?

[drmingdrmer]

You need to verify whether RaftMetrics::state has changed for your purpose. There is an active issue that involves dividing RaftMetrics into two separate entities. Your application will be less frequently waken up. Despite this, you must still check whether the specific field of interest to you has been changed.

• Split metrics into data metrics and server metrics #985

[diptanu]

@drmingdrmer Understood! I have an implementation here - tensorlakeai/indexify#221 would appreciate if you can take a look and provide some feedback :)

[drmingdrmer]

@drmingdrmer Understood! I have an implementation here - diptanu/indexify#221 would appreciate if you can take a look and provide some feedback :)

Of course. I've added some comments on your pull request.