Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the NIO framework #135

Merged
merged 11 commits into from
Jan 4, 2024
Merged

Improve the NIO framework #135

merged 11 commits into from
Jan 4, 2024

Conversation

JackyWoo
Copy link
Contributor

@JackyWoo JackyWoo commented Dec 12, 2023 •
edited
Loading

Which issues of this PR fixes:

This PR try to fix #134

Motivation

Now the NIO framework has a lot of small issue. We should improve it and make it a dependent component, so it can be used in other senarios.

Change log:

  • Make the NIO framework self contained and now it only depend some basic Poco utils.
  • No large amount of socket copy
  • Use hash algo to avoid large amount of socket comparison
  • Use std::shared_ptr instead of Poco::AutoPtr
  • Properly handle errors in poll set
  • Code refactoring to make code more readable
  • SockerAcceptor should handle error event

@JackyWoo JackyWoo added the enhancement Enhancements label Dec 12, 2023
@JackyWoo JackyWoo added this to the Release v2.0.4 milestone Dec 12, 2023
@JackyWoo
Copy link
Contributor Author

JackyWoo commented Dec 12, 2023 •
edited
Loading

Found TSan error

023-12-12T09:32:36.9474298Z WARNING: ThreadSanitizer: data race (pid=221)
2023-12-12T09:32:36.9474519Z   Read of size 1 at 0x7b6400031f78 by thread T117 (mutexes: write M0):
2023-12-12T09:32:36.9475516Z     #0 RK::ForwardingConnectionHandler::onSocketWritable(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/ForwardingConnectionHandler.cpp:301:13 (raftkeeper+0x8abe4a) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9476546Z     #1 RK::Observer<RK::ForwardingConnectionHandler, RK::WritableNotification>::notify(RK::Notification const&) const /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/Observer.h:74:17 (raftkeeper+0x8af2ee) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9477418Z     #2 RK::NotificationCenter::postNotification(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/NotificationCenter.cpp:89:19 (raftkeeper+0x874418) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9478196Z     #3 RK::SocketNotifier::dispatch(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:34:12 (raftkeeper+0x8753c0) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9479119Z     #4 RK::SocketReactor::dispatch(std::__1::shared_ptr<RK::SocketNotifier>&, RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:312:19 (raftkeeper+0x86ed18) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9479982Z     #5 RK::SocketReactor::dispatch(Poco::Net::Socket const&, RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:288:5 (raftkeeper+0x86d002) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9480573Z     #6 RK::SocketReactor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:70:29 (raftkeeper+0x86d002)
2023-12-12T09:32:36.9481370Z     #7 Poco::(anonymous namespace)::RunnableHolder::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:56:11 (raftkeeper+0xccd7bf) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9482196Z     #8 Poco::ThreadImpl::runnableEntry(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:358:27 (raftkeeper+0xccbec6) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9482203Z 
2023-12-12T09:32:36.9482468Z   Previous write of size 1 at 0x7b6400031f78 by thread T114 (mutexes: write M1):
2023-12-12T09:32:36.9483599Z     #0 RK::ForwardingConnectionHandler::ForwardingConnectionHandler(RK::Context&, Poco::Net::StreamSocket&, RK::SocketReactor&) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/ForwardingConnectionHandler.cpp:37:18 (raftkeeper+0x8a84b6) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9484572Z     #1 RK::SocketAcceptor<RK::ForwardingConnectionHandler>::createServiceHandler(Poco::Net::StreamSocket&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketAcceptor.h:118:30 (raftkeeper+0x7f38c1) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9485470Z     #2 RK::SocketAcceptor<RK::ForwardingConnectionHandler>::onAccept(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketAcceptor.h:84:9 (raftkeeper+0x7f3986) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9486508Z     #3 RK::Observer<RK::SocketAcceptor<RK::ForwardingConnectionHandler>, RK::ReadableNotification>::notify(RK::Notification const&) const /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/Observer.h:74:17 (raftkeeper+0x7f3aee) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9487434Z     #4 RK::NotificationCenter::postNotification(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/NotificationCenter.cpp:89:19 (raftkeeper+0x874418) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9488274Z     #5 RK::SocketNotifier::dispatch(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:34:12 (raftkeeper+0x8753c0) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9489191Z     #6 RK::SocketReactor::dispatch(std::__1::shared_ptr<RK::SocketNotifier>&, RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:312:19 (raftkeeper+0x86ed18) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9490052Z     #7 RK::SocketReactor::dispatch(Poco::Net::Socket const&, RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:288:5 (raftkeeper+0x86ce33) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9490659Z     #8 RK::SocketReactor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:65:29 (raftkeeper+0x86ce33)
2023-12-12T09:32:36.9491455Z     #9 Poco::(anonymous namespace)::RunnableHolder::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:56:11 (raftkeeper+0xccd7bf) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9492253Z     #10 Poco::ThreadImpl::runnableEntry(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:358:27 (raftkeeper+0xccbec6) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9492259Z 
2023-12-12T09:32:36.9492526Z   Location is heap block of size 1152 at 0x7b6400031b00 allocated by thread T114:
2023-12-12T09:32:36.9492817Z     #0 operator new(unsigned long) <null> (raftkeeper+0x7ea4c6) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9493791Z     #1 RK::SocketAcceptor<RK::ForwardingConnectionHandler>::createServiceHandler(Poco::Net::StreamSocket&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketAcceptor.h:118:26 (raftkeeper+0x7f38a3) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9494690Z     #2 RK::SocketAcceptor<RK::ForwardingConnectionHandler>::onAccept(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketAcceptor.h:84:9 (raftkeeper+0x7f3986) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9495925Z     #3 RK::Observer<RK::SocketAcceptor<RK::ForwardingConnectionHandler>, RK::ReadableNotification>::notify(RK::Notification const&) const /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/Observer.h:74:17 (raftkeeper+0x7f3aee) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9496809Z     #4 RK::NotificationCenter::postNotification(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/NotificationCenter.cpp:89:19 (raftkeeper+0x874418) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9497591Z     #5 RK::SocketNotifier::dispatch(RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:34:12 (raftkeeper+0x8753c0) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9498501Z     #6 RK::SocketReactor::dispatch(std::__1::shared_ptr<RK::SocketNotifier>&, RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:312:19 (raftkeeper+0x86ed18) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9499378Z     #7 RK::SocketReactor::dispatch(Poco::Net::Socket const&, RK::Notification const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:288:5 (raftkeeper+0x86ce33) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9499903Z     #8 RK::SocketReactor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:65:29 (raftkeeper+0x86ce33)
2023-12-12T09:32:36.9500696Z     #9 Poco::(anonymous namespace)::RunnableHolder::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:56:11 (raftkeeper+0xccd7bf) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9501487Z     #10 Poco::ThreadImpl::runnableEntry(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:358:27 (raftkeeper+0xccbec6) (BuildId: f9d11b090dbae287)
2023-12-12T09:32:36.9501562Z

@JackyWoo
Copy link
Contributor Author

@lzydmxy Please review the PR.

@JackyWoo
Copy link
Contributor Author

Need performance test.

@JackyWoo JackyWoo changed the title Refine nio framework Improve the NIO framework Jan 4, 2024
@JackyWoo JackyWoo force-pushed the refine_nio_framework branch from a564f67 to 69708e6 Compare January 4, 2024 06:28
@JackyWoo JackyWoo force-pushed the refine_nio_framework branch from 69708e6 to 3d613e7 Compare January 4, 2024 07:09
@JackyWoo
Copy link
Contributor Author

JackyWoo commented Jan 4, 2024

Performance test passed.

================= Result (Time measured in microsecond.) =================
thread_size,tps,avgRT(microsecond),TP90(microsecond),TP99(microsecond),TP999(microsecond),failRate
2000,82500,24810.0,37500.0,41600.0,53200.0,0.0


total requests: 2805000
fail count: 0
total time cost: 69593065016 microsecond
wallTime: 104 second

@JackyWoo JackyWoo merged commit 1e811b0 into JDRaftKeeper:master Jan 4, 2024
6 checks passed
@nicelulu nicelulu added enhancement Enhancements and removed enhancement Enhancements labels Feb 29, 2024
@JackyWoo JackyWoo added the performance Performance promotion label May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement Enhancements performance Performance promotion
Projects
None yet
Milestone
Release v2.0.4
Development

Successfully merging this pull request may close these issues.

Improve the NIO framework
2 participants
@JackyWoo @nicelulu