Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add configuration max_log_segment_file_size and fix illegal open segment detection #366

Merged
merged 2 commits into from
Sep 29, 2024
Merged

Add configuration max_log_segment_file_size and fix illegal open segment detection #366

merged 2 commits into from
Sep 29, 2024

Conversation

JackyWoo
Copy link
Contributor

@JackyWoo JackyWoo commented Sep 29, 2024 •
edited
Loading

Which issues of this PR fixes:

This PR try to fix #365

Change log:

  1. Add configuration max_log_segment_file_size .
  2. Fix illegal open segment detection.
  3. Change type of MAX_LOG_SEGMENT_FILE_SIZE from uint32 to uint64 to avoid overflow
  4. Little code refactoring
  5. Fix duplicated comment on PR

@JackyWoo JackyWoo added the bug Something isn't working label Sep 29, 2024
@JackyWoo JackyWoo added this to the Release v2.2.0 milestone Sep 29, 2024
@JackyWoo JackyWoo requested a review from lzydmxy September 29, 2024 03:55
@JackyWoo JackyWoo self-assigned this Sep 29, 2024
@JackyWoo JackyWoo force-pushed the fix_open_segment_detect branch 2 times, most recently from 1895964 to 775df82 Compare September 29, 2024 04:07
@raftkeeper-robot
Copy link

raftkeeper-robot bot commented Sep 29, 2024 •
edited
Loading

Unit test report for commit 9048afd.

All test cases passed!

Successful Test Cases
Classname Name Sanitize Type Status Error Message
OperatorsManipTest EscapingTest ✅
OperatorsManipTest QuouteTest ✅
OperatorsManipTest DoubleQuouteTest ✅
OperatorsManipTest binary ✅
Logger Log ✅
Common PODArrayInsert ✅
Common PODPushBackRawMany ✅
Common PODNoOverallocation ✅
Common PODArrayInsertWithIllegalPadding ✅
Common unescapeForFileName ✅
ShellCommand Execute ✅
ShellCommand ExecuteDirect ✅
ShellCommand ExecuteWithInput ✅
ShellCommand AutoWait ✅
ThreadPool ConcurrentWait ✅
ThreadPool GlobalFull1 ✅
ThreadPool GlobalFull2 ✅
ThreadPool ThreadRemoval ✅
ThreadPool Loop ✅
ThreadPool ExceptionFromSchedule ✅
RaftLog writeAndReadUInt32 ✅
RaftLog serializeStr ✅
RaftLog serializeRaw ✅
RaftLog serializeEntry ✅
RaftLog parseLogEntrybody ✅
RaftLog appendEntry ✅
RaftLog appendEntries ✅
RaftLog loadLog ✅
RaftLog splitSegment ✅
RaftLog removeSegment ✅
RaftLog truncateLog ✅
RaftLog writeAt ✅
RaftLog compact ✅
RaftLog getEntry ✅
RaftLog getEntries ✅
RaftPerformance appendLogPerformance ✅
RaftPerformance appendLogThread ✅
RaftPerformance machineCreate ✅
RaftSnapshot parseAndSerializeKeeperNode ✅
RaftSnapshot createSnapshot_1 ✅
RaftSnapshot createSnapshot_2 ✅
RaftSnapshot readAndSaveSnapshot ✅
RaftSnapshot parseSnapshot ✅
RaftSnapshot createSnapshotWithFuzzyLog ✅
RaftStateMachine serializeAndParse ✅
RaftStateMachine appendEntry ✅
RaftStateMachine modifyEntry ✅
RaftStateMachine createSnapshot ✅
RaftStateMachine syncSnapshot ✅
RaftStateMachine initStateMachine ✅
RaftStateManager load_srv_state ✅

@raftkeeper-robot
Copy link

raftkeeper-robot bot commented Sep 29, 2024 •
edited
Loading

Integration test report for commit 9048afd.

All test cases passed!

Successful Test Cases
Classname Name Sanitize Type Status Error Message
test_auth test_digest_auth_basic[get_genuine_zk] ✅
test_auth test_digest_auth_basic[get_fake_zk] ✅
test_auth test_no_auth[get_genuine_zk] ✅
test_auth test_no_auth[get_fake_zk] ✅
test_auth test_super_auth ✅
test_auth test_digest_auth_multiple[get_genuine_zk] ✅
test_auth test_digest_auth_multiple[get_fake_zk] ✅
test_auth test_partial_auth[get_genuine_zk] ✅
test_auth test_partial_auth[get_fake_zk] ✅
test_auth test_bad_auth ✅
test_auth test_auth_snapshot ✅
test_auth test_get_set_acl[get_genuine_zk] ✅
test_auth test_get_set_acl[get_fake_zk] ✅
test_back_to_back test_simple_commands ✅
test_back_to_back test_sequential_nodes ✅
test_back_to_back test_stats ✅
test_back_to_back test_watchers ✅
test_back_to_back test_multi_transactions ✅
test_back_to_back test_filtered_list ✅
test_back_to_back test_multi_read ✅
test_back_to_back test_random_requests ✅
test_back_to_back test_end_of_session ✅
test_back_to_back test_end_of_watches_session ✅
test_back_to_back test_concurrent_watches ✅
test_back_to_back test_system_nodes ✅
test_back_to_back test_unregister_watch ✅
test_converter test_smoke[True] ✅
test_converter test_smoke[False] ✅
test_converter test_simple_crud_requests[True] ✅
test_converter test_simple_crud_requests[False] ✅
test_converter test_multi_and_failed_requests[True] ✅
test_converter test_multi_and_failed_requests[False] ✅
test_forward_expiration test_forward_expiration ✅
test_four_word_command test_cmd_ruok ✅
test_four_word_command test_cmd_mntr ✅
test_four_word_command test_cmd_srst ✅
test_four_word_command test_cmd_conf ✅
test_four_word_command test_cmd_isro ✅
test_four_word_command test_cmd_srvr ✅
test_four_word_command test_cmd_stat ✅
test_four_word_command test_cmd_cons ✅
test_four_word_command test_cmd_crst ✅
test_four_word_command test_cmd_dump ✅
test_four_word_command test_cmd_wchs ✅
test_four_word_command test_cmd_wchc ✅
test_four_word_command test_cmd_wchp ✅
test_four_word_command test_cmd_csnp ✅
test_four_word_command test_cmd_lgif ✅
test_four_word_command test_cmd_rqld ✅
test_four_word_command test_white_list ✅
test_learner test_read_write_multinode ✅
test_learner test_watch_on_follower ✅
test_learner test_session_expiration ✅
test_learner test_follower_restart ✅
test_learner test_simple_sleep_test ✅
test_learner test_stop_learner ✅
test_multinode_simple test_read_write_multi_node ✅
test_multinode_simple test_watch_on_follower ✅
test_multinode_simple test_session_expiration ✅
test_multinode_simple test_follower_restart ✅
test_multinode_simple test_simple_sleep_test ✅
test_nodes_add test_nodes_add ✅
test_nodes_remove test_nodes_remove ✅
test_nodes_replace test_node_replace ✅
test_non_snapshot_restart test_restart ✅
test_persistent_log test_open_and_closed_log_segment ✅
test_persistent_log test_state_after_restart ✅
test_persistent_log test_state_duplicate_restart ✅
test_persistent_log test_ephemeral_after_restart ✅
test_persistent_log_multinode test_restart_multi_node ✅
test_restore_from_snapshot test_recover_from_snapshot ✅
test_session test_reconnection ✅
test_session_fake_client test_session_timeout ✅
test_session_fake_client test_session_max_min_session_timeout ✅
test_session_fake_client test_invalid_timeout_setting ✅
test_snapshot_restart test_restart[False] ✅
test_snapshot_restart test_restart[True] ✅
test_snapshot_small_distance test_snapshot_and_load[False] ✅
test_snapshot_small_distance test_snapshot_and_load[True] ✅
test_snapshots test_state_after_restart[node0] ✅
test_snapshots test_state_after_restart[node1] ✅
test_snapshots test_ephemeral_after_restart[node0] ✅
test_snapshots test_ephemeral_after_restart[node1] ✅
test_snapshots test_restart_with_no_log[node0] ✅
test_snapshots test_restart_with_no_log[node1] ✅
test_snapshots test_snapshot_clear[node0] ✅
test_snapshots test_snapshot_clear[node1] ✅
test_snapshots_multinode test_restart_multinode[False] ✅
test_snapshots_multinode test_restart_multinode[True] ✅
test_stale_node_recovery test_stale_node_recovery ✅
test_three_nodes_two_alive test_start_offline ✅
test_three_nodes_two_alive test_start_non_existing ✅
test_three_nodes_two_alive test_restart_third_node ✅
test_two_nodes_cluster test_read_write_two_nodes ✅
test_two_nodes_cluster test_read_write_two_nodes_with_blocked ✅
test_znode_time test_between_servers ✅
test_znode_time test_server_restart ✅

@JackyWoo JackyWoo force-pushed the fix_open_segment_detect branch from 775df82 to bab8418 Compare September 29, 2024 05:18
@raftkeeper-robot
Copy link

Unit test report for commit 233903e.

All test cases passed!

Successful Test Cases
Classname Name Sanitize Type Status Error Message
OperatorsManipTest EscapingTest ✅
OperatorsManipTest QuouteTest ✅
OperatorsManipTest DoubleQuouteTest ✅
OperatorsManipTest binary ✅
Logger Log ✅
Common PODArrayInsert ✅
Common PODPushBackRawMany ✅
Common PODNoOverallocation ✅
Common PODArrayInsertWithIllegalPadding ✅
Common unescapeForFileName ✅
ShellCommand Execute ✅
ShellCommand ExecuteDirect ✅
ShellCommand ExecuteWithInput ✅
ShellCommand AutoWait ✅
ThreadPool ConcurrentWait ✅
ThreadPool GlobalFull1 ✅
ThreadPool GlobalFull2 ✅
ThreadPool ThreadRemoval ✅
ThreadPool Loop ✅
ThreadPool ExceptionFromSchedule ✅
RaftLog writeAndReadUInt32 ✅
RaftLog serializeStr ✅
RaftLog serializeRaw ✅
RaftLog serializeEntry ✅
RaftLog parseLogEntrybody ✅
RaftLog appendEntry ✅
RaftLog appendEntries ✅
RaftLog loadLog ✅
RaftLog splitSegment ✅
RaftLog removeSegment ✅
RaftLog truncateLog ✅
RaftLog writeAt ✅
RaftLog compact ✅
RaftLog getEntry ✅
RaftLog getEntries ✅
RaftPerformance appendLogPerformance ✅
RaftPerformance appendLogThread ✅
RaftPerformance machineCreate ✅
RaftSnapshot parseAndSerializeKeeperNode ✅
RaftSnapshot createSnapshot_1 ✅
RaftSnapshot createSnapshot_2 ✅
RaftSnapshot readAndSaveSnapshot ✅
RaftSnapshot parseSnapshot ✅
RaftSnapshot createSnapshotWithFuzzyLog ✅
RaftStateMachine serializeAndParse ✅
RaftStateMachine appendEntry ✅
RaftStateMachine modifyEntry ✅
RaftStateMachine createSnapshot ✅
RaftStateMachine syncSnapshot ✅
RaftStateMachine initStateMachine ✅
RaftStateManager load_srv_state ✅

@raftkeeper-robot
Copy link

Integration test report for commit 233903e.

All test cases passed!

Successful Test Cases
Classname Name Sanitize Type Status Error Message
test_auth test_digest_auth_basic[get_genuine_zk] ✅
test_auth test_digest_auth_basic[get_fake_zk] ✅
test_auth test_no_auth[get_genuine_zk] ✅
test_auth test_no_auth[get_fake_zk] ✅
test_auth test_super_auth ✅
test_auth test_digest_auth_multiple[get_genuine_zk] ✅
test_auth test_digest_auth_multiple[get_fake_zk] ✅
test_auth test_partial_auth[get_genuine_zk] ✅
test_auth test_partial_auth[get_fake_zk] ✅
test_auth test_bad_auth ✅
test_auth test_auth_snapshot ✅
test_auth test_get_set_acl[get_genuine_zk] ✅
test_auth test_get_set_acl[get_fake_zk] ✅
test_back_to_back test_simple_commands ✅
test_back_to_back test_sequential_nodes ✅
test_back_to_back test_stats ✅
test_back_to_back test_watchers ✅
test_back_to_back test_multi_transactions ✅
test_back_to_back test_filtered_list ✅
test_back_to_back test_multi_read ✅
test_back_to_back test_random_requests ✅
test_back_to_back test_end_of_session ✅
test_back_to_back test_end_of_watches_session ✅
test_back_to_back test_concurrent_watches ✅
test_back_to_back test_system_nodes ✅
test_back_to_back test_unregister_watch ✅
test_converter test_smoke[True] ✅
test_converter test_smoke[False] ✅
test_converter test_simple_crud_requests[True] ✅
test_converter test_simple_crud_requests[False] ✅
test_converter test_multi_and_failed_requests[True] ✅
test_converter test_multi_and_failed_requests[False] ✅
test_forward_expiration test_forward_expiration ✅
test_four_word_command test_cmd_ruok ✅
test_four_word_command test_cmd_mntr ✅
test_four_word_command test_cmd_srst ✅
test_four_word_command test_cmd_conf ✅
test_four_word_command test_cmd_isro ✅
test_four_word_command test_cmd_srvr ✅
test_four_word_command test_cmd_stat ✅
test_four_word_command test_cmd_cons ✅
test_four_word_command test_cmd_crst ✅
test_four_word_command test_cmd_dump ✅
test_four_word_command test_cmd_wchs ✅
test_four_word_command test_cmd_wchc ✅
test_four_word_command test_cmd_wchp ✅
test_four_word_command test_cmd_csnp ✅
test_four_word_command test_cmd_lgif ✅
test_four_word_command test_cmd_rqld ✅
test_four_word_command test_white_list ✅
test_learner test_read_write_multinode ✅
test_learner test_watch_on_follower ✅
test_learner test_session_expiration ✅
test_learner test_follower_restart ✅
test_learner test_simple_sleep_test ✅
test_learner test_stop_learner ✅
test_multinode_simple test_read_write_multi_node ✅
test_multinode_simple test_watch_on_follower ✅
test_multinode_simple test_session_expiration ✅
test_multinode_simple test_follower_restart ✅
test_multinode_simple test_simple_sleep_test ✅
test_nodes_add test_nodes_add ✅
test_nodes_remove test_nodes_remove ✅
test_nodes_replace test_node_replace ✅
test_non_snapshot_restart test_restart ✅
test_persistent_log test_open_and_closed_log_segment ✅
test_persistent_log test_state_after_restart ✅
test_persistent_log test_state_duplicate_restart ✅
test_persistent_log test_ephemeral_after_restart ✅
test_persistent_log_multinode test_restart_multi_node ✅
test_restore_from_snapshot test_recover_from_snapshot ✅
test_session test_reconnection ✅
test_session_fake_client test_session_timeout ✅
test_session_fake_client test_session_max_min_session_timeout ✅
test_session_fake_client test_invalid_timeout_setting ✅
test_snapshot_restart test_restart[False] ✅
test_snapshot_restart test_restart[True] ✅
test_snapshot_small_distance test_snapshot_and_load[False] ✅
test_snapshot_small_distance test_snapshot_and_load[True] ✅
test_snapshots test_state_after_restart[node0] ✅
test_snapshots test_state_after_restart[node1] ✅
test_snapshots test_ephemeral_after_restart[node0] ✅
test_snapshots test_ephemeral_after_restart[node1] ✅
test_snapshots test_restart_with_no_log[node0] ✅
test_snapshots test_restart_with_no_log[node1] ✅
test_snapshots test_snapshot_clear[node0] ✅
test_snapshots test_snapshot_clear[node1] ✅
test_snapshots_multinode test_restart_multinode[False] ✅
test_snapshots_multinode test_restart_multinode[True] ✅
test_stale_node_recovery test_stale_node_recovery ✅
test_three_nodes_two_alive test_start_offline ✅
test_three_nodes_two_alive test_start_non_existing ✅
test_three_nodes_two_alive test_restart_third_node ✅
test_two_nodes_cluster test_read_write_two_nodes ✅
test_two_nodes_cluster test_read_write_two_nodes_with_blocked ✅
test_znode_time test_between_servers ✅
test_znode_time test_server_restart ✅

@raftkeeper-robot
Copy link

Unit test report for commit 483e155.

All test cases passed!

Successful Test Cases
Classname Name Sanitize Type Status Error Message
OperatorsManipTest EscapingTest ✅
OperatorsManipTest QuouteTest ✅
OperatorsManipTest DoubleQuouteTest ✅
OperatorsManipTest binary ✅
Logger Log ✅
Common PODArrayInsert ✅
Common PODPushBackRawMany ✅
Common PODNoOverallocation ✅
Common PODArrayInsertWithIllegalPadding ✅
Common unescapeForFileName ✅
ShellCommand Execute ✅
ShellCommand ExecuteDirect ✅
ShellCommand ExecuteWithInput ✅
ShellCommand AutoWait ✅
ThreadPool ConcurrentWait ✅
ThreadPool GlobalFull1 ✅
ThreadPool GlobalFull2 ✅
ThreadPool ThreadRemoval ✅
ThreadPool Loop ✅
ThreadPool ExceptionFromSchedule ✅
RaftLog writeAndReadUInt32 ✅
RaftLog serializeStr ✅
RaftLog serializeRaw ✅
RaftLog serializeEntry ✅
RaftLog parseLogEntrybody ✅
RaftLog appendEntry ✅
RaftLog appendEntries ✅
RaftLog loadLog ✅
RaftLog splitSegment ✅
RaftLog removeSegment ✅
RaftLog truncateLog ✅
RaftLog writeAt ✅
RaftLog compact ✅
RaftLog getEntry ✅
RaftLog getEntries ✅
RaftPerformance appendLogPerformance ✅
RaftPerformance appendLogThread ✅
RaftPerformance machineCreate ✅
RaftSnapshot parseAndSerializeKeeperNode ✅
RaftSnapshot createSnapshot_1 ✅
RaftSnapshot createSnapshot_2 ✅
RaftSnapshot readAndSaveSnapshot ✅
RaftSnapshot parseSnapshot ✅
RaftSnapshot createSnapshotWithFuzzyLog ✅
RaftStateMachine serializeAndParse ✅
RaftStateMachine appendEntry ✅
RaftStateMachine modifyEntry ✅
RaftStateMachine createSnapshot ✅
RaftStateMachine syncSnapshot ✅
RaftStateMachine initStateMachine ✅
RaftStateManager load_srv_state ✅

@JackyWoo JackyWoo force-pushed the fix_open_segment_detect branch from 329ba78 to 86c74a4 Compare September 29, 2024 06:57
lzydmxy
lzydmxy approved these changes Sep 29, 2024
View reviewed changes
Copy link
Contributor

@lzydmxy lzydmxy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lzydmxy lzydmxy merged commit 732926d into JDRaftKeeper:master Sep 29, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lzydmxy lzydmxy lzydmxy approved these changes

Assignees

@JackyWoo JackyWoo

Labels
bug Something isn't working
Projects
None yet
Milestone
Release v2.2.0
Development

Successfully merging this pull request may close these issues.

Illegal open segment identifying
2 participants
@JackyWoo @lzydmxy