Skip to content
/ server Public
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRAFT: MDEV-34705: Storing binlog in InnoDB #3775

Draft
wants to merge 44 commits into
base: 11.4
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
019afb7
MDEV-34705: Binlog in Engine: Very first sketch, able to create and w…
knielsen Feb 25, 2024
ae128d0
MDEV-34705: Binlog in Engine: Early draft, first binlogging of DML to…
knielsen Jun 18, 2024
c2f41ca
MDEV-34705: Binlog in Engine: Allocate next binlog tablespace as needed.
knielsen Jun 19, 2024
7f06ab4
MDEV-34705: Binlog in Engine: Fix re-using ids for binlog tablespaces
knielsen Jun 21, 2024
2d5590b
MDEV-34705: Binlog in Engine: Start of binlog reader (untested, incom…
knielsen Jul 27, 2024
d0bff63
MDEV-34705: Binlog in Engine: Change option to --binlog-storage-engin…
knielsen Jul 27, 2024
c88c7a9
MDEV-34705: Binlog in Engine
knielsen Aug 7, 2024
0143421
MDEV-34705: Binlog in Engine
knielsen Aug 8, 2024
eece900
MDEV-34705: Binlog in Engine: Also binlog standalone (eg. DDL) in the…
knielsen Aug 9, 2024
8b63488
MDEV-34705: Binlog in Engine: Pre-allocate binlog tablespaces in back…
knielsen Aug 16, 2024
5f866b1
MDEV-34705: Binlog in Engine: Fix losing last part of event group > t…
knielsen Aug 24, 2024
eef6447
MDEV-34705: Binlog in Engine: Resume from existing binlogs
knielsen Sep 22, 2024
1975b78
MDEV-34705: Binlog in Engine: Refactor rpl_gtid.h
knielsen Oct 1, 2024
b4b873e
MDEV-34705: Binlog in Engine: Refactor nlz() in shared header
knielsen Oct 1, 2024
08f73b0
MDEV-34705: Binlog in Engine: Add functions for compressed int
knielsen Oct 1, 2024
d0e7018
MDEV-34705: Binlog in Engine: Refactor to add rpl_binlog_state_base::…
knielsen Oct 2, 2024
f76dccd
MDEV-34705: Binlog in Engine: Searchability for GTID position
knielsen Oct 3, 2024
37b2f16
MDEV-34705: Binlog in Engine: Clean up read_binlog_data()
knielsen Oct 9, 2024
40b1995
MDEV-34705: Code to restore binlog GTID state at restart
knielsen Oct 30, 2024
df538af
MDEV-34705: Fix race that could corrupt last mtr data in a tablespace
knielsen Oct 31, 2024
2c58d8a
MDEV-34705: Inplement starting from a specific GTID position
knielsen Nov 4, 2024
c62c132
MDEV-34705: Improved error handling when searching GTID position
knielsen Nov 8, 2024
a623e2e
MDEV-34705: Binlog-in-engine: Working replication to slave
knielsen Nov 12, 2024
1f6b8a0
MDEV-34705: Binlog-in-engine: Fix chunk flags
knielsen Nov 14, 2024
ffd3d16
MDEV-34705: Binlog-in-engine: Refactor fsp_binlog_write_cache()
knielsen Nov 15, 2024
7458fc3
MDEV-34705: out-of band binlogging, partial untested commit to do a s…
knielsen Nov 17, 2024
28cbd70
MDEV-34705: out-of band binlogging, fix trx_cache handling for out-of…
knielsen Nov 20, 2024
4f4c4f6
MDEV-34705: out-of band binlogging, link to oob data from commit reco…
knielsen Nov 20, 2024
8524966
MDEV-34705: Binlog-in-engine: Small visibility tweak in handler_binlo…
knielsen Nov 22, 2024
6e38794
MDEV-34705: Binlog-in-engine: Read side of out-of-band binlogging
knielsen Dec 21, 2024
c9da3dd
MDEV-34705: Binlog-in-engine: Refactor InnoDB part
knielsen Dec 30, 2024
31cfb98
MDEV-34705: Binlog-in-engine: Fix incorrect binlog data
knielsen Jan 3, 2025
3c9d248
MDEV-34705: Binlog-in-engine: Drop old X/X0Y.cc name convention for n…
knielsen Jan 5, 2025
88d94d3
MDEV-34705: Binlog-in-engine: Configurable binlog directory
knielsen Jan 6, 2025
ab260b1
MDEV-34705: Binlog-in-engine: Implement SHOW BINARY LOGS
knielsen Jan 7, 2025
fcd373e
MDEV-34705: Binlog-in-engine: Implement FLUSH BINARY LOGS
knielsen Jan 13, 2025
504f4ee
MDEV-34705: Binlog-in-engine: Implement RESET MASTER
knielsen Jan 14, 2025
0f6a19c
MDEV-34705: Binlog-in-engine: Misc. small fixes to make normal test s…
knielsen Jan 17, 2025
c60bea9
MDEV-34075: Binlog-in-engine: Some test and review fixes
knielsen Jan 17, 2025
18932be
MDEV-34705: Binlog-in-engine: Buildbot fixes
knielsen Jan 17, 2025
4814d20
MDEV-34705: Binlog-in-engine: Handful of fixes
knielsen Jan 22, 2025
633bcc8
MDEV-34705: Binlog-in-engine: Implement PURGE BINARY LOGS
knielsen Feb 4, 2025
cc4695b
MDEV-34705: Binlog-in-engine: Implement DELETE_DOMAIN_ID for FLUSH
knielsen Feb 11, 2025
56d1946
MDEV-34705: Binlog-in-engine: No use of InnoDB tablespace and bufferpool
knielsen Feb 15, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
MDEV-34705: Improved error handling when searching GTID position
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
  • Loading branch information
knielsen committed Jan 17, 2025
commit c62c132e2a85acc63c874de815650943a045e4cc
96 changes: 70 additions & 26 deletions storage/innobase/fsp/fsp0fsp.cc
Original file line number Diff line number Diff line change
@@ -5977,12 +5977,23 @@ innodb_get_binlog_reader()


class gtid_search {
/*
Note that this enum is set up to be compatible with int results -1/0/1 for
error/not found/fount from read_gtid_state_from_page().
*/
enum Read_Result {
READ_ENOENT= -2,
READ_ERROR= -1,
READ_NOT_FOUND= 0,
READ_FOUND= 1
};
public:
gtid_search();
~gtid_search();
int read_gtid_state_file_no(rpl_binlog_state_base *state, uint64_t file_no,
uint32_t page_no, uint64_t *out_file_end,
uint64_t *out_diff_state_interval);
enum Read_Result read_gtid_state_file_no(rpl_binlog_state_base *state,
uint64_t file_no, uint32_t page_no,
uint64_t *out_file_end,
uint64_t *out_diff_state_interval);
int find_gtid_pos(slave_connection_state *pos,
rpl_binlog_state_base *out_state, uint64_t *out_file_no,
uint64_t *out_offset);
@@ -6012,11 +6023,12 @@ gtid_search::~gtid_search()
Read a GTID state record from file_no and page_no.

Returns:
1 State record found
0 No state record found
-1 Error
READ_ERROR Error reading the file or corrupt data
READ_ENOENT File not found
READ_NOT_FOUND No GTID state record found on the page
READ_FOUND Record found
*/
int
enum gtid_search::Read_Result
gtid_search::read_gtid_state_file_no(rpl_binlog_state_base *state,
uint64_t file_no, uint32_t page_no,
uint64_t *out_file_end,
@@ -6027,7 +6039,7 @@ gtid_search::read_gtid_state_file_no(rpl_binlog_state_base *state,
*out_file_end= 0;
uint64_t active2= active_binlog_file_no.load(std::memory_order_acquire);
if (file_no > active2)
return 0;
return READ_ENOENT;

for (;;)
{
@@ -6037,8 +6049,16 @@ gtid_search::read_gtid_state_file_no(rpl_binlog_state_base *state,
uint64_t end_offset=
binlog_cur_end_offset[file_no&1].load(std::memory_order_acquire);
if (file_no + 1 >= active &&
end_offset != ~(uint64_t)0 &&
page_no <= (end_offset >> srv_page_size_shift))
{
/*
See if the page is available in the buffer pool.
Since we only use the low bit of file_no to determine the tablespace
id, the buffer pool page will only be valid if the active file_no did
not change while getting the page (otherwise it might belong to a
later tablespace file).
*/
mtr.start();
mtr_started= true;
uint32_t space_id= SRV_SPACE_ID_BINLOG0 + (file_no & 1);
@@ -6047,7 +6067,7 @@ gtid_search::read_gtid_state_file_no(rpl_binlog_state_base *state,
nullptr, BUF_GET_IF_IN_POOL, &mtr, &err);
if (err != DB_SUCCESS) {
mtr.commit();
return -1;
return READ_ERROR;
}
}
else
@@ -6063,21 +6083,28 @@ gtid_search::read_gtid_state_file_no(rpl_binlog_state_base *state,
if (file_no + 1 >= active)
{
*out_file_end= end_offset;
/*
Note: if end_offset is ~0, it means that the tablespace has been closed
and needs to be read as a plain file. Then this condition will be false
and we fall through to the file-reading code below, no need for an
extra conditional jump here.
*/
if (page_no > (end_offset >> srv_page_size_shift))
{
ut_ad(!mtr_started);
return 0;
return READ_NOT_FOUND;
}
}

if (block)
{
ut_ad(end_offset != ~(uint64_t)0);
int res= read_gtid_state_from_page(state, block->page.frame, page_no,
out_diff_state_interval);
ut_ad(mtr_started);
if (mtr_started)
mtr.commit();
return res;
return (Read_Result)res;
}
else
{
@@ -6096,24 +6123,28 @@ gtid_search::read_gtid_state_file_no(rpl_binlog_state_base *state,
{
char filename[BINLOG_NAME_LEN];
binlog_name_make(filename, file_no);
// ToDo: Here, if the file does not exist, it is not an error (no MY_WME), but it could mean we need to return "gtid pos too old". Or at least that we can go no further back and have to return the oldest state we found for the upper layer to deal with in terms of error etc.
cur_open_file= my_open(filename, O_RDONLY | O_BINARY, MYF(MY_WME));
cur_open_file= my_open(filename, O_RDONLY | O_BINARY, MYF(0));
if (cur_open_file < (File)0)
return -1;
{
if (errno == ENOENT)
return READ_ENOENT;
my_error(ER_CANT_OPEN_FILE, MYF(0), filename, errno);
return READ_ERROR;
}
MY_STAT stat_buf;
if (my_fstat(cur_open_file, &stat_buf, MYF(0))) {
my_error(ER_CANT_GET_STAT, MYF(0), filename, errno);
my_close(cur_open_file, MYF(0));
cur_open_file= (File)-1;
return -1;
return READ_ERROR;
}
cur_open_file_length= stat_buf.st_size;
cur_open_file_no= file_no;
}
if (!*out_file_end)
*out_file_end= cur_open_file_length;
return read_gtid_state(state, cur_open_file, page_no,
out_diff_state_interval);
return (Read_Result)read_gtid_state(state, cur_open_file, page_no,
out_diff_state_interval);
}
}
}
@@ -6150,13 +6181,23 @@ gtid_search::find_gtid_pos(slave_connection_state *pos,
base_state.init();
for (;;)
{
int res= read_gtid_state_file_no(&base_state, file_no, 0, &file_end,
&diff_state_interval);
// ToDo: catch the error "file does not exist" specifically and return "position not found" in this case.
if (res < 0)
enum Read_Result res=
read_gtid_state_file_no(&base_state, file_no, 0, &file_end,
&diff_state_interval);
if (res == READ_ENOENT)
return 0;
if (res == READ_ERROR)
return -1;
if (res == 0)
if (res == READ_NOT_FOUND)
{
if (file_no == 0)
{
/* Handle the special case of a completely empty binlog file. */
out_state->reset_nolock();
*out_file_no= file_no;
*out_offset= 0;
return 1;
}
ut_ad(0 /* Not expected to find no state, should always be written. */);
return -1;
}
@@ -6193,11 +6234,14 @@ gtid_search::find_gtid_pos(slave_connection_state *pos,
ut_ad((page1 - page0) % diff_state_interval == 0);
diff_state.reset_nolock();
diff_state.load_nolock(&base_state);
int res= read_gtid_state_file_no(&diff_state, file_no, 0, &file_end,
&diff_state_interval);
if (res < 0)
enum Read_Result res=
read_gtid_state_file_no(&diff_state, file_no, 0, &file_end,
&diff_state_interval);
if (res == READ_ENOENT)
return 0; /* File purged while we are reading from it? */
if (res == READ_ERROR)
return -1;
if (res == 0)
if (res == READ_NOT_FOUND)
{
/*
If the diff state record was not written here for some reason, just