large binlog replication fix

[noname0443]

Pull request description

Describe what this PR fix

Replication on large binlogs in case the replica has been restarted.

Please provide steps to reproduce (if it's a bug)

You can take test config for mysync. There are steps to reproduce the bug:

1. cd ./tests/images
2. Insert into docker-compose.yml mysql2 and mysql3 the next lines

deploy:
 resources:
  limits:
   cpus: '0.1'

3. Run the next script:

docker rm -f $(docker ps -aq)

cd ../../
GOOS=linux go build -tags netgo,osusergo -o ./tests/images/mysql/mysync ./cmd/mysync/...
docker build --tag=mysync-test-base8.0 tests/images/base --build-arg MYSQL_VERSION=8.0
cd tests/images/
VERSION=8.0 docker-compose up --build -d

docker rm -f images-mysql3-1

docker cp images-zoo1-1:/etc/zk-ssl/ca.cert.pem .
docker cp images-zoo1-1:/etc/zk-ssl/server.crt .
docker cp images-zoo1-1:/etc/zk-ssl/server.key .

docker exec -it images-mysql1-1 mkdir -p /etc/zk-ssl
docker exec -it images-mysql2-1 mkdir -p /etc/zk-ssl

docker cp ca.cert.pem images-mysql1-1:/etc/zk-ssl/
docker cp ca.cert.pem images-mysql2-1:/etc/zk-ssl/

docker cp server.crt images-mysql1-1:/etc/zk-ssl/
docker cp server.crt images-mysql2-1:/etc/zk-ssl/

docker cp server.key images-mysql1-1:/etc/zk-ssl/
docker cp server.key images-mysql2-1:/etc/zk-ssl/

echo "Trying to add mysql1 to mysync..."
while
  timeout -k 70 60 docker exec images-mysql1-1 mysync host add mysql1
  [ $? != 0 ]
do
  echo "Let's try for another time..."
  sleep 2
done

echo "Trying to add mysql2 to mysync..."
while
  timeout -k 70 60 docker exec images-mysql1-1 mysync host add mysql2
  [ $? != 0 ]
do
  echo "Let's try for another time..."
  sleep 2
done

docker exec images-mysql2-1 mysync host add mysql1
docker exec images-mysql2-1 mysync host add mysql2

docker exec images-mysql1-1 mysql -e "CREATE TABLE test1.big_table (id integer, info text);"

VALUES="(1, 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')"
for ((a = 0; a < 15; a++)); do
  VALUES="${VALUES},${VALUES}"
done

BIG_QUERY="INSERT INTO test1.big_table (id, info) values ${VALUES}"

echo "${BIG_QUERY}" >query.txt
docker cp query.txt images-mysql1-1:/tmp/query.txt
docker exec images-mysql1-1 bash -c "mysql </tmp/query.txt"
docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"
docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"
docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"
docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"

docker exec images-mysql2-1 bash -c 'kill -9 $(ps -e -o pid,comm | grep mysqld | awk "{ print $1 }")'

docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"

docker exec images-mysql2-1 bash -c 'echo "server_id = 2" >> /etc/mysql/my.cnf'

docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"

docker exec images-mysql2-1 bash -c "service mysql start"

docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table"

4. After that, we will see a lot of STOP SLAVE IO_THREAD warnings in mysync manager's logs. If we don't see them, then we should kill mysql on replica and run command docker exec images-mysql1-1 mysql -e "INSERT INTO test1.big_table SELECT * FROM test1.big_table" for another time and then sudo service mysql start to take up mysql.

If you can, provide logs

Logs before

root@mysql2:/# cat /var/log/mysync.log | grep "calc active nodes"
2024-08-27T07:45:33Z ERROR: calc active nodes: mysql2 is down, deleting from active...
2024-08-27T07:45:35Z ERROR: calc active nodes: mysql2 is down, deleting from active...
2024-08-27T07:45:37Z ERROR: calc active nodes: mysql2 is down, deleting from active...
2024-08-27T07:45:43Z WARN: calc active nodes: mysql2 should become active, but it has data lag 154017228, delaying...
2024-08-27T07:45:46Z WARN: calc active nodes: mysql2 should become active, but it has data lag 148827375, delaying...
2024-08-27T07:45:47Z WARN: calc active nodes: mysql2 should become active, but it has data lag 164286151, delaying...
2024-08-27T07:45:50Z WARN: calc active nodes: mysql2 should become active, but it has data lag 122854828, delaying...
2024-08-27T07:45:52Z WARN: calc active nodes: mysql2 should become active, but it has data lag 118496183, delaying...
2024-08-27T07:45:55Z WARN: calc active nodes: mysql2 should become active, but it has data lag 122162333, delaying...
2024-08-27T07:45:56Z WARN: calc active nodes: mysql2 should become active, but it has data lag 123131826, delaying...
2024-08-27T07:45:58Z WARN: calc active nodes: mysql2 should become active, but it has data lag 164285998, delaying...
2024-08-27T07:46:00Z WARN: calc active nodes: mysql2 should become active, but it has data lag 164285998, delaying...
2024-08-27T07:46:03Z WARN: calc active nodes: mysql2 should become active, but it has data lag 117795541, delaying...
2024-08-27T07:46:04Z WARN: calc active nodes: mysql2 should become active, but it has data lag 111945995, delaying...
2024-08-27T07:46:06Z WARN: calc active nodes: mysql2 should become active, but it has data lag 123099238, delaying...
2024-08-27T07:46:08Z WARN: calc active nodes: mysql2 should become active, but it has data lag 123123679, delaying...
2024-08-27T07:46:09Z WARN: calc active nodes: mysql2 should become active, but it has data lag 123107385, delaying...
2024-08-27T07:46:12Z WARN: calc active nodes: mysql2 should become active, but it has data lag 123213515, delaying...
2024-08-27T07:46:14Z WARN: calc active nodes: mysql2 should become active, but it has data lag 120280376, delaying...
2024-08-27T07:46:16Z WARN: calc active nodes: mysql2 should become active, but it has data lag 122602271, delaying...

Logs after

root@mysql2:/# cat /var/log/mysync.log | grep "calc active"
2024-08-27T07:07:59Z ERROR: calc active nodes: mysql2 is down, deleting from active...
2024-08-27T07:08:01Z ERROR: calc active nodes: mysql2 is down, deleting from active...
2024-08-27T07:08:03Z ERROR: calc active nodes: mysql2 is down, deleting from active...
2024-08-27T07:08:09Z WARN: calc active nodes: mysql2 has data lag 158466804, but it's IO is working
2024-08-27T07:08:12Z WARN: calc active nodes: mysql2 has data lag 153585223, but it's IO is working
2024-08-27T07:08:14Z WARN: calc active nodes: mysql2 has data lag 143610514, but it's IO is working
2024-08-27T07:08:14Z WARN: calc active nodes: mysql2 has data lag 139211134, but it's IO is working
2024-08-27T07:08:16Z WARN: calc active nodes: mysql2 has data lag 118626535, but it's IO is working
2024-08-27T07:08:18Z WARN: calc active nodes: mysql2 has data lag 110332889, but it's IO is working
2024-08-27T07:08:27Z WARN: calc active nodes: mysql2 has data lag 109892951, but it's IO is working