diff --git a/doc/proxysql_cluster/proxysql_cluster_working.md b/doc/proxysql_cluster/proxysql_cluster_working.md new file mode 100644 index 0000000000..bb91a5089b --- /dev/null +++ b/doc/proxysql_cluster/proxysql_cluster_working.md @@ -0,0 +1,222 @@ +# Introduction +This documentation provides an in-depth look at the internal workings of the ProxySQL Cluster feature. It is intended for readers who are already familiar with the basic concepts and functionality of ProxySQL Cluster. + +# Prerequisites +Before reading this documentation, it is mandatory that the reader has gone through the official ProxySQL Cluster documentation available at [https://proxysql.com/documentation/proxysql-cluster/](https://proxysql.com/documentation/proxysql-cluster/). This will provide the necessary background knowledge and understanding of terminologies to understand the internal workings of the feature. + +# Important classes used by ProxySQL Cluster +This section describes the classes that are used by the ProxySQL Cluster feature. + +## ProxySQL_Cluster +The `ProxySQL_Cluster` class is the core component for managing ProxySQL clusters. It provides a wide range of operations and functionalities to handle and manage the cluster effectively. With `ProxySQL_Cluster`, you can add or remove nodes, manage node weights, monitor cluster status, and perform other essential tasks. + +## ProxySQL_Node_Entry +The `ProxySQL_Node_Entry` class represents an individual node within a ProxySQL cluster. It serves as a container for storing information and provides convenient methods to access node-specific details. It consists of important node attributes such as its status, weight, connection statistics, and contains global_checksum and checksum, epoch and version of individual modules of that node. + +## ProxySQL_Cluster_Nodes +The `ProxySQL_Cluster_Nodes` class serves as a centralized manager for all the nodes within a ProxySQL cluster. It provides a cohesive interface to handle operations related to the entire cluster's nodes efficiently. Through the `ProxySQL_Cluster_Nodes` class, you can easily manage and manipulate various node-specific actions within the cluster context. It consists of an unordered map that contains all the nodes. + +## ProxySQL_Node_Address +The `ProxySQL_Node_Address` class represents a ProxySQL node's address and encapsulates the hostname, port, node UUID, admin_mysql_ifaces, and optional IP address. + +## ProxySQL_GlobalVariables +The `ProxySQL_GlobalVariables` class contains checksum, epoch, and version of individual modules of the current ProxySQL instance. It provides a way to access and manage the global variables related to the ProxySQL Cluster. + +## ProxySQL_Checksum_Value +The `ProxySQL_Checksum_Value` class is used to store checksum values for modules. It includes member variables such as version and epoch to keep track of the version and epoch of the checksum value. + + +# Initializing Monitoring Threads +Each node within the ProxySQL cluster monitors core nodes to identify any updates in module configuration. This is achieved by dedicating a separate thread for each core node, for continuous monitoring for changes. + +## Load ProxySQL Servers +When the ProxySQL instance is started or the `LOAD PROXYSQL SERVERS TO RUNTIME` command is executed, the latest `proxysql_servers` table records are fetched from the database. This result set is passed to the ProxySQL Cluster's `load_servers_list` method. + +## Initialize ProxySQL Cluster Nodes +The retrieved result set is used to initialize the ProxySQL Cluster Nodes. The `load_servers_list` method of `ProxySQL_Cluster_Nodes` creates a map of all the nodes by utilizing the `ProxySQL_Node_Entry` objects. Each `ProxySQL_Node_Entry` object represents a node in the cluster and holds information such as the host, port, weight, and comment of the node. + +## Mark all nodes Inactive +Mark all the nodes in the `ProxySQL_Cluster_Nodes` map as inactive. + +## Check Node Existence and Update +For each node in the retrieved result set, the existence of the node is checked within the `ProxySQL_Cluster_Nodes` map. If the node already exists in the map, it is marked as active, and any changes in the weights and comments are updated accordingly. + +## Create New Node Entry +If a node does not exist in the map, a new `ProxySQL_Node_Entry` object is created for that node. The `ProxySQL_Node_Entry` is initialized with the host, port, weight, and comment of the node. This new entry is then inserted into the ProxySQL Cluster Nodes map. + +## Create Monitoring Thread +Subsequently, a monitoring thread is created for each node in the ProxySQL cluster. These monitoring threads are responsible for effectively managing and monitoring the status and performance of each node. The `ProxySQL_Node_Address`, containing the host and port of the node, is passed as an argument to the monitoring thread. + +## Remove Inactive Nodes +After iterating through all the entries in the result set and performing the necessary operations, any inactive nodes present in the `ProxySQL_Cluster_Nodes` map are deleted. This ensures that only active and relevant nodes remain within the cluster. + +## Flowchart: +```mermaid +flowchart TD + +subgraph Cluster Nodes Monitoring Threads + start[Start] --> load[LOAD PROXYSQL SERVERS TO RUNTIME] --> select[SELECT hostname, port, weight, comment FROM proxysql_servers ORDER BY hostname, port] + select -- ProxySQL_Cluster::load_servers_list -> ProxySQL_Cluster_Nodes::load_servers_list ->ProxySQL_Cluster_Nodes::set_all_inactive --> test + test[Set all nodes to inactive state] + test -- ProxySQL_Cluster::load_servers_list -> ProxySQL_Cluster_Nodes::load_servers_list --> check[Next node available in resultset?] + check -- NO --> preend[PREEND] + check -- YES
ProxySQL_Cluster_Nodes::umap_proxy_nodes --> check_map{Node exists in map?} + check_map -- YES --> update[Update node weight, comment, and set node state to active] + check_map -- NO --> create1[Create new ProxySQL_Node_Entry with node host, port, weight, and comment and set node state to active and insert it in the map
Create new thread for node and send ProxySQL_Node_Address with node host and port as argument to that thread] --> check + update --> check + preend[Remove all inactive nodes] --> END[END] + END[END] +end +``` + +# Monitoring Thread Working +This section describes how each monitoring thread representing a specific ProxySQL cluster node performs its operations. + +## Check ProxySQL Version +The thread compares the ProxySQL version of the node with the local instance's ProxySQL version by executing the `SELECT @@version` query. If the versions do not match, the connection with the remote peer is terminated. + +## Register Node +If the versions match, `PROXYSQL CLUSTER_NODE_UUID ProxySQL_GlobalVariables::uuid ProxySQL_Cluster::admin_mysql_ifaces` command is sent to the remote peer to register current node as a client. + +## Global Checksum +Global checksum is fetched from the remote peer by sending `SELECT GLOBAL_CHECKSUM()` query. The resultset is passed to the `ProxySQL_Cluster::Update_Global_Checksum` method, which finds the corresponding `ProxySQL_Node_Entry` in the `ProxySQL_Cluster_Nodes` map. If the new global checksum is different from the previously saved one, the global checksum is updated in the `ProxySQL_Node_Entry`. + +## Module Checksum +If the global checksum is updated, indicating changes in one or more modules. `SELECT * FROM runtime_checksums_values ORDER BY name` query is sent to the remote peer to fetch the latest checksums of all modules, including epoch and version. + +## Compare and Update Checksums +The resultset of the query is passed to the `ProxySQL_Cluster::Update_Node_Checksums` method, which finds the corresponding `ProxySQL_Node_Entry` in the `ProxySQL_Cluster_Nodes` map. The method then compares the checksum values of each module with the recently fetched resultset. If a checksum value is different, it is replaced with the latest one in the `ProxySQL_Node_Entry`, and the diff_check of that module is incremented. + +## Sync Configuration +For each module that has a `ProxySQL_Cluster::cluster_*module_name*_diffs_before_sync` value greater than zero, indicating that the module is enabled for syncing, the thread proceeds to check the node's version and epoch. If the node's version is greater than 1 and own_version is equal to 1 (means instance is just booted) or node's epoch is greater than own_epoch, then it checks diff_check. If diff_check is greater than `ProxySQL_Cluster::cluster_*module_name*_diffs_before_sync`, it fetches latest configuration of that module using `ProxySQL_Cluster::pull_*module_name*_from_peer` method. + +## Select Sync Source +The thread calls `ProxySQL_Cluster_Nodes::get_peer_to_sync_*module_name*` to iterate over `ProxySQL_Cluster_Nodes` map and find a node with a version greater than 1 and maximum epoch value among all nodes. This node is selected as source of truth for syncing. + +## Fetch and Apply Configuration +Connection is created to selected node using credentials obtained from `ProxySQL_Cluster::get_credentials` and fetches latest configuration for the module. After fetching configuration, it computes checksum locally and compares it with checksum received from node. If checksums match, changes are applied to runtime. Otherwise changes are discarded. + +## Save Configuration +If "save to disk" variable is set to true, configuration of that module is saved to disk ensuring persistence. + +## Flowchart (Simplified): +```mermaid + graph TB + A[Start] --> A1 + A1["Connect to remote peer and fetch ProxySQL version"] --> B + B{Compare ProxySQL version of remote peer with local instance} + B -->|Versions are not same| C[Close connection with the remote peer] + B -->|Versions are same| D[Register current node as a client] + D --> E[Fetch global checksum from remote peer] + E --> F{Compare global checksum with local global checksum} + F -->|Checksums are same| G[Do nothing] + F -->|Checksums are different| H[Override previous local global checksum with new one] + H --> K[Fetch checksums of all modules from peer node] + K --> N{Compare local module checksum_values with fetched remote peer resultset} + N -->|Checksums are same| O[Do nothing] + N -->|Checksums are different| P[Replace checksum and increment module diff_check] + P --> Q{Check if module cluster_*module_name*_diffs_before_sync > 0} + Q -->|Value is not greater than 0| R[Do nothing] + Q -->|Value is greater than 0| S{Own_version == 1 or local epoch > own_epoch?} + S -->|False| T[Do nothing] + S -->|True| U{diff_check >= cluster_*module_name*_diffs_before_sync?} + U -->|False| V[Do nothing] + U -->|True| X[Find node with version > 1 and max_epoch value] + X --> Y[Connect and fetch module's latest configuration from thet remote peer] + Y --> Y1[Compute checksum of fetched module configuration] + Y1 --> Z{Compare locally computed checksum with remote peer checksum} + Z -->|Checksums are not same| AA[Discard changes] + Z -->|Checksums are same| AB["Apply configuration changes to runtime"] + AB --> AC{Check cluster_*module_name*_save_to_disk == true?} + AC -->|False| AD[Do nothing] + AC -->|True| AE[Save configuration of the module to disk] + AE --> AF[End] +``` + +## Flowchart (Detailed): +```mermaid +graph TD + subgraph ProxySQL_Cluster_Monitor_thread[ProxySQL_Cluster_Monitor_thread] + direction TB + A[Start] -- "ProxySQL_Cluster_Monitor_thread
ProxySQL_Node_Address -> host, port" --> B + B[Connect to remote peer using host and port] --> C + C["Send `SELECT @@version`"] -- "resultset contains remote peer ProxySQL version" --> D + D{Remote peer ProxySQL version == Local PROXYSQL_VERSION?} + D -- "No" --> CLOSE_CONNECTION + D -- "Yes
Register yourself with a remote peer" --> E + E["Send `PROXYSQL CLUSTER_NODE_UUID ProxySQL_GlobalVariables::uuid ProxySQL_Cluster::admin_mysql_ifaces`"] --> F + F{ProxySQL_GlobalVariables::shutdown == 0} + F -- "Yes
Fetch Global Checksum from remote peer" --> G + F -- "No" --> CLOSE_CONNECTION + G["Send `SELECT GLOBAL_CHECKSUM()`"] -- "resultset contains global checksum of remote peer
ProxySQL_Cluster::Update_Global_Checksum" --> I + I{Local global checksum == Remote peer global checksum?} + I -- "Yes" --> AAA + I -- "No" --> J + J[Update local global checksum with remote peer global checksum value] -- "return checksum_updated = true
ProxySQL_Cluster_Monitor_thread

one or more module configuration had been changed. Fetching all the modules checksum, version and epoch from remote peer" --> K + K["Send `SELECT * FROM runtime_checksums_values ORDER BY name`"] -- "resultset contains module name, checksum, version and epoch
ProxySQL_Cluster_Nodes::Update_Node_Checksums
ProxySQL_Node_Entry::set_checksums" --> M + subgraph set_checksum[ProxySQL_Node_Entry::set_checksums] + direction TB + M[Load cluster_*module_name*_diffs_before_sync variables] --> N + N{resultset != NULL && record availabe in resultset?} -- "Yes

**logic is same for all module: module name == admin_variables
module name == mysql_query_rules
module name == mysql_servers
module name == mysql_users
module name == mysql_variables
module name == proxysql_servers
module name == ldap_variables**" --> O + O["Update local module version, epoch and last_updated value with the remote peer value"] --> P + P{Local module checksum == Remote peer module checksum?} + P -- "Yes" --> Q + P -- "No" --> S + Q[Set local module diff_check += 1] --> T + S[Update local module checksum with remote peer checksum value, last_changed to current time and diff_check = 1] --> T + T{Own module checksum == Local module checksum?} + T -- "Yes" --> U + T -- "No" --> N + U[Set local module diff_check = 0] --> N + N -- "No
cluster_*module_name*_diffs_before_sync variables" --> R + R{resultset == NULL?} + R -- "Yes
For every module" --> YY + YY[Set local module last_updated = current time] --> XX + XX{Local module checksum == Own module version?} + XX -- "Yes" --> WW + WW[Set local module diff_check = 0] --> V + XX -- "No" --> VV + VV[Set local module diff_check += 1] --> V + V{cluster_*module_name*_diffs_before_sync variables != 0?} + V -- "Yes" --> W + V -- "No" --> DELAY + W{Local module version > 1?} + W -- "Yes" --> X + W -- "No" --> V + X{"Own module version == 1 || Local module epoch > Own module epoch?"} + X -- "Yes" --> Y + X -- "No" --> V + Y{Local module diff_check >= cluster_*module_name*_diffs_before_sync?} + Y -- "No" --> V + Y -- "Yes
ProxySQL_Cluster_Nodes::umap_proxy_nodes" --> Y1 + Y1["Find local node with version > 1 and max_epoch"] -- "ProxySQL_Cluster::pull_*module_name*_from_peer" --> Z + Z["Connect to that remote peer and fetch latest module configuration"] --> AA + AA["Locally compute checksum of fetched configuration"] --> BB + BB{"Locally computed checksum == Remote peer checksum?"} + BB -- "Yes" --> DD + BB -- "No" --> DELAY + DD["Delete records in local module configuration table(s)"] --> EE + EE["Insert retrieved data from remote peer into local module configuration table(s)"] --> FF + FF["Issue internal `LOAD *module_name* TO RUNTIME`"] --> GG + GG{"Check cluster_*module_name*_save_to_disk == true?"} + GG -- "Yes" --> II + GG -- "No" --> AAA + II[Issue internal `SAVE *module_name* TO DISK`] + II --> AAA + end + AAA{"Local cluster_check_status_frequency_count >= cluster_check_status_frequency?"} + AAA -- "No" --> BBB + AAA -- Yes --> DDD + BBB[Set cluster_check_status_frequency_count += 1] --> DELAY + DDD[Set cluster_check_status_frequency_count = 0] --> EEE + EEE[Send `SELECT * FROM stats_mysql_global ORDER BY Variable_Name`] + EEE -- "resultset contains Client_Connections_connected, Client_Connections_created, ProxySQL_Uptime, Questions, Servers_table_version of remote peer
ProxySQL_Cluster_Nodes::Update_Node_Metrics
ProxySQL_Node_Entry::set_metrics" --> FFF + subgraph set_metrics[ProxySQL_Node_Entry::set_metrics] + direction TB + FFF[Update local metrices with values from resultset] --> DELAY + end + DELAY[Sleep cluster_check_interval_ms] --> F + CLOSE_CONNECTION[Close Connection] --> END + END[End] +end +``` \ No newline at end of file diff --git a/include/MySQL_HostGroups_Manager.h b/include/MySQL_HostGroups_Manager.h index 675cf090e2..e7f617ec5d 100644 --- a/include/MySQL_HostGroups_Manager.h +++ b/include/MySQL_HostGroups_Manager.h @@ -66,6 +66,55 @@ using json = nlohmann::json; #define MYHGM_MYSQL_HOSTGROUP_ATTRIBUTES "CREATE TABLE mysql_hostgroup_attributes (hostgroup_id INT NOT NULL PRIMARY KEY , max_num_online_servers INT CHECK (max_num_online_servers>=0 AND max_num_online_servers <= 1000000) NOT NULL DEFAULT 1000000 , autocommit INT CHECK (autocommit IN (-1, 0, 1)) NOT NULL DEFAULT -1 , free_connections_pct INT CHECK (free_connections_pct >= 0 AND free_connections_pct <= 100) NOT NULL DEFAULT 10 , init_connect VARCHAR NOT NULL DEFAULT '' , multiplex INT CHECK (multiplex IN (0, 1)) NOT NULL DEFAULT 1 , connection_warming INT CHECK (connection_warming IN (0, 1)) NOT NULL DEFAULT 0 , throttle_connections_per_sec INT CHECK (throttle_connections_per_sec >= 1 AND throttle_connections_per_sec <= 1000000) NOT NULL DEFAULT 1000000 , ignore_session_variables VARCHAR CHECK (JSON_VALID(ignore_session_variables) OR ignore_session_variables = '') NOT NULL DEFAULT '' , servers_defaults VARCHAR CHECK (JSON_VALID(servers_defaults) OR servers_defaults = '') NOT NULL DEFAULT '' , comment VARCHAR NOT NULL DEFAULT '')" +/* + * @brief Generates the 'runtime_mysql_servers' resultset exposed to other ProxySQL cluster members. + * @details Makes 'SHUNNED' and 'SHUNNED_REPLICATION_LAG' statuses equivalent to 'ONLINE'. 'SHUNNED' states + * are by definition local transitory states, this is why a 'mysql_servers' table reconfiguration isn't + * normally performed when servers are internally imposed with these statuses. This means, that propagating + * this state to other cluster members is undesired behavior, and so it's generating a different checksum, + * due to a server having this particular state, that will result in extra unnecessary fetching operations. + * The query also filters out 'OFFLINE_HARD' servers, 'OFFLINE_HARD' is a local status which is equivalent to + * a server no longer being part of the table (DELETED state). And so, they shouldn't be propagated. + * + * For placing the query into a single line for debugging purposes: + * ``` + * sed 's/^\t\+"//g; s/"\s\\$//g; s/\\"/"/g' /tmp/select.sql | paste -sd '' + * ``` + */ +#define MYHGM_GEN_CLUSTER_ADMIN_RUNTIME_SERVERS \ + "SELECT " \ + "hostgroup_id, hostname, port, gtid_port," \ + "CASE status" \ + " WHEN 0 THEN \"ONLINE\"" \ + " WHEN 1 THEN \"ONLINE\"" \ + " WHEN 2 THEN \"OFFLINE_SOFT\"" \ + " WHEN 3 THEN \"OFFLINE_HARD\"" \ + " WHEN 4 THEN \"ONLINE\" " \ + "END status," \ + "weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment " \ + "FROM mysql_servers " \ + "WHERE status != 3 " \ + "ORDER BY hostgroup_id, hostname, port" \ + +/** + * @brief Generates the 'mysql_servers_v2' resultset exposed to other ProxySQL cluster members. + * @details The generated resultset is used for the checksum computation of the runtime ProxySQL config + * ('mysql_servers_v2' checksum), and it's also forwarded to other cluster members when querying the Admin + * interface with 'CLUSTER_QUERY_MYSQL_SERVERS_V2'. It makes 'SHUNNED' state equivalent to 'ONLINE', and also + * filters out any 'OFFLINE_HARD' entries. This is done because none of the statuses are valid configuration + * statuses, they are local, transient status that ProxySQL uses during operation. + */ +#define MYHGM_GEN_CLUSTER_ADMIN_MYSQL_SERVERS \ + "SELECT " \ + "hostgroup_id, hostname, port, gtid_port, " \ + "CASE" \ + " WHEN status=\"SHUNNED\" THEN \"ONLINE\"" \ + " ELSE status " \ + "END AS status, " \ + "weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment " \ + "FROM main.mysql_servers " \ + "WHERE status != \"OFFLINE_HARD\" " \ + "ORDER BY hostgroup_id, hostname, port" \ typedef std::unordered_map umap_mysql_errors; @@ -446,12 +495,13 @@ class MySQL_HostGroups_Manager { #endif enum HGM_TABLES { - MYSQL_SERVERS = 0, + MYSQL_SERVERS_V2 = 0, MYSQL_REPLICATION_HOSTGROUPS, MYSQL_GROUP_REPLICATION_HOSTGROUPS, MYSQL_GALERA_HOSTGROUPS, MYSQL_AWS_AURORA_HOSTGROUPS, MYSQL_HOSTGROUP_ATTRIBUTES, + MYSQL_SERVERS, __HGM_TABLES_SIZE }; @@ -554,7 +604,7 @@ class MySQL_HostGroups_Manager { /** * @brief This resultset holds the current values for 'runtime_mysql_servers' computed by either latest * 'commit' or fetched from another Cluster node. It's also used by ProxySQL_Admin to respond to the - * intercepted query 'CLUSTER_QUERY_MYSQL_SERVERS'. + * intercepted query 'CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS'. * @details This resultset can't right now just contain the value for 'incoming_mysql_servers' as with the * rest of the intercepted resultset. This is due to 'runtime_mysql_servers' reconfigurations that can be * triggered by monitoring actions like 'Galera' currently performs. These actions not only trigger status @@ -605,6 +655,8 @@ class MySQL_HostGroups_Manager { void generate_mysql_hostgroup_attributes_table(); SQLite3_result *incoming_hostgroup_attributes; + SQLite3_result* incoming_mysql_servers_v2; + std::thread *HGCU_thread; std::thread *GTID_syncer_thread; @@ -748,9 +800,11 @@ class MySQL_HostGroups_Manager { void wrlock(); void wrunlock(); int servers_add(SQLite3_result *resultset); - bool commit(SQLite3_result* runtime_mysql_servers = nullptr, const std::string& checksum = "", const time_t epoch = 0); - void commit_update_checksums_from_tables(SpookyHash& myhash, bool& init); - void CUCFT1(SpookyHash& myhash, bool& init, const string& TableName, const string& ColumnName, uint64_t& raw_checksum); // used by commit_update_checksums_from_tables() + bool commit(SQLite3_result* runtime_mysql_servers = nullptr, const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server = {}, + SQLite3_result* mysql_servers_v2 = nullptr, const mysql_servers_v2_checksum_t& peer_mysql_server_v2 = {}, + bool only_commit_runtime_mysql_servers = true, bool update_version = false); + void commit_update_checksums_from_tables(); + void CUCFT1(const string& TableName, const string& ColumnName, uint64_t& raw_checksum); // used by commit_update_checksums_from_tables() /** * @brief Store the resultset for the 'runtime_mysql_servers' table set that have been loaded to runtime. @@ -758,6 +812,14 @@ class MySQL_HostGroups_Manager { * @param The resulset to be stored replacing the current one. */ void save_runtime_mysql_servers(SQLite3_result *); + + /** + * @brief Store the resultset for the 'mysql_servers_v2' table. + * The store configuration is later used by Cluster to propagate current config. + * @param The resulset to be stored replacing the current one. + */ + void save_mysql_servers_v2(SQLite3_result* s); + /** * @brief These setters/getter functions store and retrieve the currently hold resultset for the * 'incoming_*' table set that have been loaded to runtime. The store configuration is later used by @@ -769,7 +831,20 @@ class MySQL_HostGroups_Manager { SQLite3_result* get_current_mysql_table(const string& name); SQLite3_result * execute_query(char *query, char **error); - SQLite3_result *dump_table_mysql(const string&); + /** + * @brief Creates a resultset with the current full content of the target table. + * @param string The target table. Valid values are: + * - "mysql_aws_aurora_hostgroups" + * - "mysql_galera_hostgroups" + * - "mysql_group_replication_hostgroups" + * - "mysql_replication_hostgroups" + * - "mysql_hostgroup_attributes" + * - "mysql_servers" + * - "cluster_mysql_servers" + * When targeting 'mysql_servers' table is purged and regenerated. + * @return The generated resultset. + */ + SQLite3_result* dump_table_mysql(const string&); /** * @brief Update the public member resulset 'mysql_servers_to_monitor'. This resulset should contain the latest @@ -864,6 +939,11 @@ class MySQL_HostGroups_Manager { void shutdown(); void unshun_server_all_hostgroups(const char * address, uint16_t port, time_t t, int max_wait_sec, unsigned int *skip_hid); MySrvC* find_server_in_hg(unsigned int _hid, const std::string& addr, int port); + +private: + void update_hostgroup_manager_mappings(); + uint64_t get_mysql_servers_checksum(SQLite3_result* runtime_mysql_servers = nullptr); + uint64_t get_mysql_servers_v2_checksum(SQLite3_result* incoming_mysql_servers_v2 = nullptr); }; #endif /* __CLASS_MYSQL_HOSTGROUPS_MANAGER_H */ diff --git a/include/ProxySQL_Cluster.hpp b/include/ProxySQL_Cluster.hpp index 4ca9cfd7ff..feb7a71ed4 100644 --- a/include/ProxySQL_Cluster.hpp +++ b/include/ProxySQL_Cluster.hpp @@ -19,17 +19,20 @@ * the queries issued for generating the checksum for each of the target modules, for simpler reasoning, they should * also represent the actual resultset being received when issuing them, since this resultset is used for computing the * 'expected checksum' for the fetched config before loading it to runtime. This is done for the following modules: - * - 'runtime_mysql_servers': tables 'mysql_servers', 'mysql_replication_hostgroups', 'mysql_group_replication_hostroups', - * 'mysql_galera_hostgroups', 'mysql_aws_aurora_hostgroups', 'mysql_hostgroup_attributes'. + * - 'runtime_mysql_servers': tables 'mysql_servers' * - 'runtime_mysql_users'. * - 'runtime_mysql_query_rules'. - * + * - 'mysql_servers_v2': tables admin 'mysql_servers', 'mysql_replication_hostgroups', 'mysql_group_replication_hostroups', + * 'mysql_galera_hostgroups', 'mysql_aws_aurora_hostgroups', 'mysql_hostgroup_attributes'. * IMPORTANT: For further clarify this means that it's important that the actual resultset produced by the intercepted * query preserve the filtering and ordering expressed in this queries. */ /* @brief Query to be intercepted by 'ProxySQL_Admin' for 'runtime_mysql_servers'. See top comment for details. */ -#define CLUSTER_QUERY_MYSQL_SERVERS "PROXY_SELECT hostgroup_id, hostname, port, gtid_port, status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment FROM runtime_mysql_servers WHERE status<>'OFFLINE_HARD' ORDER BY hostgroup_id, hostname, port" +#define CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS "PROXY_SELECT hostgroup_id, hostname, port, gtid_port, status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment FROM runtime_mysql_servers WHERE status<>'OFFLINE_HARD' ORDER BY hostgroup_id, hostname, port" + +/* @brief Query to be intercepted by 'ProxySQL_Admin' for 'mysql_servers_v2'. See top comment for details. */ +#define CLUSTER_QUERY_MYSQL_SERVERS_V2 "PROXY_SELECT hostgroup_id, hostname, port, gtid_port, status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment FROM mysql_servers_v2 WHERE status<>'OFFLINE_HARD' ORDER BY hostgroup_id, hostname, port" /* @brief Query to be intercepted by 'ProxySQL_Admin' for 'runtime_mysql_replication_hostgroups'. See top comment for details. */ #define CLUSTER_QUERY_MYSQL_REPLICATION_HOSTGROUPS "PROXY_SELECT writer_hostgroup, reader_hostgroup, comment FROM runtime_mysql_replication_hostgroups ORDER BY writer_hostgroup" @@ -189,6 +192,7 @@ class ProxySQL_Node_Entry { ProxySQL_Checksum_Value_2 mysql_servers; ProxySQL_Checksum_Value_2 mysql_users; ProxySQL_Checksum_Value_2 proxysql_servers; + ProxySQL_Checksum_Value_2 mysql_servers_v2; } checksums_values; uint64_t global_checksum; }; @@ -279,7 +283,9 @@ class ProxySQL_Cluster_Nodes { SQLite3_result * stats_proxysql_servers_checksums(); SQLite3_result * stats_proxysql_servers_metrics(); void get_peer_to_sync_mysql_query_rules(char **host, uint16_t *port, char** ip_address); - void get_peer_to_sync_mysql_servers(char **host, uint16_t *port, char **peer_checksum, char** ip_address); + void get_peer_to_sync_runtime_mysql_servers(char **host, uint16_t *port, char **peer_checksum, char** ip_address); + void get_peer_to_sync_mysql_servers_v2(char** host, uint16_t* port, char** peer_mysql_servers_v2_checksum, + char** peer_runtime_mysql_servers_checksum, char** ip_address); void get_peer_to_sync_mysql_users(char **host, uint16_t *port, char** ip_address); void get_peer_to_sync_mysql_variables(char **host, uint16_t *port, char** ip_address); void get_peer_to_sync_admin_variables(char **host, uint16_t* port, char** ip_address); @@ -366,6 +372,31 @@ struct variable_type { }; }; +/** + * @brief Specifies the sync algorithm to use when pulling config from another ProxySQL cluster peer. + */ +enum class mysql_servers_sync_algorithm { + /** + * @brief Sync 'runtime_mysql_servers' and 'mysql_server_v2' from remote peer. + * @details This means that both config and runtime servers info are to be synced, in other words, both + * user promoted config and runtime changes performed by ProxySQL ('Monitor') in the remote peer will + * trigger the start of syncing operations. + */ + runtime_mysql_servers_and_mysql_servers_v2 = 1, + /** + * @brief Sync only mysql_server_v2 (config) from the remote peer. + * @details Since 'runtime_mysql_servers' isn't considered for fetching, only config changes promoted by + * the user in the remote peer will by acknowledge and trigger the start of a syncing operation. + */ + mysql_servers_v2 = 2, + /** + * @brief Dependent on whether ProxySQL has been started with the -M flag. + * @details If '-M' is not present, 'runtime_mysql_servers_and_mysql_servers_v2' is selected, otherwise + * 'mysql_servers_v2' is chosen. + */ + auto_select = 3 +}; + /** * @brief Simple struct for holding a query, and three messages to report * the progress of the query execution. @@ -378,21 +409,23 @@ struct fetch_query { }; class ProxySQL_Cluster { - private: +private: + SQLite3DB* mydb; pthread_mutex_t mutex; std::vector term_threads; ProxySQL_Cluster_Nodes nodes; - char *cluster_username; - char *cluster_password; + char* cluster_username; + char* cluster_password; struct { - std::array p_counter_array {}; - std::array p_gauge_array {}; + std::array p_counter_array{}; + std::array p_gauge_array{}; } metrics; int fetch_and_store(MYSQL* conn, const fetch_query& f_query, MYSQL_RES** result); friend class ProxySQL_Node_Entry; - public: +public: pthread_mutex_t update_mysql_query_rules_mutex; - pthread_mutex_t update_mysql_servers_mutex; + pthread_mutex_t update_runtime_mysql_servers_mutex; + pthread_mutex_t update_mysql_servers_v2_mutex; pthread_mutex_t update_mysql_users_mutex; pthread_mutex_t update_mysql_variables_mutex; pthread_mutex_t update_proxysql_servers_mutex; @@ -407,7 +440,7 @@ class ProxySQL_Cluster { */ SQLite3_result* proxysql_servers_to_monitor; - char *admin_mysql_ifaces; + char* admin_mysql_ifaces; int cluster_check_interval_ms; int cluster_check_status_frequency; int cluster_mysql_query_rules_diffs_before_sync; @@ -417,6 +450,7 @@ class ProxySQL_Cluster { int cluster_mysql_variables_diffs_before_sync; int cluster_ldap_variables_diffs_before_sync; int cluster_admin_variables_diffs_before_sync; + int cluster_mysql_servers_sync_algorithm; bool cluster_mysql_query_rules_save_to_disk; bool cluster_mysql_servers_save_to_disk; bool cluster_mysql_users_save_to_disk; @@ -428,7 +462,7 @@ class ProxySQL_Cluster { ~ProxySQL_Cluster(); void init() {}; void print_version(); - void load_servers_list(SQLite3_result *r, bool _lock = true) { + void load_servers_list(SQLite3_result* r, bool _lock = true) { nodes.load_servers_list(r, _lock); } void update_table_proxysql_servers_for_monitor(SQLite3_result* resultset) { @@ -440,17 +474,17 @@ class ProxySQL_Cluster { MySQL_Monitor::trigger_dns_cache_update(); } - void get_credentials(char **, char **); - void set_username(char *); - void set_password(char *); - void set_admin_mysql_ifaces(char *); - bool Update_Node_Metrics(char * _h, uint16_t _p, MYSQL_RES *_r, unsigned long long _response_time) { + void get_credentials(char**, char**); + void set_username(char*); + void set_password(char*); + void set_admin_mysql_ifaces(char*); + bool Update_Node_Metrics(char* _h, uint16_t _p, MYSQL_RES* _r, unsigned long long _response_time) { return nodes.Update_Node_Metrics(_h, _p, _r, _response_time); } - bool Update_Global_Checksum(char * _h, uint16_t _p, MYSQL_RES *_r) { + bool Update_Global_Checksum(char* _h, uint16_t _p, MYSQL_RES* _r) { return nodes.Update_Global_Checksum(_h, _p, _r); } - bool Update_Node_Checksums(char * _h, uint16_t _p, MYSQL_RES *_r = NULL) { + bool Update_Node_Checksums(char* _h, uint16_t _p, MYSQL_RES* _r = NULL) { return nodes.Update_Node_Checksums(_h, _p, _r); } void Reset_Global_Checksums(bool lock) { @@ -459,17 +493,19 @@ class ProxySQL_Cluster { SQLite3_result *dump_table_proxysql_servers() { return nodes.dump_table_proxysql_servers(); } - SQLite3_result * get_stats_proxysql_servers_checksums() { + SQLite3_result* get_stats_proxysql_servers_checksums() { return nodes.stats_proxysql_servers_checksums(); } - SQLite3_result * get_stats_proxysql_servers_metrics() { + SQLite3_result* get_stats_proxysql_servers_metrics() { return nodes.stats_proxysql_servers_metrics(); } void p_update_metrics(); void thread_ending(pthread_t); void join_term_thread(); void pull_mysql_query_rules_from_peer(const std::string& expected_checksum, const time_t epoch); - void pull_mysql_servers_from_peer(const std::string& expected_checksum, const time_t epoch); + void pull_runtime_mysql_servers_from_peer(const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server); + void pull_mysql_servers_v2_from_peer(const mysql_servers_v2_checksum_t& peer_mysql_server_v2, + const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server = {}, bool fetch_runtime_mysql_servers = false); void pull_mysql_users_from_peer(const std::string& expected_checksum, const time_t epoch); /** * @brief Pulls from peer the specified global variables by the type parameter. diff --git a/include/proxysql_admin.h b/include/proxysql_admin.h index 24a365294e..bda578fd78 100644 --- a/include/proxysql_admin.h +++ b/include/proxysql_admin.h @@ -118,6 +118,7 @@ struct admin_metrics_map_idx { extern int admin__web_verbosity; struct incoming_servers_t { + SQLite3_result* incoming_mysql_servers_v2 = NULL; SQLite3_result* runtime_mysql_servers = NULL; SQLite3_result* incoming_replication_hostgroups = NULL; SQLite3_result* incoming_group_replication_hostgroups = NULL; @@ -126,9 +127,27 @@ struct incoming_servers_t { SQLite3_result* incoming_hostgroup_attributes = NULL; incoming_servers_t(); - incoming_servers_t(SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*); + incoming_servers_t(SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*, SQLite3_result*); }; +// Separate structs for runtime mysql server and mysql server v2 to avoid human error +struct runtime_mysql_servers_checksum_t { + std::string checksum; + time_t epoch; + + runtime_mysql_servers_checksum_t(); + runtime_mysql_servers_checksum_t(const std::string& checksum, time_t epoch); +}; + +struct mysql_servers_v2_checksum_t { + std::string checksum; + time_t epoch; + + mysql_servers_v2_checksum_t(); + mysql_servers_v2_checksum_t(const std::string& checksum, time_t epoch); +}; +// + class ProxySQL_Admin { private: volatile int main_shutdown; @@ -186,6 +205,7 @@ class ProxySQL_Admin { int cluster_mysql_variables_diffs_before_sync; int cluster_admin_variables_diffs_before_sync; int cluster_ldap_variables_diffs_before_sync; + int cluster_mysql_servers_sync_algorithm; bool cluster_mysql_query_rules_save_to_disk; bool cluster_mysql_servers_save_to_disk; bool cluster_mysql_users_save_to_disk; @@ -410,7 +430,8 @@ class ProxySQL_Admin { // void flush_admin_variables__from_disk_to_memory(); // commented in 2.3 because unused void flush_admin_variables__from_memory_to_disk(); void flush_ldap_variables__from_memory_to_disk(); - void load_mysql_servers_to_runtime(const incoming_servers_t& incoming_servers = {}, const std::string& checksum = "", const time_t epoch = 0); + void load_mysql_servers_to_runtime(const incoming_servers_t& incoming_servers = {}, const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server = {}, + const mysql_servers_v2_checksum_t& peer_mysql_server_v2 = {}); void save_mysql_servers_from_runtime(); /** * @brief Performs the load to runtime of the current configuration in 'main' for 'mysql_query_rules' and diff --git a/include/proxysql_glovars.hpp b/include/proxysql_glovars.hpp index fe23e756fd..b6595cb9ba 100644 --- a/include/proxysql_glovars.hpp +++ b/include/proxysql_glovars.hpp @@ -149,6 +149,7 @@ class ProxySQL_GlobalVariables { ProxySQL_Checksum_Value mysql_variables; ProxySQL_Checksum_Value ldap_variables; ProxySQL_Checksum_Value proxysql_servers; + ProxySQL_Checksum_Value mysql_servers_v2; uint64_t global_checksum; unsigned long long updates_cnt; unsigned long long dumped_at; diff --git a/include/proxysql_structs.h b/include/proxysql_structs.h index 4352380c4b..29895a298e 100644 --- a/include/proxysql_structs.h +++ b/include/proxysql_structs.h @@ -140,6 +140,7 @@ enum debug_module { PROXY_DEBUG_QUERY_STATISTICS, PROXY_DEBUG_RESTAPI, PROXY_DEBUG_MONITOR, + PROXY_DEBUG_CLUSTER, PROXY_DEBUG_UNKNOWN // this module doesn't exist. It is used only to define the last possible module }; diff --git a/lib/MySQL_HostGroups_Manager.cpp b/lib/MySQL_HostGroups_Manager.cpp index 22c13d6398..22ddd6ac6a 100644 --- a/lib/MySQL_HostGroups_Manager.cpp +++ b/lib/MySQL_HostGroups_Manager.cpp @@ -1359,6 +1359,7 @@ MySQL_HostGroups_Manager::MySQL_HostGroups_Manager() { incoming_galera_hostgroups=NULL; incoming_aws_aurora_hostgroups = NULL; incoming_hostgroup_attributes = NULL; + incoming_mysql_servers_v2 = NULL; pthread_rwlock_init(>id_rwlock, NULL); gtid_missing_nodes = false; gtid_ev_loop=NULL; @@ -1589,7 +1590,7 @@ SQLite3_result * MySQL_HostGroups_Manager::execute_query(char *query, char **err return resultset; } -void MySQL_HostGroups_Manager::CUCFT1(SpookyHash& myhash, bool& init, const string& TableName, const string& ColumnName, uint64_t& raw_checksum) { +void MySQL_HostGroups_Manager::CUCFT1(const string& TableName, const string& ColumnName, uint64_t& raw_checksum) { char *error=NULL; int cols=0; int affected_rows=0; @@ -1598,14 +1599,8 @@ void MySQL_HostGroups_Manager::CUCFT1(SpookyHash& myhash, bool& init, const stri mydb->execute_statement(query.c_str(), &error , &cols , &affected_rows , &resultset); if (resultset) { if (resultset->rows_count) { - if (init == false) { - init = true; - myhash.Init(19,3); - } - uint64_t hash1_ = resultset->raw_checksum(); - raw_checksum = hash1_; - myhash.Update(&hash1_, sizeof(hash1_)); - proxy_info("Checksum for table %s is 0x%lX\n", TableName.c_str(), hash1_); + raw_checksum = resultset->raw_checksum(); + proxy_info("Checksum for table %s is 0x%lX\n", TableName.c_str(), raw_checksum); } delete resultset; } else { @@ -1613,17 +1608,98 @@ void MySQL_HostGroups_Manager::CUCFT1(SpookyHash& myhash, bool& init, const stri } } -void MySQL_HostGroups_Manager::commit_update_checksums_from_tables(SpookyHash& myhash, bool& init) { - CUCFT1(myhash,init,"mysql_replication_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]); - CUCFT1(myhash,init,"mysql_group_replication_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_GROUP_REPLICATION_HOSTGROUPS]); - CUCFT1(myhash,init,"mysql_galera_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_GALERA_HOSTGROUPS]); - CUCFT1(myhash,init,"mysql_aws_aurora_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_AWS_AURORA_HOSTGROUPS]); - CUCFT1(myhash,init,"mysql_hostgroup_attributes","hostgroup_id", table_resultset_checksum[HGM_TABLES::MYSQL_HOSTGROUP_ATTRIBUTES]); +void MySQL_HostGroups_Manager::commit_update_checksums_from_tables() { + CUCFT1("mysql_replication_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]); + CUCFT1("mysql_group_replication_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_GROUP_REPLICATION_HOSTGROUPS]); + CUCFT1("mysql_galera_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_GALERA_HOSTGROUPS]); + CUCFT1("mysql_aws_aurora_hostgroups","writer_hostgroup", table_resultset_checksum[HGM_TABLES::MYSQL_AWS_AURORA_HOSTGROUPS]); + CUCFT1("mysql_hostgroup_attributes","hostgroup_id", table_resultset_checksum[HGM_TABLES::MYSQL_HOSTGROUP_ATTRIBUTES]); +} + +/** + * @brief This code updates the 'hostgroup_server_mapping' table with the most recent mysql_servers and mysql_replication_hostgroups + * records while utilizing checksums to prevent unnecessary updates. + * + * IMPORTANT: Make sure wrlock() is called before calling this method. + * +*/ +void MySQL_HostGroups_Manager::update_hostgroup_manager_mappings() { + + if (hgsm_mysql_servers_checksum != table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS] || + hgsm_mysql_replication_hostgroups_checksum != table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]) + { + proxy_info("Rebuilding 'Hostgroup_Manager_Mapping' due to checksums change - mysql_servers { old: 0x%lX, new: 0x%lX }, mysql_replication_hostgroups { old:0x%lX, new:0x%lX }\n", + hgsm_mysql_servers_checksum, table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS], + hgsm_mysql_replication_hostgroups_checksum, table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]); + + char* error = NULL; + int cols = 0; + int affected_rows = 0; + SQLite3_result* resultset = NULL; + + hostgroup_server_mapping.clear(); + + const char* query = "SELECT DISTINCT hostname, port, '1' is_writer, status, reader_hostgroup, writer_hostgroup, mem_pointer FROM mysql_replication_hostgroups JOIN mysql_servers ON hostgroup_id=writer_hostgroup WHERE status<>3 \ + UNION \ + SELECT DISTINCT hostname, port, '0' is_writer, status, reader_hostgroup, writer_hostgroup, mem_pointer FROM mysql_replication_hostgroups JOIN mysql_servers ON hostgroup_id=reader_hostgroup WHERE status<>3 \ + ORDER BY hostname, port"; + + mydb->execute_statement(query, &error, &cols, &affected_rows, &resultset); + + if (resultset && resultset->rows_count) { + std::string fetched_server_id; + HostGroup_Server_Mapping* fetched_server_mapping = NULL; + + for (std::vector::iterator it = resultset->rows.begin(); it != resultset->rows.end(); ++it) { + SQLite3_row* r = *it; + + const std::string& server_id = std::string(r->fields[0]) + ":::" + r->fields[1]; + + if (fetched_server_mapping == NULL || server_id != fetched_server_id) { + + auto itr = hostgroup_server_mapping.find(server_id); + + if (itr == hostgroup_server_mapping.end()) { + std::unique_ptr server_mapping(new HostGroup_Server_Mapping(this)); + fetched_server_mapping = server_mapping.get(); + hostgroup_server_mapping.insert( std::pair> { + server_id, std::move(server_mapping) + } ); + } else { + fetched_server_mapping = itr->second.get(); + } + + fetched_server_id = server_id; + } + + HostGroup_Server_Mapping::Node node; + //node.server_status = static_cast(atoi(r->fields[3])); + node.reader_hostgroup_id = atoi(r->fields[4]); + node.writer_hostgroup_id = atoi(r->fields[5]); + node.srv = reinterpret_cast(atoll(r->fields[6])); + + HostGroup_Server_Mapping::Type type = (r->fields[2] && r->fields[2][0] == '1') ? HostGroup_Server_Mapping::Type::WRITER : HostGroup_Server_Mapping::Type::READER; + fetched_server_mapping->add(type, node); + } + } + delete resultset; + + hgsm_mysql_servers_checksum = table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS]; + hgsm_mysql_replication_hostgroups_checksum = table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]; + } } bool MySQL_HostGroups_Manager::commit( - SQLite3_result* runtime_mysql_servers, const std::string& checksum, const time_t epoch + SQLite3_result* runtime_mysql_servers, const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server, + SQLite3_result* mysql_servers_v2, const mysql_servers_v2_checksum_t& peer_mysql_server_v2, + bool only_commit_runtime_mysql_servers, bool update_version ) { + // if only_commit_runtime_mysql_servers is true, mysql_servers_v2 resultset will not be entertained and will cause memory leak. + if (only_commit_runtime_mysql_servers) { + proxy_info("Generating runtime mysql servers records only.\n"); + } else { + proxy_info("Generating runtime mysql servers and mysql servers v2 records.\n"); + } unsigned long long curtime1=monotonic_time(); wrlock(); @@ -1652,9 +1728,9 @@ bool MySQL_HostGroups_Manager::commit( } if (resultset) { delete resultset; resultset=NULL; } } - char *query=NULL; + char *query=NULL; query=(char *)"SELECT mem_pointer, t1.hostgroup_id, t1.hostname, t1.port FROM mysql_servers t1 LEFT OUTER JOIN mysql_servers_incoming t2 ON (t1.hostgroup_id=t2.hostgroup_id AND t1.hostname=t2.hostname AND t1.port=t2.port) WHERE t2.hostgroup_id IS NULL"; - mydb->execute_statement(query, &error , &cols , &affected_rows , &resultset); + mydb->execute_statement(query, &error , &cols , &affected_rows , &resultset); if (error) { proxy_error("Error on %s : %s\n", query, error); } else { @@ -1682,15 +1758,12 @@ bool MySQL_HostGroups_Manager::commit( //mydb->execute("DELETE FROM mysql_servers"); //generate_mysql_servers_table(); -// INSERT OR IGNORE INTO mysql_servers SELECT ... FROM mysql_servers_incoming -// proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "INSERT OR IGNORE INTO mysql_servers(hostgroup_id, hostname, port, weight, status, compression, max_connections) SELECT hostgroup_id, hostname, port, weight, status, compression, max_connections FROM mysql_servers_incoming\n"); mydb->execute("INSERT OR IGNORE INTO mysql_servers(hostgroup_id, hostname, port, gtid_port, weight, status, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment) SELECT hostgroup_id, hostname, port, gtid_port, weight, status, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment FROM mysql_servers_incoming"); - // SELECT FROM mysql_servers whatever is not identical in mysql_servers_incoming, or where mem_pointer=0 (where there is no pointer yet) query=(char *)"SELECT t1.*, t2.gtid_port, t2.weight, t2.status, t2.compression, t2.max_connections, t2.max_replication_lag, t2.use_ssl, t2.max_latency_ms, t2.comment FROM mysql_servers t1 JOIN mysql_servers_incoming t2 ON (t1.hostgroup_id=t2.hostgroup_id AND t1.hostname=t2.hostname AND t1.port=t2.port) WHERE mem_pointer=0 OR t1.gtid_port<>t2.gtid_port OR t1.weight<>t2.weight OR t1.status<>t2.status OR t1.compression<>t2.compression OR t1.max_connections<>t2.max_connections OR t1.max_replication_lag<>t2.max_replication_lag OR t1.use_ssl<>t2.use_ssl OR t1.max_latency_ms<>t2.max_latency_ms or t1.comment<>t2.comment"; proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "%s\n", query); - mydb->execute_statement(query, &error , &cols , &affected_rows , &resultset); + mydb->execute_statement(query, &error , &cols , &affected_rows , &resultset); if (error) { proxy_error("Error on %s : %s\n", query, error); } else { @@ -1833,202 +1906,100 @@ bool MySQL_HostGroups_Manager::commit( has_gtid_port = false; } if (resultset) { delete resultset; resultset=NULL; } - proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_servers_incoming\n"); - mydb->execute("DELETE FROM mysql_servers_incoming"); - - // replication - if (incoming_replication_hostgroups) { // this IF is extremely important, otherwise replication hostgroups may disappear - proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_replication_hostgroups\n"); - mydb->execute("DELETE FROM mysql_replication_hostgroups"); - generate_mysql_replication_hostgroups_table(); - } - // group replication - if (incoming_group_replication_hostgroups) { - proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_group_replication_hostgroups\n"); - mydb->execute("DELETE FROM mysql_group_replication_hostgroups"); - generate_mysql_group_replication_hostgroups_table(); - } - - // galera - if (incoming_galera_hostgroups) { - proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_galera_hostgroups\n"); - mydb->execute("DELETE FROM mysql_galera_hostgroups"); - generate_mysql_galera_hostgroups_table(); - } - - // AWS Aurora - if (incoming_aws_aurora_hostgroups) { - proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_aws_aurora_hostgroups\n"); - mydb->execute("DELETE FROM mysql_aws_aurora_hostgroups"); - generate_mysql_aws_aurora_hostgroups_table(); - } + uint64_t mysql_servers_v2_checksum = 0; - // hostgroup attributes - if (incoming_hostgroup_attributes) { - proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_hostgroup_attributes\n"); - mydb->execute("DELETE FROM mysql_hostgroup_attributes"); - generate_mysql_hostgroup_attributes_table(); - } - - // Checksums are always generated - 'admin-checksum_*' deprecated - { - uint64_t hash1 = 0, hash2 = 0; - SpookyHash myhash; - char buf[ProxySQL_Checksum_Value_LENGTH]; - bool init = false; - { - mydb->execute("DELETE FROM mysql_servers"); - generate_mysql_servers_table(); - char *error=NULL; - int cols=0; - int affected_rows=0; - SQLite3_result *resultset=NULL; - char *query=(char *)"SELECT hostgroup_id, hostname, port, gtid_port, CASE status WHEN 0 OR 1 OR 4 THEN 0 ELSE status END status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment FROM mysql_servers WHERE status<>3 ORDER BY hostgroup_id, hostname, port"; - mydb->execute_statement(query, &error , &cols , &affected_rows , &resultset); - if (runtime_mysql_servers == nullptr) { - char* error = NULL; - int cols = 0; - int affected_rows = 0; - SQLite3_result* resultset = NULL; - - mydb->execute_statement(MYHGM_GEN_ADMIN_RUNTIME_SERVERS, &error, &cols, &affected_rows, &resultset); - - // Remove 'OFFLINE_HARD' servers since they are not relevant to propagate to other Cluster - // nodes, or relevant for checksum computation. - const size_t init_row_count = resultset->rows_count; - size_t rm_rows_count = 0; - const auto is_offline_server = [&rm_rows_count] (SQLite3_row* row) { - if (strcasecmp(row->fields[4], "OFFLINE_HARD") == 0) { - rm_rows_count += 1; - return true; - } else { - return false; - } - }; - resultset->rows.erase( - std::remove_if(resultset->rows.begin(), resultset->rows.end(), is_offline_server), - resultset->rows.end() - ); - resultset->rows_count = init_row_count - rm_rows_count; - - save_runtime_mysql_servers(resultset); - } else { - save_runtime_mysql_servers(runtime_mysql_servers); - } - - // reset all checksum - table_resultset_checksum.fill(0); - - if (resultset) { - if (resultset->rows_count) { - if (init == false) { - init = true; - myhash.Init(19,3); - } - uint64_t hash1_ = resultset->raw_checksum(); - - table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS] = hash1_; + if (only_commit_runtime_mysql_servers == false) { + // replication + if (incoming_replication_hostgroups) { // this IF is extremely important, otherwise replication hostgroups may disappear + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_replication_hostgroups\n"); + mydb->execute("DELETE FROM mysql_replication_hostgroups"); + generate_mysql_replication_hostgroups_table(); + } - myhash.Update(&hash1_, sizeof(hash1_)); - proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers", hash1_); - } - delete resultset; - } else { - proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers", (long unsigned int)0); - } + // group replication + if (incoming_group_replication_hostgroups) { + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_group_replication_hostgroups\n"); + mydb->execute("DELETE FROM mysql_group_replication_hostgroups"); + generate_mysql_group_replication_hostgroups_table(); } - commit_update_checksums_from_tables(myhash, init); + // galera + if (incoming_galera_hostgroups) { + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_galera_hostgroups\n"); + mydb->execute("DELETE FROM mysql_galera_hostgroups"); + generate_mysql_galera_hostgroups_table(); + } - if (init == true) { - myhash.Final(&hash1, &hash2); + // AWS Aurora + if (incoming_aws_aurora_hostgroups) { + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_aws_aurora_hostgroups\n"); + mydb->execute("DELETE FROM mysql_aws_aurora_hostgroups"); + generate_mysql_aws_aurora_hostgroups_table(); } - uint32_t d32[2]; - memcpy(&d32,&hash1,sizeof(hash1)); - sprintf(buf,"0x%0X%0X", d32[0], d32[1]); - pthread_mutex_lock(&GloVars.checksum_mutex); - GloVars.checksums_values.mysql_servers.set_checksum(buf); - GloVars.checksums_values.mysql_servers.version++; - //struct timespec ts; - //clock_gettime(CLOCK_REALTIME, &ts); - time_t t = time(NULL); - if (epoch != 0 && checksum != "" && GloVars.checksums_values.mysql_servers.checksum == checksum) { - GloVars.checksums_values.mysql_servers.epoch = epoch; - } else { - GloVars.checksums_values.mysql_servers.epoch = t; + + // hostgroup attributes + if (incoming_hostgroup_attributes) { + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_hostgroup_attributes\n"); + mydb->execute("DELETE FROM mysql_hostgroup_attributes"); + generate_mysql_hostgroup_attributes_table(); } - GloVars.checksums_values.updates_cnt++; - GloVars.generate_global_checksum(); - GloVars.epoch_version = t; - pthread_mutex_unlock(&GloVars.checksum_mutex); + + mysql_servers_v2_checksum = get_mysql_servers_v2_checksum(mysql_servers_v2); } - // fill Hostgroup_Manager_Mapping with latest records - if (hgsm_mysql_servers_checksum != table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS] || - hgsm_mysql_replication_hostgroups_checksum != table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]) - { - proxy_info("Rebuilding 'Hostgroup_Manager_Mapping' due to checksums change - mysql_servers { old: 0x%lX, new: 0x%lX }, mysql_replication_hostgroups { old:0x%lX, new:0x%lX }\n", - hgsm_mysql_servers_checksum, table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS], - hgsm_mysql_replication_hostgroups_checksum, table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]); - - char* error = NULL; - int cols = 0; - int affected_rows = 0; - SQLite3_result* resultset = NULL; + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "DELETE FROM mysql_servers_incoming\n"); + mydb->execute("DELETE FROM mysql_servers_incoming"); + + // regenerating mysql_servers records + mydb->execute("DELETE FROM mysql_servers"); + generate_mysql_servers_table(); - const char* query = "SELECT DISTINCT hostname, port, '1' is_writer, status, reader_hostgroup, writer_hostgroup, mem_pointer FROM mysql_replication_hostgroups JOIN mysql_servers ON hostgroup_id=writer_hostgroup WHERE status<>3 \ - UNION \ - SELECT DISTINCT hostname, port, '0' is_writer, status, reader_hostgroup, writer_hostgroup, mem_pointer FROM mysql_replication_hostgroups JOIN mysql_servers ON hostgroup_id=reader_hostgroup WHERE status<>3 \ - ORDER BY hostname, port"; - - mydb->execute_statement(query, &error, &cols, &affected_rows, &resultset); + const auto mysql_servers_checksum = get_mysql_servers_checksum(runtime_mysql_servers); - hostgroup_server_mapping.clear(); + char buf[80]; + uint32_t d32[2]; + const time_t t = time(NULL); - if (resultset && resultset->rows_count) { - std::string fetched_server_id; - HostGroup_Server_Mapping* fetched_server_mapping = NULL; + memcpy(&d32, &mysql_servers_checksum, sizeof(mysql_servers_checksum)); + sprintf(buf, "0x%0X%0X", d32[0], d32[1]); + pthread_mutex_lock(&GloVars.checksum_mutex); + GloVars.checksums_values.mysql_servers.set_checksum(buf); - for (std::vector::iterator it = resultset->rows.begin(); it != resultset->rows.end(); ++it) { - SQLite3_row* r = *it; + if (update_version) + GloVars.checksums_values.mysql_servers.version++; - const std::string& server_id = std::string(r->fields[0]) + ":::" + r->fields[1]; + if (peer_runtime_mysql_server.epoch != 0 && peer_runtime_mysql_server.checksum.empty() == false && + GloVars.checksums_values.mysql_servers.checksum == peer_runtime_mysql_server.checksum) { + GloVars.checksums_values.mysql_servers.epoch = peer_runtime_mysql_server.epoch; + } else { + GloVars.checksums_values.mysql_servers.epoch = t; + } + + if (only_commit_runtime_mysql_servers == false) { + memcpy(&d32, &mysql_servers_v2_checksum, sizeof(mysql_servers_v2_checksum)); + sprintf(buf, "0x%0X%0X", d32[0], d32[1]); + GloVars.checksums_values.mysql_servers_v2.set_checksum(buf); - if (fetched_server_mapping == NULL || server_id != fetched_server_id) { - - auto itr = hostgroup_server_mapping.find(server_id); + if (update_version) + GloVars.checksums_values.mysql_servers_v2.version++; - if (itr == hostgroup_server_mapping.end()) { - std::unique_ptr server_mapping(new HostGroup_Server_Mapping(this)); - fetched_server_mapping = server_mapping.get(); - hostgroup_server_mapping.insert( std::pair> { - server_id, std::move(server_mapping) - } ); - } - else { - fetched_server_mapping = itr->second.get(); - } - - fetched_server_id = server_id; - } - - HostGroup_Server_Mapping::Node node; - //node.server_status = static_cast(atoi(r->fields[3])); - node.reader_hostgroup_id = atoi(r->fields[4]); - node.writer_hostgroup_id = atoi(r->fields[5]); - node.srv = reinterpret_cast(atoll(r->fields[6])); - - HostGroup_Server_Mapping::Type type = (r->fields[2] && r->fields[2][0] == '1') ? HostGroup_Server_Mapping::Type::WRITER : HostGroup_Server_Mapping::Type::READER; - fetched_server_mapping->add(type, node); - } + if (peer_mysql_server_v2.epoch != 0 && peer_mysql_server_v2.checksum.empty() == false && + GloVars.checksums_values.mysql_servers_v2.checksum == peer_mysql_server_v2.checksum) { + GloVars.checksums_values.mysql_servers_v2.epoch = peer_mysql_server_v2.epoch; + } else { + GloVars.checksums_values.mysql_servers_v2.epoch = t; } - delete resultset; - - hgsm_mysql_servers_checksum = table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS]; - hgsm_mysql_replication_hostgroups_checksum = table_resultset_checksum[HGM_TABLES::MYSQL_REPLICATION_HOSTGROUPS]; } + GloVars.checksums_values.updates_cnt++; + GloVars.generate_global_checksum(); + GloVars.epoch_version = t; + pthread_mutex_unlock(&GloVars.checksum_mutex); + + // fill Hostgroup_Manager_Mapping with latest records + update_hostgroup_manager_mappings(); + ev_async_send(gtid_ev_loop, gtid_ev_async); __sync_fetch_and_add(&status.servers_table_version,1); @@ -2061,6 +2032,159 @@ bool MySQL_HostGroups_Manager::commit( return true; } +/** + * @brief Calculate the checksum for the runtime mysql_servers record, after excluding all the rows + * with the status OFFLINE_HARD from the result set + * + * @details The runtime mysql_servers is now considered as a distinct module and have a separate checksum calculation. + * This is because the records in the runtime module may differ from those in the admin mysql_servers module, which + * can cause synchronization issues within the cluster. + * + * @param runtime_mysql_servers resultset of runtime mysql_servers or can be a nullptr. +*/ +uint64_t MySQL_HostGroups_Manager::get_mysql_servers_checksum(SQLite3_result* runtime_mysql_servers) { + + //Note: GloVars.checksum_mutex needs to be locked + SQLite3_result* resultset = nullptr; + + if (runtime_mysql_servers == nullptr) { + char* error = NULL; + int cols = 0; + int affected_rows = 0; + + mydb->execute_statement(MYHGM_GEN_CLUSTER_ADMIN_RUNTIME_SERVERS, &error, &cols, &affected_rows, &resultset); + + if (resultset) { + save_runtime_mysql_servers(resultset); + } else { + proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers", (long unsigned int)0); + } + } else { + resultset = runtime_mysql_servers; + save_runtime_mysql_servers(runtime_mysql_servers); + } + + table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS] = resultset != nullptr ? resultset->raw_checksum() : 0; + proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers", table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS]); + + return table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS]; +} + +/** + * @brief Computes checksum for the admin mysql_server resultset and generates an accumulated checksum by including the following modules: + * MYSQL_REPLICATION_HOSTGROUPS, MYSQL_GROUP_REPLICATION_HOSTGROUPS, MYSQL_GALERA_HOSTGROUPS, and MYSQL_HOSTGROUP_ATTRIBUTES. + * + * @details The 'incoming_mysql_servers_v2' includes the same records as 'main.mysql_servers'. If this set of records is empty, then the + * records are retrieved from 'main.mysql_servers'. + * + * @param incoming_mysql_servers_v2 resultset of admin mysql_servers or can be a nullptr. + */ +uint64_t MySQL_HostGroups_Manager::get_mysql_servers_v2_checksum(SQLite3_result* incoming_mysql_servers_v2) { + + //Note: GloVars.checksum_mutex needs to be locked + SQLite3_result* resultset = nullptr; + + if (incoming_mysql_servers_v2 == nullptr) + { + char* error = nullptr; + int cols = 0; + int affected_rows = 0; + + GloAdmin->admindb->execute_statement(MYHGM_GEN_CLUSTER_ADMIN_MYSQL_SERVERS, &error, &cols, &affected_rows, &resultset); + + if (resultset) { + save_mysql_servers_v2(resultset); + } else { + proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers_v2", (long unsigned int)0); + } + } else { + resultset = incoming_mysql_servers_v2; + save_mysql_servers_v2(incoming_mysql_servers_v2); + } + + // reset checksum excluding MYSQL_SERVERS_V2 + table_resultset_checksum[MYSQL_REPLICATION_HOSTGROUPS] = 0; + table_resultset_checksum[MYSQL_GROUP_REPLICATION_HOSTGROUPS] = 0; + table_resultset_checksum[MYSQL_GALERA_HOSTGROUPS] = 0; + table_resultset_checksum[MYSQL_AWS_AURORA_HOSTGROUPS] = 0; + table_resultset_checksum[MYSQL_HOSTGROUP_ATTRIBUTES] = 0; + + table_resultset_checksum[MYSQL_SERVERS_V2] = resultset != nullptr ? resultset->raw_checksum() : 0; + proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers_v2", table_resultset_checksum[MYSQL_SERVERS_V2]); + + commit_update_checksums_from_tables(); + + uint64_t hash1 = 0, hash2 = 0; + SpookyHash myhash; + bool init = false; + + hash1 = table_resultset_checksum[MYSQL_SERVERS_V2]; + if (hash1) { + if (init == false) { + init = true; + myhash.Init(19, 3); + } + + myhash.Update(&hash1, sizeof(hash1)); + } + + hash1 = table_resultset_checksum[MYSQL_REPLICATION_HOSTGROUPS]; + if (hash1) { + if (init == false) { + init = true; + myhash.Init(19, 3); + } + + myhash.Update(&hash1, sizeof(hash1)); + } + + hash1 = table_resultset_checksum[MYSQL_GROUP_REPLICATION_HOSTGROUPS]; + if (hash1) { + if (init == false) { + init = true; + myhash.Init(19, 3); + } + + myhash.Update(&hash1, sizeof(hash1)); + } + + hash1 = table_resultset_checksum[MYSQL_GALERA_HOSTGROUPS]; + if (hash1) { + if (init == false) { + init = true; + myhash.Init(19, 3); + } + + myhash.Update(&hash1, sizeof(hash1)); + } + + hash1 = table_resultset_checksum[MYSQL_AWS_AURORA_HOSTGROUPS]; + if (hash1) { + if (init == false) { + init = true; + myhash.Init(19, 3); + } + + myhash.Update(&hash1, sizeof(hash1)); + } + + hash1 = table_resultset_checksum[MYSQL_HOSTGROUP_ATTRIBUTES]; + if (hash1) { + if (init == false) { + init = true; + myhash.Init(19, 3); + } + + myhash.Update(&hash1, sizeof(hash1)); + } + + if (init == true) { + myhash.Final(&hash1, &hash2); + } + + return hash1; +} + bool MySQL_HostGroups_Manager::gtid_exists(MySrvC *mysrvc, char * gtid_uuid, uint64_t gtid_trxid) { bool ret = false; pthread_rwlock_rdlock(>id_rwlock); @@ -2593,7 +2717,9 @@ SQLite3_result * MySQL_HostGroups_Manager::dump_table_mysql(const string& name) } else if (name == "mysql_hostgroup_attributes") { query=(char *)"SELECT hostgroup_id, max_num_online_servers, autocommit, free_connections_pct, init_connect, multiplex, connection_warming, throttle_connections_per_sec, ignore_session_variables, servers_defaults, comment FROM mysql_hostgroup_attributes ORDER BY hostgroup_id"; } else if (name == "mysql_servers") { - query = (char *)MYHGM_GEN_ADMIN_RUNTIME_SERVERS; + query = (char *)MYHGM_GEN_ADMIN_RUNTIME_SERVERS; + } else if (name == "cluster_mysql_servers") { + query = (char *)MYHGM_GEN_CLUSTER_ADMIN_RUNTIME_SERVERS; } else { assert(0); } @@ -3879,6 +4005,14 @@ void MySQL_HostGroups_Manager::save_runtime_mysql_servers(SQLite3_result *s) { runtime_mysql_servers=s; } +void MySQL_HostGroups_Manager::save_mysql_servers_v2(SQLite3_result* s) { + if (incoming_mysql_servers_v2) { + delete incoming_mysql_servers_v2; + incoming_mysql_servers_v2 = nullptr; + } + incoming_mysql_servers_v2 = s; +} + SQLite3_result* MySQL_HostGroups_Manager::get_current_mysql_table(const string& name) { if (name == "mysql_aws_aurora_hostgroups") { return this->incoming_aws_aurora_hostgroups; @@ -3890,8 +4024,10 @@ SQLite3_result* MySQL_HostGroups_Manager::get_current_mysql_table(const string& return this->incoming_replication_hostgroups; } else if (name == "mysql_hostgroup_attributes") { return this->incoming_hostgroup_attributes; - } else if (name == "mysql_servers") { + } else if (name == "cluster_mysql_servers") { return this->runtime_mysql_servers; + } else if (name == "mysql_servers_v2") { + return this->incoming_mysql_servers_v2; } else { assert(0); } @@ -4752,72 +4888,23 @@ void MySQL_HostGroups_Manager::read_only_action_v2(const std::listexecute("DELETE FROM mysql_servers"); generate_mysql_servers_table(); - //if (GloAdmin && GloAdmin->checksum_variables.checksum_mysql_servers) - { - char* error = NULL; - int cols = 0; - int affected_rows = 0; - SQLite3_result* resultset = NULL; - mydb->execute_statement(MYHGM_GEN_ADMIN_RUNTIME_SERVERS, &error, &cols, &affected_rows, &resultset); - save_runtime_mysql_servers(resultset); // assigning runtime_mysql_servers with updated mysql server resultset - - resultset = NULL; - - // reset mysql_server checksum - table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS] = 0; - hgsm_mysql_servers_checksum = 0; - - char* query = (char*)"SELECT hostgroup_id, hostname, port, gtid_port, CASE status WHEN 0 OR 1 OR 4 THEN 0 ELSE status END status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment FROM mysql_servers WHERE status<>3 ORDER BY hostgroup_id, hostname, port"; - mydb->execute_statement(query, &error, &cols, &affected_rows, &resultset); - - if (resultset) { - if (resultset->rows_count) { - table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS] = resultset->raw_checksum(); - - hgsm_mysql_servers_checksum = table_resultset_checksum[HGM_TABLES::MYSQL_SERVERS]; - } - delete resultset; - } else { - proxy_info("Checksum for table %s is 0x%lX\n", "mysql_servers", (long unsigned int)0); - } - - uint64_t hash = 0, hash2 = 0; - SpookyHash myhash; - bool init = false; + const auto mysql_servers_checksum = get_mysql_servers_checksum(); + hgsm_mysql_servers_checksum = mysql_servers_checksum; - for (uint64_t hash_val : table_resultset_checksum) { - if (hash_val) { - if (init == false) { - init = true; - myhash.Init(19, 3); - } - - myhash.Update(&hash_val, sizeof(hash_val)); - } - } - - if (init == true) { - myhash.Final(&hash, &hash2); - } - - char buf[ProxySQL_Checksum_Value_LENGTH]; - uint32_t d32[2]; - memcpy(&d32, &hash, sizeof(hash)); - sprintf(buf, "0x%0X%0X", d32[0], d32[1]); - pthread_mutex_lock(&GloVars.checksum_mutex); - GloVars.checksums_values.mysql_servers.set_checksum(buf); - GloVars.checksums_values.mysql_servers.version++; - //struct timespec ts; - //clock_gettime(CLOCK_REALTIME, &ts); - time_t t = time(NULL); + char buf[ProxySQL_Checksum_Value_LENGTH]; + uint32_t d32[2]; + memcpy(&d32, &mysql_servers_checksum, sizeof(mysql_servers_checksum)); + sprintf(buf, "0x%0X%0X", d32[0], d32[1]); + pthread_mutex_lock(&GloVars.checksum_mutex); + GloVars.checksums_values.mysql_servers.set_checksum(buf); + time_t t = time(NULL); - GloVars.checksums_values.mysql_servers.epoch = t; + GloVars.checksums_values.mysql_servers.epoch = t; - GloVars.checksums_values.updates_cnt++; - GloVars.generate_global_checksum(); - GloVars.epoch_version = t; - pthread_mutex_unlock(&GloVars.checksum_mutex); - } + GloVars.checksums_values.updates_cnt++; + GloVars.generate_global_checksum(); + GloVars.epoch_version = t; + pthread_mutex_unlock(&GloVars.checksum_mutex); } wrunlock(); unsigned long long curtime2 = monotonic_time(); diff --git a/lib/ProxySQL_Admin.cpp b/lib/ProxySQL_Admin.cpp index 5e46ffc161..1626e6c535 100644 --- a/lib/ProxySQL_Admin.cpp +++ b/lib/ProxySQL_Admin.cpp @@ -666,6 +666,7 @@ static char * admin_variables_names[]= { (char *)"cluster_mysql_variables_save_to_disk", (char *)"cluster_admin_variables_save_to_disk", (char *)"cluster_ldap_variables_save_to_disk", + (char *)"cluster_mysql_servers_sync_algorithm", (char *)"checksum_mysql_query_rules", (char *)"checksum_mysql_servers", (char *)"checksum_mysql_users", @@ -1075,21 +1076,34 @@ bool FlushCommandWrapper(MySQL_Session *sess, const string& modname, char *query incoming_servers_t::incoming_servers_t() {} incoming_servers_t::incoming_servers_t( - SQLite3_result* runtime_mysql_servers, + SQLite3_result* incoming_mysql_servers_v2, SQLite3_result* incoming_replication_hostgroups, SQLite3_result* incoming_group_replication_hostgroups, SQLite3_result* incoming_galera_hostgroups, SQLite3_result* incoming_aurora_hostgroups, - SQLite3_result* incoming_hostgroup_attributes + SQLite3_result* incoming_hostgroup_attributes, + SQLite3_result* runtime_mysql_servers ) : - runtime_mysql_servers(runtime_mysql_servers), + incoming_mysql_servers_v2(incoming_mysql_servers_v2), incoming_replication_hostgroups(incoming_replication_hostgroups), incoming_group_replication_hostgroups(incoming_group_replication_hostgroups), incoming_galera_hostgroups(incoming_galera_hostgroups), incoming_aurora_hostgroups(incoming_aurora_hostgroups), - incoming_hostgroup_attributes(incoming_hostgroup_attributes) + incoming_hostgroup_attributes(incoming_hostgroup_attributes), + runtime_mysql_servers(runtime_mysql_servers) {} +runtime_mysql_servers_checksum_t::runtime_mysql_servers_checksum_t() : epoch(0) {} + +runtime_mysql_servers_checksum_t::runtime_mysql_servers_checksum_t(const std::string& checksum, time_t epoch) : + checksum(checksum), epoch(epoch) {} + +mysql_servers_v2_checksum_t::mysql_servers_v2_checksum_t() : epoch(0) {} + +mysql_servers_v2_checksum_t::mysql_servers_v2_checksum_t(const std::string& checksum, time_t epoch) : + checksum(checksum), epoch(epoch) {} + + int ProxySQL_Test___GetDigestTable(bool reset, bool use_swap) { int r = 0; if (!GloQPro) return 0; @@ -3745,8 +3759,8 @@ void admin_session_handler(MySQL_Session *sess, void *_pa, PtrSize_t *pkt) { if (sess->session_type == PROXYSQL_SESSION_ADMIN) { // no stats string tn = ""; - if (!strncasecmp(CLUSTER_QUERY_MYSQL_SERVERS, query_no_space, strlen(CLUSTER_QUERY_MYSQL_SERVERS))) { - tn = "mysql_servers"; + if (!strncasecmp(CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS, query_no_space, strlen(CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS))) { + tn = "cluster_mysql_servers"; } else if (!strncasecmp(CLUSTER_QUERY_MYSQL_REPLICATION_HOSTGROUPS, query_no_space, strlen(CLUSTER_QUERY_MYSQL_REPLICATION_HOSTGROUPS))) { tn = "mysql_replication_hostgroups"; } else if (!strncasecmp(CLUSTER_QUERY_MYSQL_GROUP_REPLICATION_HOSTGROUPS, query_no_space, strlen(CLUSTER_QUERY_MYSQL_GROUP_REPLICATION_HOSTGROUPS))) { @@ -3757,6 +3771,8 @@ void admin_session_handler(MySQL_Session *sess, void *_pa, PtrSize_t *pkt) { tn = "mysql_aws_aurora_hostgroups"; } else if (!strncasecmp(CLUSTER_QUERY_MYSQL_HOSTGROUP_ATTRIBUTES, query_no_space, strlen(CLUSTER_QUERY_MYSQL_HOSTGROUP_ATTRIBUTES))) { tn = "mysql_hostgroup_attributes"; + } else if (!strncasecmp(CLUSTER_QUERY_MYSQL_SERVERS_V2, query_no_space, strlen(CLUSTER_QUERY_MYSQL_SERVERS_V2))) { + tn = "mysql_servers_v2"; } if (tn != "") { GloAdmin->mysql_servers_wrlock(); @@ -3764,7 +3780,28 @@ void admin_session_handler(MySQL_Session *sess, void *_pa, PtrSize_t *pkt) { GloAdmin->mysql_servers_wrunlock(); if (resultset == nullptr) { - resultset=MyHGM->dump_table_mysql(tn); + // 'mysql_servers_v2' is a virtual table that represents the latest 'main.mysql_servers' + // records promoted by the user. This section shouldn't be reached, since the initial resulset + // for this table ('MySQL_HostGroups_Manager::incoming_mysql_servers') is generated during + // initialization, and it's only updated in subsequent user config promotions. In case we + // reach here, an empty resultset should be replied, as it would mean that no user + // config has ever been promoted to runtime, and thus, this virtual table should remain empty. + if (tn == "mysql_servers_v2") { + const string query_empty_resultset { + string { MYHGM_GEN_CLUSTER_ADMIN_RUNTIME_SERVERS } + " LIMIT 0" + }; + + char *error=NULL; + int cols=0; + int affected_rows=0; + proxy_debug(PROXY_DEBUG_MYSQL_CONNPOOL, 4, "%s\n", query); + GloAdmin->mysql_servers_wrlock(); + GloAdmin->admindb->execute_statement(query_empty_resultset.c_str(), &error, &cols, &affected_rows, &resultset); + GloAdmin->mysql_servers_wrunlock(); + } else { + resultset = MyHGM->dump_table_mysql(tn); + } + if (resultset) { sess->SQLite3_to_MySQL(resultset, error, affected_rows, &sess->client_myds->myprot); delete resultset; @@ -5901,6 +5938,7 @@ ProxySQL_Admin::ProxySQL_Admin() : variables.cluster_mysql_variables_diffs_before_sync = 3; variables.cluster_admin_variables_diffs_before_sync = 3; variables.cluster_ldap_variables_diffs_before_sync = 3; + variables.cluster_mysql_servers_sync_algorithm = 1; checksum_variables.checksum_mysql_query_rules = true; checksum_variables.checksum_mysql_servers = true; checksum_variables.checksum_mysql_users = true; @@ -8008,6 +8046,10 @@ char * ProxySQL_Admin::get_variable(char *name) { sprintf(intbuf,"%d",variables.cluster_ldap_variables_diffs_before_sync); return strdup(intbuf); } + if (!strcasecmp(name,"cluster_mysql_servers_sync_algorithm")) { + sprintf(intbuf, "%d", variables.cluster_mysql_servers_sync_algorithm); + return strdup(intbuf); + } if (!strcasecmp(name,"cluster_mysql_query_rules_save_to_disk")) { return strdup((variables.cluster_mysql_query_rules_save_to_disk ? "true" : "false")); } @@ -8477,6 +8519,22 @@ bool ProxySQL_Admin::set_variable(char *name, char *value, bool lock) { // this return false; } } + if (!strcasecmp(name,"cluster_mysql_servers_sync_algorithm")) { + int intv=atoi(value); + if (intv >= 1 && intv <= 3) { + + if (variables.cluster_mysql_servers_sync_algorithm != intv) { + proxy_info("'cluster_mysql_servers_sync_algorithm' updated. Resetting global checksums to force Cluster re-sync.\n"); + GloProxyCluster->Reset_Global_Checksums(lock); + } + + variables.cluster_mysql_servers_sync_algorithm=intv; + __sync_lock_test_and_set(&GloProxyCluster->cluster_mysql_servers_sync_algorithm, intv); + return true; + } else { + return false; + } + } if (!strcasecmp(name,"version")) { if (strcasecmp(value,(char *)PROXYSQL_VERSION)==0) { return true; @@ -11544,6 +11602,14 @@ void ProxySQL_Admin::dump_checksums_values_table() { rc=(*proxy_sqlite3_clear_bindings)(statement1); ASSERT_SQLITE_OK(rc, admindb); rc=(*proxy_sqlite3_reset)(statement1); ASSERT_SQLITE_OK(rc, admindb); + rc = (*proxy_sqlite3_bind_text)(statement1, 1, "mysql_servers_v2", -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, admindb); + rc = (*proxy_sqlite3_bind_int64)(statement1, 2, GloVars.checksums_values.mysql_servers_v2.version); ASSERT_SQLITE_OK(rc, admindb); + rc = (*proxy_sqlite3_bind_int64)(statement1, 3, GloVars.checksums_values.mysql_servers_v2.epoch); ASSERT_SQLITE_OK(rc, admindb); + rc = (*proxy_sqlite3_bind_text)(statement1, 4, GloVars.checksums_values.mysql_servers_v2.checksum, -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, admindb); + SAFE_SQLITE3_STEP2(statement1); + rc = (*proxy_sqlite3_clear_bindings)(statement1); ASSERT_SQLITE_OK(rc, admindb); + rc = (*proxy_sqlite3_reset)(statement1); ASSERT_SQLITE_OK(rc, admindb); + if (GloMyLdapAuth) { rc=(*proxy_sqlite3_bind_text)(statement1, 1, "ldap_variables", -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, admindb); rc=(*proxy_sqlite3_bind_int64)(statement1, 2, GloVars.checksums_values.ldap_variables.version); ASSERT_SQLITE_OK(rc, admindb); @@ -12345,9 +12411,8 @@ void ProxySQL_Admin::load_scheduler_to_runtime() { resultset=NULL; } -void ProxySQL_Admin::load_mysql_servers_to_runtime( - const incoming_servers_t& incoming_servers, const std::string& checksum, const time_t epoch -) { +void ProxySQL_Admin::load_mysql_servers_to_runtime(const incoming_servers_t& incoming_servers, + const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server, const mysql_servers_v2_checksum_t& peer_mysql_server_v2) { // make sure that the caller has called mysql_servers_wrlock() char *error=NULL; int cols=0; @@ -12366,11 +12431,12 @@ void ProxySQL_Admin::load_mysql_servers_to_runtime( SQLite3_result* incoming_galera_hostgroups = incoming_servers.incoming_galera_hostgroups; SQLite3_result* incoming_aurora_hostgroups = incoming_servers.incoming_aurora_hostgroups; SQLite3_result* incoming_hostgroup_attributes = incoming_servers.incoming_hostgroup_attributes; + SQLite3_result* incoming_mysql_servers_v2 = incoming_servers.incoming_mysql_servers_v2; - char *query=(char *)"SELECT hostgroup_id,hostname,port,gtid_port,status,weight,compression,max_connections,max_replication_lag,use_ssl,max_latency_ms,comment FROM main.mysql_servers ORDER BY hostgroup_id, hostname, port"; - proxy_debug(PROXY_DEBUG_ADMIN, 4, "%s\n", query); + const char *query=(char *)"SELECT hostgroup_id,hostname,port,gtid_port,status,weight,compression,max_connections,max_replication_lag,use_ssl,max_latency_ms,comment FROM main.mysql_servers ORDER BY hostgroup_id, hostname, port"; if (runtime_mysql_servers == nullptr) { - admindb->execute_statement(query, &error , &cols , &affected_rows , &resultset_servers); + proxy_debug(PROXY_DEBUG_ADMIN, 4, "%s\n", query); + admindb->execute_statement(query, &error, &cols, &affected_rows, &resultset_servers); } else { resultset_servers = runtime_mysql_servers; } @@ -12534,7 +12600,7 @@ void ProxySQL_Admin::load_mysql_servers_to_runtime( } // commit all the changes - MyHGM->commit(runtime_mysql_servers, checksum, epoch); + MyHGM->commit(runtime_mysql_servers, peer_runtime_mysql_server, incoming_mysql_servers_v2, peer_mysql_server_v2, false, true); // quering runtime table will update and return latest records, so this is not needed. // GloAdmin->save_mysql_servers_runtime_to_database(true); diff --git a/lib/ProxySQL_Cluster.cpp b/lib/ProxySQL_Cluster.cpp index 6d9d42f9f0..55be2cd3a1 100644 --- a/lib/ProxySQL_Cluster.cpp +++ b/lib/ProxySQL_Cluster.cpp @@ -57,6 +57,8 @@ void * ProxySQL_Cluster_Monitor_thread(void *args) { mysql_thread_init(); pthread_detach(pthread_self()); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Thread started for peer %s:%d\n", node->hostname, node->port); + proxy_info("Cluster: starting thread for peer %s:%d\n", node->hostname, node->port); char *query1 = (char *)"SELECT GLOBAL_CHECKSUM()"; // in future this will be used for "light check" char *query2 = (char *)"SELECT * FROM stats_mysql_global ORDER BY Variable_Name"; @@ -98,6 +100,7 @@ void * ProxySQL_Cluster_Monitor_thread(void *args) { mysql_options(conn, MARIADB_OPT_SSL_KEYLOG_CALLBACK, (void*)proxysql_keylog_write_line_callback); } //rc_conn = mysql_real_connect(conn, node->hostname, username, password, NULL, node->port, NULL, CLIENT_COMPRESS); // FIXME: add optional support for compression + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Connecting to peer %s:%d\n", node->hostname, node->port); rc_conn = mysql_real_connect(conn, node->get_host_address(), username, password, NULL, node->port, NULL, 0); // if (rc_conn) { // } @@ -116,6 +119,7 @@ void * ProxySQL_Cluster_Monitor_thread(void *args) { if (row[0]) { const char* PROXYSQL_VERSION_ = GloMyLdapAuth == nullptr ? PROXYSQL_VERSION : PROXYSQL_VERSION"-Enterprise"; if (strcmp(row[0], PROXYSQL_VERSION_)==0) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Clustering with peer %s:%d . Remote version: %s . Self version: %s\n", node->hostname, node->port, row[0], PROXYSQL_VERSION_); proxy_info("Cluster: clustering with peer %s:%d . Remote version: %s . Self version: %s\n", node->hostname, node->port, row[0], PROXYSQL_VERSION_); same_version = true; std::string q = "PROXYSQL CLUSTER_NODE_UUID "; @@ -124,6 +128,7 @@ void * ProxySQL_Cluster_Monitor_thread(void *args) { pthread_mutex_lock(&GloProxyCluster->admin_mysql_ifaces_mutex); q += GloProxyCluster->admin_mysql_ifaces; pthread_mutex_unlock(&GloProxyCluster->admin_mysql_ifaces_mutex); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Sending CLUSTER_NODE_UUID %s to peer %s:%d\n", GloVars.uuid, node->hostname, node->port); proxy_info("Cluster: sending CLUSTER_NODE_UUID %s to peer %s:%d\n", GloVars.uuid, node->hostname, node->port); rc_query = mysql_query(conn, q.c_str()); } else { @@ -133,6 +138,7 @@ void * ProxySQL_Cluster_Monitor_thread(void *args) { } mysql_free_result(result); if (same_version == false) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Remote peer %s:%d proxysql version is different. Closing connection\n", node->hostname, node->port); mysql_close(conn); conn = mysql_init(NULL); int exit_after_N_seconds = 30; // hardcoded sleep time @@ -338,8 +344,7 @@ ProxySQL_Node_Entry::ProxySQL_Node_Entry(char* _hostname, uint16_t _port, uint64 weight = _weight; if (_comment == NULL) { comment = strdup((char*)""); - } - else { + } else { comment = strdup(_comment); } @@ -421,13 +426,13 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { time_t now = time(NULL); // Fetch the cluster_*_diffs_before_sync variables to ensure consistency at local scope + unsigned int diff_av = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_admin_variables_diffs_before_sync,0); unsigned int diff_mqr = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_mysql_query_rules_diffs_before_sync,0); unsigned int diff_ms = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_mysql_servers_diffs_before_sync,0); unsigned int diff_mu = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_mysql_users_diffs_before_sync,0); unsigned int diff_ps = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_proxysql_servers_diffs_before_sync,0); unsigned int diff_mv = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_mysql_variables_diffs_before_sync,0); unsigned int diff_lv = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_ldap_variables_diffs_before_sync,0); - unsigned int diff_av = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_admin_variables_diffs_before_sync,0); pthread_mutex_lock(&GloVars.checksum_mutex); @@ -463,9 +468,12 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.admin_variables.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for admin_variables from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.admin_variables.version, checksums_values.admin_variables.epoch, + checksums_values.admin_variables.checksum, GloVars.checksums_values.admin_variables.checksum, checksums_values.admin_variables.diff_check); } if (strcmp(checksums_values.admin_variables.checksum, GloVars.checksums_values.admin_variables.checksum) == 0) { checksums_values.admin_variables.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for admin_variables from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.admin_variables.checksum); } continue; } @@ -500,9 +508,12 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.mysql_query_rules.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_query_rules from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.mysql_query_rules.version, checksums_values.mysql_query_rules.epoch, + checksums_values.mysql_query_rules.checksum, GloVars.checksums_values.mysql_query_rules.checksum, checksums_values.mysql_query_rules.diff_check); } if (strcmp(checksums_values.mysql_query_rules.checksum, GloVars.checksums_values.mysql_query_rules.checksum) == 0) { checksums_values.mysql_query_rules.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_query_rules from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.mysql_query_rules.checksum); } continue; } @@ -537,6 +548,8 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.mysql_servers.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_servers from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.mysql_servers.version, checksums_values.mysql_servers.epoch, + checksums_values.mysql_servers.checksum, GloVars.checksums_values.mysql_servers.checksum, checksums_values.mysql_servers.diff_check); } if (strcmp(checksums_values.mysql_servers.checksum, GloVars.checksums_values.mysql_servers.checksum) == 0) { // See LOGGING-NOTE at 'admin_variables' above. @@ -547,6 +560,54 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { ); } checksums_values.mysql_servers.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_servers from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.mysql_servers.checksum); + } + continue; + } + if (strcmp(row[0], "mysql_servers_v2")==0) { + ProxySQL_Checksum_Value_2& checksum = checksums_values.mysql_servers_v2; + ProxySQL_Checksum_Value& global_checksum = GloVars.checksums_values.mysql_servers_v2; + checksums_values.mysql_servers_v2.version = atoll(row[1]); + checksums_values.mysql_servers_v2.epoch = atoll(row[2]); + checksums_values.mysql_servers_v2.last_updated = now; + if (strcmp(checksums_values.mysql_servers_v2.checksum, row[3])) { + strcpy(checksums_values.mysql_servers_v2.checksum, row[3]); + checksums_values.mysql_servers_v2.last_changed = now; + checksums_values.mysql_servers_v2.diff_check = 1; + const char* no_sync_message = NULL; + + if (diff_ms) { + no_sync_message = "Not syncing yet ...\n"; + } else { + no_sync_message = "Not syncing due to 'admin-cluster_mysql_servers_diffs_before_sync=0'.\n"; + } + + proxy_info( + "Cluster: detected a new checksum for %s from peer %s:%d, version %llu, epoch %llu, checksum %s . %s", + row[0], hostname, port, checksum.version, checksum.epoch, checksum.checksum, no_sync_message + ); + + if (strcmp(checksum.checksum, global_checksum.checksum) == 0) { + proxy_info( + "Cluster: checksum for %s from peer %s:%d matches with local checksum %s , we won't sync.\n", + row[0], hostname, port, global_checksum.checksum + ); + } + } else { + checksums_values.mysql_servers_v2.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_servers_v2 from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.mysql_servers_v2.version, checksums_values.mysql_servers_v2.epoch, + checksums_values.mysql_servers_v2.checksum, GloVars.checksums_values.mysql_servers_v2.checksum, checksums_values.mysql_servers_v2.diff_check); + } + if (strcmp(checksums_values.mysql_servers_v2.checksum, GloVars.checksums_values.mysql_servers_v2.checksum) == 0) { + // See LOGGING-NOTE at 'admin_variables' above. + if (checksums_values.mysql_servers_v2.last_changed == now) { + proxy_info( + "Cluster: checksum for mysql_servers_v2 from peer %s:%d matches with local checksum %s , we won't sync.\n", + hostname, port, GloVars.checksums_values.mysql_servers_v2.checksum + ); + } + checksums_values.mysql_servers_v2.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_servers_v2 from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.mysql_servers.checksum); } continue; } @@ -581,9 +642,12 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.mysql_users.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_users from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.mysql_users.version, checksums_values.mysql_users.epoch, + checksums_values.mysql_users.checksum, GloVars.checksums_values.mysql_users.checksum, checksums_values.mysql_users.diff_check); } if (strcmp(checksums_values.mysql_users.checksum, GloVars.checksums_values.mysql_users.checksum) == 0) { checksums_values.mysql_users.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_users from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.mysql_users.checksum); } continue; } @@ -618,9 +682,12 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.mysql_variables.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_variables from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.mysql_variables.version, checksums_values.mysql_variables.epoch, + checksums_values.mysql_variables.checksum, GloVars.checksums_values.mysql_variables.checksum, checksums_values.mysql_variables.diff_check); } if (strcmp(checksums_values.mysql_variables.checksum, GloVars.checksums_values.mysql_variables.checksum) == 0) { checksums_values.mysql_variables.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for mysql_variables from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.mysql_variables.checksum); } continue; } @@ -655,9 +722,12 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.proxysql_servers.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for proxysql_servers from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.proxysql_servers.version, checksums_values.proxysql_servers.epoch, + checksums_values.proxysql_servers.checksum, GloVars.checksums_values.proxysql_servers.checksum, checksums_values.proxysql_servers.diff_check); } if (strcmp(checksums_values.proxysql_servers.checksum, GloVars.checksums_values.proxysql_servers.checksum) == 0) { checksums_values.proxysql_servers.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for proxysql_servers from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.proxysql_servers.checksum); } continue; } @@ -692,9 +762,12 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { } } else { checksums_values.ldap_variables.diff_check++; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for ldap_variables from peer %s:%d, version %llu, epoch %llu, checksum %s is different from local checksum %s. Incremented diff_check %d ...\n", hostname, port, checksums_values.ldap_variables.version, checksums_values.ldap_variables.epoch, + checksums_values.ldap_variables.checksum, GloVars.checksums_values.ldap_variables.checksum, checksums_values.ldap_variables.diff_check); } if (strcmp(checksums_values.ldap_variables.checksum, GloVars.checksums_values.ldap_variables.checksum) == 0) { checksums_values.ldap_variables.diff_check = 0; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Checksum for ldap_variables from peer %s:%d matches with local checksum %s, reset diff_check to 0.\n", hostname, port, GloVars.checksums_values.ldap_variables.checksum); } continue; } @@ -720,6 +793,13 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { if (strcmp(v->checksum, GloVars.checksums_values.mysql_servers.checksum) == 0) { v->diff_check = 0; } + if (v->diff_check) + v->diff_check++; + v = &checksums_values.mysql_servers_v2; + v->last_updated = now; + if (strcmp(v->checksum, GloVars.checksums_values.mysql_servers_v2.checksum) == 0) { + v->diff_check = 0; + } if (v->diff_check) v->diff_check++; v = &checksums_values.mysql_users; @@ -756,6 +836,38 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { // note that this is done outside the critical section // as mutex on GloVars.checksum_mutex is already released ProxySQL_Checksum_Value_2 *v = NULL; + if (diff_av) { + v = &checksums_values.admin_variables; + unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.admin_variables.version, 0); + unsigned long long own_epoch = __sync_fetch_and_add(&GloVars.checksums_values.admin_variables.epoch, 0); + char* own_checksum = __sync_fetch_and_add(&GloVars.checksums_values.admin_variables.checksum, 0); + const string expected_checksum { v->checksum }; + + if (v->version > 1) { + if ( + (own_version == 1) // we just booted + || + (v->epoch > own_epoch) // epoch is newer + ) { + if (v->diff_check >= diff_av) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); + proxy_info("Cluster: detected a peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); + GloProxyCluster->pull_global_variables_from_peer("admin", expected_checksum, v->epoch); + } + } + if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_av*10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD ADMIN VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_av * 10)); + proxy_error("Cluster: detected a peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD ADMIN VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_av*10)); + GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_admin_variables_share_epoch]->Increment(); + } + } else { + if (v->diff_check && (v->diff_check % (diff_av*10)) == 0) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD ADMIN VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_av * 10)); + proxy_warning("Cluster: detected a peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD ADMIN VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_av*10)); + GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_admin_variables_version_one]->Increment(); + } + } + } if (diff_mqr) { unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.mysql_query_rules.version,0); unsigned long long own_epoch = __sync_fetch_and_add(&GloVars.checksums_values.mysql_query_rules.epoch,0); @@ -770,27 +882,37 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { (v->epoch > own_epoch) // epoch is newer ) { if (v->diff_check >= diff_mqr) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); proxy_info("Cluster: detected a peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); GloProxyCluster->pull_mysql_query_rules_from_peer(v_exp_checksum, v->epoch); } } if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_mqr*10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_mqr * 10)); proxy_error("Cluster: detected a peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_mqr*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_mysql_query_rules_share_epoch]->Increment(); } } else { if (v->diff_check && (v->diff_check % (diff_mqr*10)) == 0) { - proxy_warning("Cluster: detected a peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL QUERY RULES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mqr*10)); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected a peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL QUERY RULES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mqr * 10)); + proxy_warning("Cluster: detected a peer %s:%d with mysql_query_rules version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL QUERY RULES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mqr*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_mysql_query_rules_version_one]->Increment(); } } } if (diff_ms) { - v = &checksums_values.mysql_servers; - unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers.version,0); - unsigned long long own_epoch = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers.epoch,0); - char* own_checksum = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers.checksum,0); - const std::string v_exp_checksum { v->checksum }; + mysql_servers_sync_algorithm mysql_server_sync_algo = (mysql_servers_sync_algorithm)__sync_fetch_and_add(&GloProxyCluster->cluster_mysql_servers_sync_algorithm, 0); + + if (mysql_server_sync_algo == mysql_servers_sync_algorithm::auto_select) { + mysql_server_sync_algo = (GloVars.global.monitor == false) ? + mysql_servers_sync_algorithm::runtime_mysql_servers_and_mysql_servers_v2 : mysql_servers_sync_algorithm::mysql_servers_v2; + } + + v = &checksums_values.mysql_servers_v2; + const unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers_v2.version, 0); + const unsigned long long own_epoch = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers_v2.epoch, 0); + const char* own_checksum = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers_v2.checksum, 0); + bool runtime_mysql_servers_already_loaded = false; if (v->version > 1) { if ( @@ -799,20 +921,69 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { (v->epoch > own_epoch) // epoch is newer ) { if (v->diff_check >= diff_ms) { - proxy_info("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); - GloProxyCluster->pull_mysql_servers_from_peer(v_exp_checksum, v->epoch); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers_v2 version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); + proxy_info("Cluster: detected a peer %s:%d with mysql_servers_v2 version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); + + ProxySQL_Checksum_Value_2* runtime_mysql_server_checksum = &checksums_values.mysql_servers; + + const bool fetch_runtime = (mysql_server_sync_algo == mysql_servers_sync_algorithm::runtime_mysql_servers_and_mysql_servers_v2); + + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetch mysql_servers_v2:'YES', mysql_servers:'%s' from peer %s:%d\n", (fetch_runtime ? "YES" : "NO"), + hostname, port); + proxy_info("Cluster: Fetch mysql_servers_v2:'YES', mysql_servers:'%s' from peer %s:%d\n", (fetch_runtime ? "YES" : "NO"), + hostname, port); + + GloProxyCluster->pull_mysql_servers_v2_from_peer({ v->checksum, static_cast(v->epoch) }, + { runtime_mysql_server_checksum->checksum, static_cast(runtime_mysql_server_checksum->epoch) }, fetch_runtime); + + runtime_mysql_servers_already_loaded = fetch_runtime; } } - if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_ms*10)) == 0)) { - proxy_error("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ms*10)); + if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_ms * 10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers_v2 version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ms * 10)); + proxy_error("Cluster: detected a peer %s:%d with mysql_servers_v2 version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ms * 10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_mysql_servers_share_epoch]->Increment(); } } else { - if (v->diff_check && (v->diff_check % (diff_ms*10)) == 0) { - proxy_warning("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ms*10)); + if (v->diff_check && (v->diff_check % (diff_ms * 10)) == 0) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ms * 10)); + proxy_warning("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ms * 10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_mysql_servers_version_one]->Increment(); } } + + if (mysql_server_sync_algo == mysql_servers_sync_algorithm::runtime_mysql_servers_and_mysql_servers_v2 && + runtime_mysql_servers_already_loaded == false) { + v = &checksums_values.mysql_servers; + unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers.version, 0); + unsigned long long own_epoch = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers.epoch, 0); + char* own_checksum = __sync_fetch_and_add(&GloVars.checksums_values.mysql_servers.checksum, 0); + + if (v->version > 1) { + if ( + (own_version == 1) // we just booted + || + (v->epoch > own_epoch) // epoch is newer + ) { + if (v->diff_check >= diff_ms) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); + proxy_info("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); + GloProxyCluster->pull_runtime_mysql_servers_from_peer({ v->checksum, static_cast(v->epoch) }); + } + } + if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_ms * 10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ms * 10)); + proxy_error("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ms * 10)); + GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_mysql_servers_share_epoch]->Increment(); + } + } else { + if (v->diff_check && (v->diff_check % (diff_ms * 10)) == 0) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ms * 10)); + proxy_warning("Cluster: detected a peer %s:%d with mysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ms * 10)); + GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_mysql_servers_version_one]->Increment(); + } + } + } } if (diff_mu) { v = &checksums_values.mysql_users; @@ -828,17 +999,20 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { (v->epoch > own_epoch) // epoch is newer ) { if (v->diff_check >= diff_mu) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); proxy_info("Cluster: detected a peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); GloProxyCluster->pull_mysql_users_from_peer(v_exp_checksum, v->epoch); } } if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_mu*10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_mu * 10)); proxy_error("Cluster: detected a peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_mu*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_mysql_users_share_epoch]->Increment(); } } else { if (v->diff_check && (v->diff_check % (diff_mu*10)) == 0) { - proxy_warning("Cluster: detected a peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL USERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mu*10)); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected a peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL USERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mu * 10)); + proxy_warning("Cluster: detected a peer %s:%d with mysql_users version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL USERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mu*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_mysql_users_version_one]->Increment(); } } @@ -857,50 +1031,24 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { (v->epoch > own_epoch) // epoch is newer ) { if (v->diff_check >= diff_mv) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); proxy_info("Cluster: detected a peer %s:%d with mysql_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); GloProxyCluster->pull_global_variables_from_peer("mysql", expected_checksum, v->epoch); } } if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_mv*10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_mv * 10)); proxy_error("Cluster: detected a peer %s:%d with mysql_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_mv*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_mysql_variables_share_epoch]->Increment(); } } else { if (v->diff_check && (v->diff_check % (diff_mv*10)) == 0) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mv * 10)); proxy_warning("Cluster: detected a peer %s:%d with mysql_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_mv*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_mysql_variables_version_one]->Increment(); } } } - if (diff_av) { - v = &checksums_values.admin_variables; - unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.admin_variables.version, 0); - unsigned long long own_epoch = __sync_fetch_and_add(&GloVars.checksums_values.admin_variables.epoch, 0); - char* own_checksum = __sync_fetch_and_add(&GloVars.checksums_values.admin_variables.checksum, 0); - const string expected_checksum { v->checksum }; - - if (v->version > 1) { - if ( - (own_version == 1) // we just booted - || - (v->epoch > own_epoch) // epoch is newer - ) { - if (v->diff_check >= diff_av) { - proxy_info("Cluster: detected a peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); - GloProxyCluster->pull_global_variables_from_peer("admin", expected_checksum, v->epoch); - } - } - if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_av*10)) == 0)) { - proxy_error("Cluster: detected a peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD ADMIN VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_av*10)); - GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_admin_variables_share_epoch]->Increment(); - } - } else { - if (v->diff_check && (v->diff_check % (diff_av*10)) == 0) { - proxy_warning("Cluster: detected a peer %s:%d with admin_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD ADMIN VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_av*10)); - GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_admin_variables_version_one]->Increment(); - } - } - } if (GloMyLdapAuth && diff_lv) { v = &checksums_values.ldap_variables; unsigned long long own_version = __sync_fetch_and_add(&GloVars.checksums_values.ldap_variables.version, 0); @@ -915,16 +1063,19 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { (v->epoch > own_epoch) // epoch is newer ) { if (v->diff_check >= diff_lv) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with ldap_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); proxy_info("Cluster: detected a peer %s:%d with ldap_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); GloProxyCluster->pull_global_variables_from_peer("ldap", expected_checksum, v->epoch); } } if ((v->epoch == own_epoch) && v->diff_check && ((v->diff_check % (diff_lv*10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with ldap_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD LDAP VARIABLES is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_lv * 10)); proxy_error("Cluster: detected a peer %s:%d with ldap_variables version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD LDAP VARIABLES is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_lv*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_ldap_variables_share_epoch]->Increment(); } } else { if (v->diff_check && (v->diff_check % (diff_lv*10)) == 0) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with ldap_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD LDAP VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_lv * 10)); proxy_warning("Cluster: detected a peer %s:%d with ldap_variables version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD LDAP VARIABLES TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_lv*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_ldap_variables_version_one]->Increment(); } @@ -950,17 +1101,20 @@ void ProxySQL_Node_Entry::set_checksums(MYSQL_RES *_r) { (v->epoch > own_epoch) // epoch is newer ) { if (v->diff_check >= diff_ps) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); proxy_info("Cluster: detected a peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. Proceeding with remote sync\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch); GloProxyCluster->pull_proxysql_servers_from_peer(v_exp_checksum, v->epoch); } } if ((v_epoch == own_epoch) && v_diff_check && ((v_diff_check % (diff_ps*10)) == 0)) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v_version, v_epoch, v_diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ps * 10)); proxy_error("Cluster: detected a peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u, checksum %s. Own version: %llu, epoch: %llu, checksum %s. Sync conflict, epoch times are EQUAL, can't determine which server holds the latest config, we won't sync. This message will be repeated every %u checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v_version, v_epoch, v_diff_check, v->checksum, own_version, own_epoch, own_checksum, (diff_ps*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_conflict_proxysql_servers_share_epoch]->Increment(); } } else { if (v->diff_check && (v->diff_check % (diff_ps*10)) == 0) { - proxy_warning("Cluster: detected a peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD PROXYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ps*10)); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD PROXYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ps * 10)); + proxy_warning("Cluster: detected a peer %s:%d with proxysql_servers version %llu, epoch %llu, diff_check %u. Own version: %llu, epoch: %llu. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every %u checks until LOAD PROXYSQL SERVERS TO RUNTIME is executed on candidate master.\n", hostname, port, v->version, v->epoch, v->diff_check, own_version, own_epoch, (diff_ps*10)); GloProxyCluster->metrics.p_counter_array[p_cluster_counter::sync_delayed_proxysql_servers_version_one]->Increment(); } } @@ -1044,6 +1198,7 @@ void ProxySQL_Cluster::pull_mysql_query_rules_from_peer(const string& expected_c unsigned char val = 1; mysql_options(conn, MYSQL_OPT_SSL_ENFORCE, &val); mysql_options(conn, MARIADB_OPT_SSL_KEYLOG_CALLBACK, (void*)proxysql_keylog_write_line_callback); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Query Rules from peer %s:%d started. Expected checksum: %s\n", hostname, port, expected_checksum.c_str()); proxy_info("Cluster: Fetching MySQL Query Rules from peer %s:%d started. Expected checksum: %s\n", hostname, port, expected_checksum.c_str()); rc_conn = mysql_real_connect(conn, ip_address ? ip_address : hostname, username, password, NULL, port, NULL, 0); if (rc_conn) { @@ -1058,6 +1213,7 @@ void ProxySQL_Cluster::pull_mysql_query_rules_from_peer(const string& expected_c rc_query = mysql_query(conn,CLUSTER_QUERY_MYSQL_QUERY_RULES_FAST_ROUTING); if ( rc_query == 0) { result2 = mysql_store_result(conn); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Query Rules from peer %s:%d completed\n", hostname, port); proxy_info("Cluster: Fetching MySQL Query Rules from peer %s:%d completed\n", hostname, port); std::unique_ptr SQLite3_query_rules_resultset { get_SQLite3_resulset(result1) }; @@ -1066,10 +1222,11 @@ void ProxySQL_Cluster::pull_mysql_query_rules_from_peer(const string& expected_c const uint64_t query_rules_hash = SQLite3_query_rules_resultset->raw_checksum() + SQLite3_query_rules_fast_routing_resultset->raw_checksum(); const string computed_checksum { get_checksum_from_hash(query_rules_hash) }; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Computed checksum for MySQL Query Rules from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); proxy_info("Cluster: Computed checksum for MySQL Query Rules from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); if (expected_checksum == computed_checksum) { - + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading to runtime MySQL Query Rules from peer %s:%d\n", hostname, port); proxy_info("Cluster: Loading to runtime MySQL Query Rules from peer %s:%d\n", hostname, port); pthread_mutex_lock(&GloAdmin->sql_query_global_mutex); //GloAdmin->admindb->execute("PRAGMA quick_check"); @@ -1168,21 +1325,27 @@ void ProxySQL_Cluster::pull_mysql_query_rules_from_peer(const string& expected_c } //GloAdmin->admindb->execute("PRAGMA integrity_check"); GloAdmin->admindb->execute("COMMIT"); + + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading MySQL Query Rules to Runtime from peer %s:%d\n", hostname, port); // We release the ownership of the memory for 'SQLite3' resultsets here since now it's no longer // our responsability to free the memory, they should be directly passed to the 'Query Processor' GloAdmin->load_mysql_query_rules_to_runtime( SQLite3_query_rules_resultset.release(), SQLite3_query_rules_fast_routing_resultset.release(), expected_checksum, epoch ); if (GloProxyCluster->cluster_mysql_query_rules_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk MySQL Query Rules from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk MySQL Query Rules from peer %s:%d\n", hostname, port); GloAdmin->flush_GENERIC__from_to("mysql_query_rules", "memory_to_disk"); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "NOT saving to disk MySQL Query Rules from peer %s:%d\n", hostname, port); proxy_info("Cluster: NOT saving to disk MySQL Query Rules from peer %s:%d\n", hostname, port); } pthread_mutex_unlock(&GloAdmin->sql_query_global_mutex); metrics.p_counter_array[p_cluster_counter::pulled_mysql_query_rules_success]->Increment(); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Query Rules from peer %s:%d failed because of mismatching checksum. Expected: %s , Computed: %s\n", + hostname, port, expected_checksum.c_str(), computed_checksum.c_str()); proxy_info( "Cluster: Fetching MySQL Query Rules from peer %s:%d failed because of mismatching checksum. Expected: %s , Computed: %s\n", hostname, port, expected_checksum.c_str(), computed_checksum.c_str() @@ -1190,10 +1353,12 @@ void ProxySQL_Cluster::pull_mysql_query_rules_from_peer(const string& expected_c metrics.p_counter_array[p_cluster_counter::pulled_mysql_query_rules_failure]->Increment(); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Query Rules from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching MySQL Query Rules from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_query_rules_failure]->Increment(); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Query Rules from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching MySQL Query Rules from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_query_rules_failure]->Increment(); } @@ -1204,6 +1369,7 @@ void ProxySQL_Cluster::pull_mysql_query_rules_from_peer(const string& expected_c mysql_free_result(result2); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Query Rules from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching MySQL Query Rules from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_query_rules_failure]->Increment(); } @@ -1318,14 +1484,17 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu unsigned char val = 1; mysql_options(conn, MYSQL_OPT_SSL_ENFORCE, &val); mysql_options(conn, MARIADB_OPT_SSL_KEYLOG_CALLBACK, (void*)proxysql_keylog_write_line_callback); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Users from peer %s:%d started. Expected checksum: %s\n", hostname, port, expected_checksum.c_str()); proxy_info("Cluster: Fetching MySQL Users from peer %s:%d started. Expected checksum: %s\n", hostname, port, expected_checksum.c_str()); rc_conn = mysql_real_connect(conn, ip_address ? ip_address : hostname, username, password, NULL, port, NULL, 0); if (rc_conn == nullptr) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Users from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching MySQL Users from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_users_failure]->Increment(); if (GloMyLdapAuth) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching LDAP Mappings from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching LDAP Mappings from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_ldap_mapping_failure]->Increment(); } @@ -1340,9 +1509,11 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu MYSQL_RES* mysql_users_result = mysql_store_result(conn); MYSQL_RES* ldap_mapping_result = nullptr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Users from peer %s:%d completed\n", hostname, port); proxy_info("Cluster: Fetching MySQL Users from peer %s:%d completed\n", hostname, port); if (GloMyLdapAuth) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching LDAP Mappings from peer %s:%d.\n", hostname, port); proxy_info("Cluster: Fetching LDAP Mappings from peer %s:%d.\n", hostname, port); rc_query = mysql_query( @@ -1351,8 +1522,10 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu if (rc_query == 0) { ldap_mapping_result = mysql_store_result(conn); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching LDAP Mappings from peer %s:%d completed\n", hostname, port); proxy_info("Cluster: Fetching LDAP Mappings from peer %s:%d completed\n", hostname, port); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching LDAP Mappings from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching LDAP Mappings from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_ldap_mapping_failure]->Increment(); } @@ -1362,6 +1535,7 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu const uint64_t users_raw_checksum = get_mysql_users_checksum(mysql_users_result, ldap_mapping_result, mysql_users_resultset); const string computed_checksum { get_checksum_from_hash(users_raw_checksum) }; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Computed checksum for MySQL Users from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); proxy_info("Cluster: Computed checksum for MySQL Users from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); if (expected_checksum == computed_checksum) { @@ -1373,22 +1547,28 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu mysql_free_result(ldap_mapping_result); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading to runtime MySQL Users from peer %s:%d\n", hostname, port); proxy_info("Cluster: Loading to runtime MySQL Users from peer %s:%d\n", hostname, port); if (GloMyLdapAuth) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading to runtime LDAP Mappings from peer %s:%d\n", hostname, port); proxy_info("Cluster: Loading to runtime LDAP Mappings from peer %s:%d\n", hostname, port); } GloAdmin->init_users(std::move(mysql_users_resultset), expected_checksum, epoch); if (GloProxyCluster->cluster_mysql_users_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk MySQL Users from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk MySQL Users from peer %s:%d\n", hostname, port); if (GloMyLdapAuth) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk LDAP Mappings from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk LDAP Mappings from peer %s:%d\n", hostname, port); } GloAdmin->flush_mysql_users__from_memory_to_disk(); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "NOT saving to disk MySQL Users from peer %s:%d\n", hostname, port); proxy_info("Cluster: NOT saving to disk MySQL Users from peer %s:%d\n", hostname, port); if (GloMyLdapAuth) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "NOT Saving to disk LDAP Mappings from peer %s:%d\n", hostname, port); proxy_info("Cluster: NOT Saving to disk LDAP Mappings from peer %s:%d\n", hostname, port); } } @@ -1406,6 +1586,8 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu mysql_free_result(ldap_mapping_result); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Users from peer %s:%d failed: Checksum changed from %s to %s\n", + hostname, port, expected_checksum.c_str(), computed_checksum.c_str()); proxy_info( "Cluster: Fetching MySQL Users from peer %s:%d failed: Checksum changed from %s to %s\n", hostname, port, expected_checksum.c_str(), computed_checksum.c_str() @@ -1417,10 +1599,12 @@ void ProxySQL_Cluster::pull_mysql_users_from_peer(const string& expected_checksu } } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Users from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching MySQL Users from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_users_failure]->Increment(); if (GloMyLdapAuth) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching LDAP Mappings from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching LDAP Mappings from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_ldap_mapping_failure]->Increment(); } @@ -1497,68 +1681,10 @@ int ProxySQL_Cluster::fetch_and_store(MYSQL* conn, const fetch_query& f_query, M return query_res; } -/** - * @brief Generates a hash for the resulset received for the query 'CLUSTER_QUERY_MYSQL_SERVERS'. - * @details Remember that this query query is intercepted by 'ProxySQL_Admin' and always answered via - * 'MySQL_HostGroups_Manager::dump_table_proxysql_servers'. - * @param resultset The resulset resulting from the mentioned query. - * @return A hash representing the contents of the resulset. - */ -uint64_t mysql_servers_raw_checksum(MYSQL_RES* resultset) { - if (resultset == nullptr) { return 0; } - - uint64_t num_rows = mysql_num_rows(resultset); - if (num_rows == 0) { return 0; } - - MYSQL_FIELD* fields = mysql_fetch_fields(resultset); - uint32_t num_fields = mysql_num_fields(resultset); - uint32_t status_idx = 0; - - for (uint32_t i = 0; i < num_fields; i++) { - if (strcmp(fields[i].name, "status") == 0) { - status_idx = i; - } - } - - SpookyHash myhash {}; - myhash.Init(19,3); - - while (MYSQL_ROW row = mysql_fetch_row(resultset)) { - for (uint32_t i = 0; i < num_fields; i++) { - if (strcmp(row[status_idx], "OFFLINE_HARD") == 0) { - continue; - } - - if (row[i]) { - if (strcmp(fields[i].name, "status") == 0) { - if (strcmp(row[i], "ONLINE") == 0 || strcmp(row[i], "SHUNNED") == 0) { - myhash.Update("0", strlen("0")); - } else { - myhash.Update("2", strlen("1")); - } - } else { - // computing 'strlen' is required see @details - myhash.Update(row[i], strlen(row[i])); - } - } else { - myhash.Update("", 0); - } - } - } - - // restore the initial resulset index - mysql_data_seek(resultset, 0); - - uint64_t res_hash = 0, hash2 = 0; - myhash.Final(&res_hash, &hash2); - - return res_hash; -} - /** * @brief Generates a hash from the received resultsets from executing the following queries in the specified * order: - * - CLUSTER_QUERY_MYSQL_SERVERS. + * - CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS. * - CLUSTER_QUERY_MYSQL_REPLICATION_HOSTGROUPS. * - CLUSTER_QUERY_MYSQL_GROUP_REPLICATION_HOSTGROUPS. * - CLUSTER_QUERY_MYSQL_GALERA. @@ -1570,18 +1696,12 @@ uint64_t mysql_servers_raw_checksum(MYSQL_RES* resultset) { * @param results The resultsets from whose to compute the checksum. Previous described order is required. * @return Zero if the received resultset were empty, the computed hash otherwise. */ -uint64_t compute_servers_tables_raw_checksum(const vector& results) { +uint64_t compute_servers_tables_raw_checksum(const vector& results, size_t size) { bool init = false; SpookyHash myhash {}; - for (size_t i = 0; i < results.size(); i++) { - uint64_t raw_hash = 0; - - if (i == 0) { - raw_hash = mysql_servers_raw_checksum(results[i]); - } else { - raw_hash = mysql_raw_checksum(results[i]); - } + for (size_t i = 0; i < size; i++) { + uint64_t raw_hash = mysql_raw_checksum(results[i]); if (raw_hash != 0) { if (init == false) { @@ -1601,7 +1721,7 @@ uint64_t compute_servers_tables_raw_checksum(const vector& results) return servers_hash; } -incoming_servers_t convert_servers_resultsets(const std::vector& results) { +incoming_servers_t convert_mysql_servers_resultsets(const std::vector& results) { if (results.size() != sizeof(incoming_servers_t) / sizeof(void*)) { return incoming_servers_t {}; } else { @@ -1611,18 +1731,32 @@ incoming_servers_t convert_servers_resultsets(const std::vector& res get_SQLite3_resulset(results[2]).release(), get_SQLite3_resulset(results[3]).release(), get_SQLite3_resulset(results[4]).release(), - get_SQLite3_resulset(results[5]).release() + get_SQLite3_resulset(results[5]).release(), + get_SQLite3_resulset(results[6]).release(), }; } } -void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, const time_t epoch) { +/** + * @brief mysql_servers records will be fetched from remote peer and saved locally. + * + * @details This method involves fetching the mysql_servers records (also referred to as runtime_mysql_servers) from a remote peer + * and comparing their checksum to the remote peer's checksum. If the checksums match, the local mysql_servers (i.e., runtime_mysql_servers) + * will be updated and saved to disk, but only if the cluster_mysql_servers_save_to_disk variable is set to true. + * + * It's important to note that the runtime_mysql_servers module is distinct from the mysql_servers_v2 module. It has + * its own independent checksum (does not have dependent modules) and represents the current runtime state of the mysql_servers. + * + * @param peer_runtime_mysql_server checksum and epoch of mysql_servers from remote peer + */ +void ProxySQL_Cluster::pull_runtime_mysql_servers_from_peer(const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server) { char * hostname = NULL; char * ip_address = NULL; uint16_t port = 0; char * peer_checksum = NULL; - pthread_mutex_lock(&GloProxyCluster->update_mysql_servers_mutex); - nodes.get_peer_to_sync_mysql_servers(&hostname, &port, &peer_checksum, &ip_address); + + pthread_mutex_lock(&GloProxyCluster->update_runtime_mysql_servers_mutex); + nodes.get_peer_to_sync_runtime_mysql_servers(&hostname, &port, &peer_checksum, &ip_address); if (hostname) { char *username = NULL; char *password = NULL; @@ -1634,6 +1768,154 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, goto __exit_pull_mysql_servers_from_peer; } GloProxyCluster->get_credentials(&username, &password); + if (strlen(username)) { // do not monitor if the username is empty + unsigned int timeout = 1; + mysql_options(conn, MYSQL_OPT_CONNECT_TIMEOUT, &timeout); + { + unsigned char val = 1; mysql_options(conn, MYSQL_OPT_SSL_ENFORCE, &val); + mysql_options(conn, MARIADB_OPT_SSL_KEYLOG_CALLBACK, (void*)proxysql_keylog_write_line_callback); + } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching 'MySQL Servers' from peer %s:%d started. Expected checksum %s\n", hostname, port, peer_checksum); + proxy_info("Cluster: Fetching 'MySQL Servers' from peer %s:%d started. Expected checksum %s\n", hostname, port, peer_checksum); + rc_conn = mysql_real_connect(conn, ip_address ? ip_address : hostname, username, password, NULL, port, NULL, 0); + if (rc_conn) { + MySQL_Monitor::update_dns_cache_from_mysql_conn(conn); + + // servers messages + std::string fetch_servers_done; + string_format("Cluster: Fetching 'MySQL Servers' from peer %s:%d completed\n", fetch_servers_done, hostname, port); + std::string fetch_servers_err; + string_format("Cluster: Fetching 'MySQL Servers' from peer %s:%d failed: \n", fetch_servers_err, hostname, port); + + // Create fetching query + fetch_query query = { + CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS, + p_cluster_counter::pulled_mysql_servers_success, + p_cluster_counter::pulled_mysql_servers_failure, + { "", fetch_servers_done, fetch_servers_err } + }; + + MYSQL_RES* result = nullptr; + + if (fetch_and_store(conn, query, &result) != 0) { + if (result) { + mysql_free_result(result); + result = nullptr; + } + } + + if (result != nullptr) { + const uint64_t servers_hash = mysql_raw_checksum(result); + const string computed_checksum{ get_checksum_from_hash(servers_hash) }; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Computed checksum for MySQL Servers from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); + proxy_info("Cluster: Computed checksum for MySQL Servers from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); + + if (computed_checksum == peer_checksum) { + GloAdmin->mysql_servers_wrlock(); + std::unique_ptr runtime_mysql_servers_resultset = get_SQLite3_resulset(result); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading runtime_mysql_servers from peer %s:%d into mysql_servers_incoming", hostname, port); + MyHGM->servers_add(runtime_mysql_servers_resultset.get()); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Updating runtime_mysql_servers from peer %s:%d", hostname, port); + MyHGM->commit(runtime_mysql_servers_resultset.release(), peer_runtime_mysql_server, nullptr, {}, true, true); + + if (GloProxyCluster->cluster_mysql_servers_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving Runtime MySQL Servers to Database\n"); + GloAdmin->save_mysql_servers_runtime_to_database(false); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk MySQL Servers v2 from peer %s:%d\n", hostname, port); + proxy_info("Cluster: Saving to disk MySQL Servers v2 from peer %s:%d\n", hostname, port); + GloAdmin->flush_GENERIC__from_to("mysql_servers", "memory_to_disk"); + } + GloAdmin->mysql_servers_wrunlock(); + + // free result + mysql_free_result(result); + + metrics.p_counter_array[p_cluster_counter::pulled_mysql_servers_success]->Increment(); + } + } + } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); + proxy_info("Cluster: Fetching MySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); + metrics.p_counter_array[p_cluster_counter::pulled_mysql_servers_failure]->Increment(); + } + } + if (username) { + free(username); + } + if (password) { + free(password); + } +__exit_pull_mysql_servers_from_peer: + if (conn) { + if (conn->net.pvio) { + mysql_close(conn); + } + } + free(hostname); + + if (peer_checksum) + free(peer_checksum); + + if (ip_address) + free(ip_address); + } + pthread_mutex_unlock(&GloProxyCluster->update_runtime_mysql_servers_mutex); +} + +/** + * @brief mysql_servers_v2 records will be fetched from remote peer. mysql_servers records will be fetched if + * fetch_runtime_mysql_servers flag is true. + * + * @details The previous implementation of the "pull_mysql_servers_from_peer" method fetched data from "mysql_servers" (equivalent to runtime mysql_servers) + * and other dependent modules like "mysql_replication_hostgroups", "mysql_group_replication_hostgroups", "mysql_galera_hostgroups", + * "mysql_aws_aurora_hostgroups", and "mysql_hostgroup_attributes". It then computed an accumulated checksum and compares it with the + * peer checksum. If they matched, the configuration was loaded and saved to disk if "cluster_mysql_servers_save_to_disk" was set to true. + * + * The new implementation, "pull_mysql_servers_v2_from_peer", instead fetches data from "mysql_servers_v2" (equivalent to admin mysql_servers) + * and the same dependent modules. It then computes an accumulated checksum and compares it with the peer checksum. If they matched, the + * configuration was loaded and saved to disk if "cluster_mysql_servers_save_to_disk" was set to true. Additionally, if the "fetch_runtime_mysql_servers" + * option is enabled (if cluster_mysql_servers_sync_algorithm value is set to 1), the "mysql_servers" table will also be fetched and its checksum will be + * computed and matched with the peer checksum. If they match, the configuration will be loaded and saved to disk if the "cluster_mysql_servers_save_to_disk" + * option is true. + * + * Apart from separately fetching the runtime mysql_servers, the primary distinction between the previous and new implementations lies in the + * fetching of different tables (mysql_servers vs mysql_servers_v2) and computing of checksum. In the previous version, + * the checksum for "mysql_servers" was computed and added to the checksums of other dependent modules. In contrast, the new version + * calculates the checksum for "mysql_servers_v2" and combines it with the checksums of other dependent modules. + * + * IMPORTANT: This function performs both the fetching of config, and conditionally the 'runtime_mysql_servers', in + * order to avoid extra transitory states and checksums that would result if this operation was performed in multiple + * steps. When required by the sync algorithm ('mysql_servers_sync_algorithm'), these two fetches and configuration + * promotion should be performed in a single 'atomic' operation. + * + * @param peer_mysql_server_v2 checksum and epoch of mysql_servers_v2 from remote peer + * @param peer_runtime_mysql_server checksum and epoch of mysql_servers from remote peer + * @param fetch_runtime_mysql_servers fetch mysql_servers records if value is true + * + * NOTE: pull_mysql_servers_v2_from_peer will always be called irrespective of cluster_mysql_servers_sync_algorithm value. + */ +void ProxySQL_Cluster::pull_mysql_servers_v2_from_peer(const mysql_servers_v2_checksum_t& peer_mysql_server_v2, + const runtime_mysql_servers_checksum_t& peer_runtime_mysql_server, bool fetch_runtime_mysql_servers) { + char* hostname = NULL; + char* ip_address = NULL; + uint16_t port = 0; + char* peer_mysql_servers_v2_checksum = NULL; + char* peer_runtime_mysql_servers_checksum = NULL; + + pthread_mutex_lock(&GloProxyCluster->update_mysql_servers_v2_mutex); + nodes.get_peer_to_sync_mysql_servers_v2(&hostname, &port, &peer_mysql_servers_v2_checksum, + &peer_runtime_mysql_servers_checksum, &ip_address); + if (hostname) { + char* username = NULL; + char* password = NULL; + // bool rc_bool = true; + MYSQL* rc_conn; + MYSQL* conn = mysql_init(NULL); + if (conn == NULL) { + proxy_error("Unable to run mysql_init()\n"); + goto __exit_pull_mysql_servers_v2_from_peer; + } + GloProxyCluster->get_credentials(&username, &password); if (strlen(username)) { // do not monitor if the username is empty unsigned int timeout = 1; // unsigned int timeout_long = 60; @@ -1644,18 +1926,19 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, unsigned char val = 1; mysql_options(conn, MYSQL_OPT_SSL_ENFORCE, &val); mysql_options(conn, MARIADB_OPT_SSL_KEYLOG_CALLBACK, (void*)proxysql_keylog_write_line_callback); } - proxy_info("Cluster: Fetching MySQL Servers from peer %s:%d started. Expected checksum %s\n", hostname, port, peer_checksum); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Servers v2 from peer %s:%d started. Expected checksum %s\n", hostname, port, peer_mysql_servers_v2_checksum); + proxy_info("Cluster: Fetching MySQL Servers v2 from peer %s:%d started. Expected checksum %s\n", hostname, port, peer_mysql_servers_v2_checksum); rc_conn = mysql_real_connect(conn, ip_address ? ip_address : hostname, username, password, NULL, port, NULL, 0); if (rc_conn) { MySQL_Monitor::update_dns_cache_from_mysql_conn(conn); - std::vector results {}; + std::vector results(7,nullptr); // servers messages std::string fetch_servers_done = ""; - string_format("Cluster: Fetching MySQL Servers from peer %s:%d completed\n", fetch_servers_done, hostname, port); + string_format("Cluster: Fetching 'MySQL Servers v2' from peer %s:%d completed\n", fetch_servers_done, hostname, port); std::string fetch_servers_err = ""; - string_format("Cluster: Fetching MySQL Servers from peer %s:%d failed: \n", fetch_servers_err, hostname, port); + string_format("Cluster: Fetching 'MySQL Servers v2' from peer %s:%d failed: \n", fetch_servers_err, hostname, port); // group_replication_hostgroups messages std::string fetch_group_replication_hostgroups = ""; @@ -1690,7 +1973,7 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, */ fetch_query queries[] = { { - CLUSTER_QUERY_MYSQL_SERVERS, + CLUSTER_QUERY_MYSQL_SERVERS_V2, p_cluster_counter::pulled_mysql_servers_success, p_cluster_counter::pulled_mysql_servers_failure, { "", fetch_servers_done, fetch_servers_err } @@ -1732,40 +2015,77 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, int it_err = fetch_and_store(conn, queries[i], &fetch_res); if (it_err == 0) { - results.push_back(fetch_res); + results[i] = fetch_res; } else { fetching_error = true; break; } } + // fetch_runtime_mysql_servers value depends on 'cluster_mysql_servers_sync_algorithm' + if (fetch_runtime_mysql_servers == true) { + // Fetching runtime mysql servers (mysql_servers) configuration from remote peer + std::string fetch_runtime_servers_done = ""; + string_format("Cluster: Fetching 'MySQL Servers' from peer %s:%d completed\n", fetch_runtime_servers_done, hostname, port); + std::string fetch_runtime_servers_err = ""; + string_format("Cluster: Fetching 'MySQL Servers' from peer %s:%d failed: \n", fetch_runtime_servers_err, hostname, port); + + // Query definition used to fetch data from a peer. + fetch_query query = { + CLUSTER_QUERY_RUNTIME_MYSQL_SERVERS, + p_cluster_counter::pulled_mysql_servers_success, + p_cluster_counter::pulled_mysql_servers_failure, + { "", fetch_runtime_servers_done, fetch_runtime_servers_err } + }; + + MYSQL_RES* fetch_res = nullptr; + if (fetch_and_store(conn, query, &fetch_res) == 0) { + results[6] = fetch_res; + } else { + fetching_error = true; + } + } + if (fetching_error == false) { - const uint64_t servers_hash = compute_servers_tables_raw_checksum(results); - const string computed_checksum { get_checksum_from_hash(servers_hash) }; - proxy_info("Cluster: Computed checksum for MySQL Servers from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); + const uint64_t servers_hash = compute_servers_tables_raw_checksum(results, 6); // ignore runtime_mysql_servers in checksum calculation + const string computed_checksum{ get_checksum_from_hash(servers_hash) }; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Computed checksum for MySQL Servers v2 from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); + proxy_info("Cluster: Computed checksum for MySQL Servers v2 from peer %s:%d : %s\n", hostname, port, computed_checksum.c_str()); + + bool runtime_checksum_matches = true; + + if (results[6]) { + const uint64_t runtime_mysql_server_hash = mysql_raw_checksum(results[6]); + const std::string runtime_mysql_server_computed_checksum = get_checksum_from_hash(runtime_mysql_server_hash); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Computed checksum for MySQL Servers from peer %s:%d : %s\n", hostname, port, runtime_mysql_server_computed_checksum.c_str()); + proxy_info("Cluster: Computed checksum for MySQL Servers from peer %s:%d : %s\n", hostname, port, runtime_mysql_server_computed_checksum.c_str()); + runtime_checksum_matches = (runtime_mysql_server_computed_checksum == peer_runtime_mysql_servers_checksum); + } - if (computed_checksum == peer_checksum) { + if (computed_checksum == peer_mysql_servers_v2_checksum && runtime_checksum_matches == true) { // No need to perform the conversion if checksums don't match - const incoming_servers_t incoming_servers { convert_servers_resultsets(results) }; + const incoming_servers_t incoming_servers{ convert_mysql_servers_resultsets(results) }; // we are OK to sync! + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching checksum for 'MySQL Servers' from peer %s:%d successful. Checksum: %s\n", hostname, port, computed_checksum.c_str()); proxy_info("Cluster: Fetching checksum for 'MySQL Servers' from peer %s:%d successful. Checksum: %s\n", hostname, port, computed_checksum.c_str()); // sync mysql_servers + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Writing mysql_servers table\n"); proxy_info("Cluster: Writing mysql_servers table\n"); GloAdmin->mysql_servers_wrlock(); GloAdmin->admindb->execute("DELETE FROM mysql_servers"); MYSQL_ROW row; - char *q=(char *)"INSERT INTO mysql_servers (hostgroup_id, hostname, port, gtid_port, status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment) VALUES (%s, \"%s\", %s, %s, \"%s\", %s, %s, %s, %s, %s, %s, '%s')"; + char* q = (char*)"INSERT INTO mysql_servers (hostgroup_id, hostname, port, gtid_port, status, weight, compression, max_connections, max_replication_lag, use_ssl, max_latency_ms, comment) VALUES (%s, \"%s\", %s, %s, \"%s\", %s, %s, %s, %s, %s, %s, '%s')"; while ((row = mysql_fetch_row(results[0]))) { int i; - int l=0; - for (i=0; i<11; i++) { - l+=strlen(row[i]); + int l = 0; + for (i = 0; i < 11; i++) { + l += strlen(row[i]); } - char *o=escape_string_single_quotes(row[11],false); - char *query = (char *)malloc(strlen(q)+i+strlen(o)+64); + char* o = escape_string_single_quotes(row[11], false); + char* query = (char*)malloc(strlen(q) + i + strlen(o) + 64); - sprintf(query,q,row[0],row[1],row[2],row[3], ( strcmp(row[4],"SHUNNED")==0 ? "ONLINE" : row[4] ), row[5], row[6],row[7],row[8],row[9],row[10],o); - if (o!=row[11]) { // there was a copy + sprintf(query, q, row[0], row[1], row[2], row[3], (strcmp(row[4], "SHUNNED") == 0 ? "ONLINE" : row[4]), row[5], row[6], row[7], row[8], row[9], row[10], o); + if (o != row[11]) { // there was a copy free(o); } GloAdmin->admindb->execute(query); @@ -1773,19 +2093,20 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, } // sync mysql_replication_hostgroups + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Writing mysql_replication_hostgroups table\n"); proxy_info("Cluster: Writing mysql_replication_hostgroups table\n"); GloAdmin->admindb->execute("DELETE FROM mysql_replication_hostgroups"); - q=(char *)"INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type, comment) VALUES (%s, %s, '%s', '%s')"; + q = (char*)"INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type, comment) VALUES (%s, %s, '%s', '%s')"; while ((row = mysql_fetch_row(results[1]))) { int i; - int l=0; - for (i=0; i<3; i++) { - l+=strlen(row[i]); + int l = 0; + for (i = 0; i < 3; i++) { + l += strlen(row[i]); } - char *o=escape_string_single_quotes(row[3],false); - char *query = (char *)malloc(strlen(q)+i+strlen(o)+64); - sprintf(query,q,row[0],row[1],row[2],o); - if (o!=row[3]) { // there was a copy + char* o = escape_string_single_quotes(row[3], false); + char* query = (char*)malloc(strlen(q) + i + strlen(o) + 64); + sprintf(query, q, row[0], row[1], row[2], o); + if (o != row[3]) { // there was a copy free(o); } GloAdmin->admindb->execute(query); @@ -1793,15 +2114,16 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, } // sync mysql_group_replication_hostgroups + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Writing mysql_group_replication_hostgroups table\n"); proxy_info("Cluster: Writing mysql_group_replication_hostgroups table\n"); GloAdmin->admindb->execute("DELETE FROM mysql_group_replication_hostgroups"); - q=(char*)"INSERT INTO mysql_group_replication_hostgroups ( " + q = (char*)"INSERT INTO mysql_group_replication_hostgroups ( " "writer_hostgroup, backup_writer_hostgroup, reader_hostgroup, offline_hostgroup, active, " "max_writers, writer_is_also_reader, max_transactions_behind, comment) "; - char *error = NULL; + char* error = NULL; int cols = 0; int affected_rows = 0; - SQLite3_result *resultset = NULL; + SQLite3_result* resultset = NULL; while ((row = mysql_fetch_row(results[2]))) { int i; int l = 0; @@ -1815,7 +2137,7 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, if (row[8] != nullptr) { fqs += "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, '%s')"; o = escape_string_single_quotes(row[8], false); - query = (char *)malloc(strlen(fqs.c_str()) + i + strlen(o) + 64); + query = (char*)malloc(strlen(fqs.c_str()) + i + strlen(o) + 64); sprintf(query, fqs.c_str(), row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], o); // free in case of 'o' being a copy if (o != row[8]) { @@ -1825,22 +2147,24 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, // In case of comment being null, placeholder must not have '' fqs += "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"; o = const_cast("NULL"); - query = (char *)malloc(strlen(fqs.c_str()) + strlen("NULL") + i + 64); + query = (char*)malloc(strlen(fqs.c_str()) + strlen("NULL") + i + 64); sprintf(query, fqs.c_str(), row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], o); } GloAdmin->admindb->execute(query); free(query); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Dumping fetched 'mysql_group_replication_hostgroups'\n"); proxy_info("Dumping fetched 'mysql_group_replication_hostgroups'\n"); - GloAdmin->admindb->execute_statement((char *)"SELECT * FROM mysql_group_replication_hostgroups", &error , &cols , &affected_rows , &resultset); + GloAdmin->admindb->execute_statement((char*)"SELECT * FROM mysql_group_replication_hostgroups", &error, &cols, &affected_rows, &resultset); resultset->dump_to_stderr(); delete resultset; // sync mysql_galera_hostgroups + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Writing mysql_galera_hostgroups table\n"); proxy_info("Cluster: Writing mysql_galera_hostgroups table\n"); GloAdmin->admindb->execute("DELETE FROM mysql_galera_hostgroups"); - q=(char *)"INSERT INTO mysql_galera_hostgroups ( " + q = (char*)"INSERT INTO mysql_galera_hostgroups ( " "writer_hostgroup, backup_writer_hostgroup, reader_hostgroup, offline_hostgroup, active, " "max_writers, writer_is_also_reader, max_transactions_behind, comment) "; while ((row = mysql_fetch_row(results[3]))) { @@ -1856,7 +2180,7 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, if (row[8] != nullptr) { fqs += "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, '%s')"; o = escape_string_single_quotes(row[8], false); - query = (char *)malloc(strlen(fqs.c_str()) + i + strlen(o) + 64); + query = (char*)malloc(strlen(fqs.c_str()) + i + strlen(o) + 64); sprintf(query, fqs.c_str(), row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], o); // free in case of 'o' being a copy if (o != row[8]) { @@ -1866,22 +2190,24 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, // In case of comment being null, placeholder must not have '' fqs += "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"; o = const_cast("NULL"); - query = (char *)malloc(strlen(fqs.c_str()) + i + strlen("NULL") + 64); + query = (char*)malloc(strlen(fqs.c_str()) + i + strlen("NULL") + 64); sprintf(query, fqs.c_str(), row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], o); } GloAdmin->admindb->execute(query); free(query); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Dumping fetched 'mysql_galera_hostgroups'\n"); proxy_info("Dumping fetched 'mysql_galera_hostgroups'\n"); - GloAdmin->admindb->execute_statement((char *)"SELECT * FROM mysql_galera_hostgroups", &error , &cols , &affected_rows , &resultset); + GloAdmin->admindb->execute_statement((char*)"SELECT * FROM mysql_galera_hostgroups", &error, &cols, &affected_rows, &resultset); resultset->dump_to_stderr(); delete resultset; // sync mysql_aws_aurora_hostgroups + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Writing mysql_aws_aurora_hostgroups table\n"); proxy_info("Cluster: Writing mysql_aws_aurora_hostgroups table\n"); GloAdmin->admindb->execute("DELETE FROM mysql_aws_aurora_hostgroups"); - q=(char *)"INSERT INTO mysql_aws_aurora_hostgroups ( " + q = (char*)"INSERT INTO mysql_aws_aurora_hostgroups ( " "writer_hostgroup, reader_hostgroup, active, aurora_port, domain_name, max_lag_ms, check_interval_ms, " "check_timeout_ms, writer_is_also_reader, new_reader_weight, add_lag_ms, min_lag_ms, lag_num_checks, comment) "; while ((row = mysql_fetch_row(results[4]))) { @@ -1897,7 +2223,7 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, if (row[13] != nullptr) { fqs += "VALUES (%s, %s, %s, %s, '%s', %s, %s, %s, %s, %s, %s, %s, %s, '%s')"; o = escape_string_single_quotes(row[13], false); - query = (char *)malloc(strlen(fqs.c_str()) + i + strlen(o) + 64); + query = (char*)malloc(strlen(fqs.c_str()) + i + strlen(o) + 64); sprintf(query, fqs.c_str(), row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], row[8], row[9], row[10], row[11], row[12], o); // free in case of 'o' being a copy if (o != row[13]) { @@ -1907,67 +2233,80 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, // In case of comment being null, placeholder must not have '' fqs += "VALUES (%s, %s, %s, %s, '%s', %s, %s, %s, %s, %s, %s, %s, %s, %s)"; o = const_cast("NULL"); - query = (char *)malloc(strlen(fqs.c_str()) + i + strlen("NULL") + 64); + query = (char*)malloc(strlen(fqs.c_str()) + i + strlen("NULL") + 64); sprintf(query, fqs.c_str(), row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], row[8], row[9], row[10], row[11], row[12], o); } GloAdmin->admindb->execute(query); free(query); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Dumping fetched 'mysql_aws_aurora_hostgroups'\n"); proxy_info("Dumping fetched 'mysql_aws_aurora_hostgroups'\n"); - GloAdmin->admindb->execute_statement((char *)"SELECT * FROM mysql_aws_aurora_hostgroups", &error , &cols , &affected_rows , &resultset); + GloAdmin->admindb->execute_statement((char*)"SELECT * FROM mysql_aws_aurora_hostgroups", &error, &cols, &affected_rows, &resultset); resultset->dump_to_stderr(); delete resultset; // sync mysql_hostgroup_attributes + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Writing mysql_hostgroup_attributes table\n"); proxy_info("Cluster: Writing mysql_hostgroup_attributes table\n"); GloAdmin->admindb->execute("DELETE FROM mysql_hostgroup_attributes"); { - const char * q=(const char *)"INSERT INTO mysql_hostgroup_attributes ( " + const char* q = (const char*)"INSERT INTO mysql_hostgroup_attributes ( " "hostgroup_id, max_num_online_servers, autocommit, free_connections_pct, " "init_connect, multiplex, connection_warming, throttle_connections_per_sec, " "ignore_session_variables, comment) VALUES " "(?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)"; - sqlite3_stmt *statement1 = NULL; + sqlite3_stmt* statement1 = NULL; int rc = GloAdmin->admindb->prepare_v2(q, &statement1); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); while ((row = mysql_fetch_row(results[5]))) { - rc=(*proxy_sqlite3_bind_int64)(statement1, 1, atol(row[0])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // hostgroup_id - rc=(*proxy_sqlite3_bind_int64)(statement1, 2, atol(row[1])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // max_num_online_servers - rc=(*proxy_sqlite3_bind_int64)(statement1, 3, atol(row[2])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // autocommit - rc=(*proxy_sqlite3_bind_int64)(statement1, 4, atol(row[3])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // free_connections_pct - rc=(*proxy_sqlite3_bind_text)(statement1, 5, row[4], -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // variable_name - rc=(*proxy_sqlite3_bind_int64)(statement1, 6, atol(row[5])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // multiplex - rc=(*proxy_sqlite3_bind_int64)(statement1, 7, atol(row[6])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // connection_warming - rc=(*proxy_sqlite3_bind_int64)(statement1, 8, atol(row[7])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // throttle_connections_per_sec - rc=(*proxy_sqlite3_bind_text)(statement1, 9, row[8], -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // ignore_session_variables - rc=(*proxy_sqlite3_bind_text)(statement1, 10, row[9], -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // comment + rc = (*proxy_sqlite3_bind_int64)(statement1, 1, atol(row[0])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // hostgroup_id + rc = (*proxy_sqlite3_bind_int64)(statement1, 2, atol(row[1])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // max_num_online_servers + rc = (*proxy_sqlite3_bind_int64)(statement1, 3, atol(row[2])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // autocommit + rc = (*proxy_sqlite3_bind_int64)(statement1, 4, atol(row[3])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // free_connections_pct + rc = (*proxy_sqlite3_bind_text)(statement1, 5, row[4], -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // variable_name + rc = (*proxy_sqlite3_bind_int64)(statement1, 6, atol(row[5])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // multiplex + rc = (*proxy_sqlite3_bind_int64)(statement1, 7, atol(row[6])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // connection_warming + rc = (*proxy_sqlite3_bind_int64)(statement1, 8, atol(row[7])); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // throttle_connections_per_sec + rc = (*proxy_sqlite3_bind_text)(statement1, 9, row[8], -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // ignore_session_variables + rc = (*proxy_sqlite3_bind_text)(statement1, 10, row[9], -1, SQLITE_TRANSIENT); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); // comment SAFE_SQLITE3_STEP2(statement1); - rc=(*proxy_sqlite3_clear_bindings)(statement1); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); - rc=(*proxy_sqlite3_reset)(statement1); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); + rc = (*proxy_sqlite3_clear_bindings)(statement1); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); + rc = (*proxy_sqlite3_reset)(statement1); ASSERT_SQLITE_OK(rc, GloAdmin->admindb); } (*proxy_sqlite3_finalize)(statement1); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Dumping fetched 'mysql_hostgroup_attributes'\n"); proxy_info("Dumping fetched 'mysql_hostgroup_attributes'\n"); - GloAdmin->admindb->execute_statement((char *)"SELECT * FROM mysql_hostgroup_attributes", &error , &cols , &affected_rows , &resultset); + GloAdmin->admindb->execute_statement((char*)"SELECT * FROM mysql_hostgroup_attributes", &error, &cols, &affected_rows, &resultset); resultset->dump_to_stderr(); delete resultset; - proxy_info("Cluster: Loading to runtime MySQL Servers from peer %s:%d\n", hostname, port); - GloAdmin->load_mysql_servers_to_runtime(incoming_servers, checksum, epoch); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading to runtime MySQL Servers v2 from peer %s:%d\n", hostname, port); + proxy_info("Cluster: Loading to runtime MySQL Servers v2 from peer %s:%d\n", hostname, port); + GloAdmin->load_mysql_servers_to_runtime(incoming_servers, peer_runtime_mysql_server, peer_mysql_server_v2); + if (GloProxyCluster->cluster_mysql_servers_save_to_disk == true) { - proxy_info("Cluster: Saving to disk MySQL Servers from peer %s:%d\n", hostname, port); + if (fetch_runtime_mysql_servers == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving Runtime MySQL Servers to Database\n"); + GloAdmin->save_mysql_servers_runtime_to_database(false); + } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk MySQL Servers v2 from peer %s:%d\n", hostname, port); + proxy_info("Cluster: Saving to disk MySQL Servers v2 from peer %s:%d\n", hostname, port); GloAdmin->flush_GENERIC__from_to("mysql_servers", "memory_to_disk"); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Not saving to disk MySQL Servers from peer %s:%d failed.\n", hostname, port); proxy_info("Cluster: Not saving to disk MySQL Servers from peer %s:%d failed.\n", hostname, port); } GloAdmin->mysql_servers_wrunlock(); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Servers v2 from peer %s:%d failed: Checksum changed from %s to %s\n", + hostname, port, peer_mysql_servers_v2_checksum, computed_checksum.c_str()); proxy_info( - "Cluster: Fetching MySQL Servers from peer %s:%d failed: Checksum changed from %s to %s\n", - hostname, port, peer_checksum, computed_checksum.c_str() + "Cluster: Fetching MySQL Servers v2 from peer %s:%d failed: Checksum changed from %s to %s\n", + hostname, port, peer_mysql_servers_v2_checksum, computed_checksum.c_str() ); metrics.p_counter_array[p_cluster_counter::pulled_mysql_variables_failure]->Increment(); } @@ -1980,6 +2319,7 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, metrics.p_counter_array[p_cluster_counter::pulled_mysql_servers_success]->Increment(); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching MySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching MySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_mysql_servers_failure]->Increment(); } @@ -1990,7 +2330,7 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, if (password) { free(password); } -__exit_pull_mysql_servers_from_peer: + __exit_pull_mysql_servers_v2_from_peer: if (conn) { if (conn->net.pvio) { mysql_close(conn); @@ -2000,8 +2340,14 @@ void ProxySQL_Cluster::pull_mysql_servers_from_peer(const std::string& checksum, if (ip_address) free(ip_address); + + if (peer_mysql_servers_v2_checksum) + free (peer_mysql_servers_v2_checksum); + + if (peer_runtime_mysql_servers_checksum) + free(peer_runtime_mysql_servers_checksum); } - pthread_mutex_unlock(&GloProxyCluster->update_mysql_servers_mutex); + pthread_mutex_unlock(&GloProxyCluster->update_mysql_servers_v2_mutex); } void ProxySQL_Cluster::pull_global_variables_from_peer(const string& var_type, const string& expected_checksum, const time_t epoch) { @@ -2088,10 +2434,12 @@ void ProxySQL_Cluster::pull_global_variables_from_peer(const string& var_type, c if (rc_query == 0) { MYSQL_RES *result = mysql_store_result(conn); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching %s Variables from peer %s:%d completed\n", vars_type_str, hostname, port); proxy_info("Cluster: Fetching %s Variables from peer %s:%d completed\n", vars_type_str, hostname, port); uint64_t glovars_hash = mysql_raw_checksum(result); string computed_checksum { get_checksum_from_hash(glovars_hash) }; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Computed checksum for %s Variables from peer %s:%d : %s\n", vars_type_str, hostname, port, computed_checksum.c_str()); proxy_info("Cluster: Computed checksum for %s Variables from peer %s:%d : %s\n", vars_type_str, hostname, port, computed_checksum.c_str()); if (expected_checksum == computed_checksum) { @@ -2127,12 +2475,14 @@ void ProxySQL_Cluster::pull_global_variables_from_peer(const string& var_type, c } mysql_free_result(result); + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading to runtime %s Variables from peer %s:%d\n", vars_type_str, hostname, port); proxy_info("Cluster: Loading to runtime %s Variables from peer %s:%d\n", vars_type_str, hostname, port); if (var_type == "mysql") { GloAdmin->load_mysql_variables_to_runtime(expected_checksum, epoch); if (GloProxyCluster->cluster_mysql_variables_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk MySQL Variables from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk MySQL Variables from peer %s:%d\n", hostname, port); GloAdmin->flush_mysql_variables__from_memory_to_disk(); } @@ -2140,13 +2490,16 @@ void ProxySQL_Cluster::pull_global_variables_from_peer(const string& var_type, c GloAdmin->load_admin_variables_to_runtime(expected_checksum, epoch, false); if (GloProxyCluster->cluster_admin_variables_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk Admin Variables from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk Admin Variables from peer %s:%d\n", hostname, port); GloAdmin->flush_admin_variables__from_memory_to_disk(); } + } else if (var_type == "ldap") { GloAdmin->load_ldap_variables_to_runtime(expected_checksum, epoch); if (GloProxyCluster->cluster_ldap_variables_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk LDAP Variables from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk LDAP Variables from peer %s:%d\n", hostname, port); GloAdmin->flush_ldap_variables__from_memory_to_disk(); } @@ -2155,8 +2508,9 @@ void ProxySQL_Cluster::pull_global_variables_from_peer(const string& var_type, c assert(0); } metrics.p_counter_array[success_metric]->Increment(); - } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching %s Variables from peer %s:%d failed: Checksum changed from %s to %s\n", + vars_type_str, hostname, port, expected_checksum.c_str(), computed_checksum.c_str()); proxy_info( "Cluster: Fetching %s Variables from peer %s:%d failed: Checksum changed from %s to %s\n", vars_type_str, hostname, port, expected_checksum.c_str(), computed_checksum.c_str() @@ -2164,10 +2518,12 @@ void ProxySQL_Cluster::pull_global_variables_from_peer(const string& var_type, c metrics.p_counter_array[p_cluster_counter::pulled_mysql_variables_failure]->Increment(); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching %s Variables from peer %s:%d failed: %s\n", vars_type_str, hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching %s Variables from peer %s:%d failed: %s\n", vars_type_str, hostname, port, mysql_error(conn)); metrics.p_counter_array[failure_metric]->Increment(); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching %s Variables from peer %s:%d failed: %s\n", vars_type_str, hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching %s Variables from peer %s:%d failed: %s\n", vars_type_str, hostname, port, mysql_error(conn)); metrics.p_counter_array[failure_metric]->Increment(); } @@ -2220,6 +2576,8 @@ void ProxySQL_Cluster::pull_proxysql_servers_from_peer(const std::string& expect unsigned char val = 1; mysql_options(conn, MYSQL_OPT_SSL_ENFORCE, &val); mysql_options(conn, MARIADB_OPT_SSL_KEYLOG_CALLBACK, (void*)proxysql_keylog_write_line_callback); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching ProxySQL Servers from peer %s:%d started. Expected checksum: %s\n", + hostname, port, expected_checksum.c_str()); proxy_info( "Cluster: Fetching ProxySQL Servers from peer %s:%d started. Expected checksum: %s\n", hostname, port, expected_checksum.c_str() @@ -2233,6 +2591,7 @@ void ProxySQL_Cluster::pull_proxysql_servers_from_peer(const std::string& expect MYSQL_RES* result = mysql_store_result(conn); uint64_t proxy_servers_hash = mysql_raw_checksum(result); const string computed_cks { get_checksum_from_hash(proxy_servers_hash) }; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching ProxySQL Servers from peer %s:%d completed. Computed checksum: %s\n", hostname, port, computed_cks.c_str()); proxy_info("Cluster: Fetching ProxySQL Servers from peer %s:%d completed. Computed checksum: %s\n", hostname, port, computed_cks.c_str()); if (computed_cks == expected_checksum) { @@ -2255,6 +2614,7 @@ void ProxySQL_Cluster::pull_proxysql_servers_from_peer(const std::string& expect free(query); } + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Dumping fetched 'proxysql_servers'\n"); proxy_info("Dumping fetched 'proxysql_servers'\n"); char *error = NULL; int cols = 0; @@ -2264,16 +2624,21 @@ void ProxySQL_Cluster::pull_proxysql_servers_from_peer(const std::string& expect resultset->dump_to_stderr(); delete resultset; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Loading to runtime ProxySQL Servers from peer %s:%d\n", hostname, port); proxy_info("Cluster: Loading to runtime ProxySQL Servers from peer %s:%d\n", hostname, port); GloAdmin->load_proxysql_servers_to_runtime(false, expected_checksum, epoch); if (GloProxyCluster->cluster_proxysql_servers_save_to_disk == true) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Saving to disk ProxySQL Servers from peer %s:%d\n", hostname, port); proxy_info("Cluster: Saving to disk ProxySQL Servers from peer %s:%d\n", hostname, port); GloAdmin->flush_GENERIC__from_to("proxysql_servers","memory_to_disk"); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "NOT saving to disk ProxySQL Servers from peer %s:%d\n", hostname, port); proxy_info("Cluster: NOT saving to disk ProxySQL Servers from peer %s:%d\n", hostname, port); } metrics.p_counter_array[p_cluster_counter::pulled_proxysql_servers_success]->Increment(); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching ProxySQL Servers from peer %s:%d failed: Checksum changed from %s to %s\n", + hostname, port, expected_checksum.c_str(), computed_cks.c_str()); proxy_info( "Cluster: Fetching ProxySQL Servers from peer %s:%d failed: Checksum changed from %s to %s\n", hostname, port, expected_checksum.c_str(), computed_cks.c_str() @@ -2282,10 +2647,12 @@ void ProxySQL_Cluster::pull_proxysql_servers_from_peer(const std::string& expect } mysql_free_result(result); } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching ProxySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching ProxySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_proxysql_servers_failure]->Increment(); } } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Fetching ProxySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); proxy_info("Cluster: Fetching ProxySQL Servers from peer %s:%d failed: %s\n", hostname, port, mysql_error(conn)); metrics.p_counter_array[p_cluster_counter::pulled_proxysql_servers_failure]->Increment(); } @@ -2544,6 +2911,9 @@ void ProxySQL_Cluster_Nodes::load_servers_list(SQLite3_result *resultset, bool _ node = new ProxySQL_Node_Entry(h_, p_, w_ , c_); node->set_active(true); umap_proxy_nodes.insert(std::make_pair(hash_, node)); + + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Added new peer %s:%d\n", h_, p_); + ProxySQL_Node_Address * a = new ProxySQL_Node_Address(h_, p_, node->get_ipaddress()); pthread_attr_t attr; pthread_attr_init(&attr); @@ -2561,6 +2931,8 @@ void ProxySQL_Cluster_Nodes::load_servers_list(SQLite3_result *resultset, bool _ node->set_active(true); node->set_weight(w_); node->set_comment(c_); + + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Peer %s:%d already exists. Updating it\n", h_, p_); } } remove_inactives(); @@ -2596,8 +2968,10 @@ bool ProxySQL_Cluster_Nodes::Update_Global_Checksum(char * _h, uint16_t _p, MYSQ while ((row = mysql_fetch_row(_r))) { unsigned long long v = atoll(row[0]); if (v == node->global_checksum) { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Global checksum 0x%llX for peer %s:%d matches\n", v, node->get_hostname(), node->get_port()); ret = false; } else { + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Global checksum for peer %s:%d is different from fetched one. Local checksum:[0x%lX] Fetched checksum:[0x%lX]\n", node->get_hostname(), node->get_port(), node->global_checksum, v); node->global_checksum = v; } } @@ -2697,11 +3071,12 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_mysql_query_rules(char **host, uin *host = hostname; *port = p; *ip_address = ip_addr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_query_rules version %llu, epoch %llu\n", hostname, p, version, epoch); proxy_info("Cluster: detected peer %s:%d with mysql_query_rules version %llu, epoch %llu\n", hostname, p, version, epoch); } } -void ProxySQL_Cluster_Nodes::get_peer_to_sync_mysql_servers(char **host, uint16_t *port, char **peer_checksum, char** ip_address) { +void ProxySQL_Cluster_Nodes::get_peer_to_sync_runtime_mysql_servers(char **host, uint16_t *port, char **peer_checksum, char** ip_address) { unsigned long long version = 0; unsigned long long epoch = 0; unsigned long long max_epoch = 0; @@ -2763,8 +3138,88 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_mysql_servers(char **host, uint16_ *host = hostname; *port = p; *ip_address = ip_addr; - proxy_info("Cluster: detected peer %s:%d with mysql_servers version %llu, epoch %llu\n", hostname, p, version, epoch); *peer_checksum = pc; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers version %llu, epoch %llu, checksum %s\n", hostname, p, version, epoch, pc); + proxy_info("Cluster: detected peer %s:%d with mysql_servers version %llu, epoch %llu\n", hostname, p, version, epoch); + } +} + +void ProxySQL_Cluster_Nodes::get_peer_to_sync_mysql_servers_v2(char** host, uint16_t* port, + char** peer_mysql_servers_v2_checksum, char** peer_runtime_mysql_servers_checksum, char** ip_address) { + unsigned long long version = 0; + unsigned long long epoch = 0; + unsigned long long max_epoch = 0; + char* hostname = NULL; + char* ip_addr = NULL; + uint16_t p = 0; + char* mysql_servers_v2_checksum = NULL; + char* runtime_mysql_servers_checksum = NULL; + //pthread_mutex_lock(&mutex); + //unsigned long long curtime = monotonic_time(); + unsigned int diff_ms = (unsigned int)__sync_fetch_and_add(&GloProxyCluster->cluster_mysql_servers_diffs_before_sync, 0); + for (std::unordered_map::iterator it = umap_proxy_nodes.begin(); it != umap_proxy_nodes.end(); ) { + ProxySQL_Node_Entry* node = it->second; + ProxySQL_Checksum_Value_2* v = &node->checksums_values.mysql_servers_v2; + if (v->version > 1) { + if (v->epoch > epoch) { + max_epoch = v->epoch; + if (v->diff_check >= diff_ms) { + epoch = v->epoch; + version = v->version; + if (mysql_servers_v2_checksum) { + free(mysql_servers_v2_checksum); + } + if (runtime_mysql_servers_checksum) { + free(runtime_mysql_servers_checksum); + } + if (hostname) { + free(hostname); + } + if (ip_addr) { + free(ip_addr); + } + mysql_servers_v2_checksum = strdup(v->checksum); + runtime_mysql_servers_checksum = strdup(node->checksums_values.mysql_servers.checksum); + hostname = strdup(node->get_hostname()); + const char* ip = node->get_ipaddress(); + if (ip) + ip_addr = strdup(ip); + p = node->get_port(); + } + } + } + it++; + } + // pthread_mutex_unlock(&mutex); + if (epoch) { + if (max_epoch > epoch) { + proxy_warning("Cluster: detected a peer with mysql_servers_v2 epoch %llu , but not enough diff_check. We won't sync from epoch %llu: temporarily skipping sync\n", max_epoch, epoch); + if (hostname) { + free(hostname); + hostname = NULL; + } + if (mysql_servers_v2_checksum) { + free(mysql_servers_v2_checksum); + mysql_servers_v2_checksum = NULL; + } + if (runtime_mysql_servers_checksum) { + free(runtime_mysql_servers_checksum); + runtime_mysql_servers_checksum = NULL; + } + if (ip_addr) { + free(ip_addr); + ip_addr = NULL; + } + } + } + if (hostname) { + *host = hostname; + *port = p; + *ip_address = ip_addr; + *peer_mysql_servers_v2_checksum = mysql_servers_v2_checksum; + *peer_runtime_mysql_servers_checksum = runtime_mysql_servers_checksum; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_servers_v2 version %llu, epoch %llu, mysql_servers_v2 checksum %s, runtime_mysql_servers %s\n", hostname, p, version, epoch, mysql_servers_v2_checksum, runtime_mysql_servers_checksum); + proxy_info("Cluster: detected peer %s:%d with mysql_servers_v2 version %llu, epoch %llu\n", hostname, p, version, epoch); } } @@ -2821,6 +3276,7 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_mysql_users(char **host, uint16_t *host = hostname; *port = p; *ip_address = ip_addr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_users version %llu, epoch %llu\n", hostname, p, version, epoch); proxy_info("Cluster: detected peer %s:%d with mysql_users version %llu, epoch %llu\n", hostname, p, version, epoch); } } @@ -2875,6 +3331,7 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_mysql_variables(char **host, uint1 *host = hostname; *port = p; *ip_address = ip_addr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with mysql_variables version %llu, epoch %llu\n", hostname, p, version, epoch); proxy_info("Cluster: detected peer %s:%d with mysql_variables version %llu, epoch %llu\n", hostname, p, version, epoch); } } @@ -2930,6 +3387,7 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_admin_variables(char **host, uint1 *host = hostname; *port = p; *ip_address = ip_addr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with admin_variables version %llu, epoch %llu\n", hostname, p, version, epoch); proxy_info("Cluster: detected peer %s:%d with admin_variables version %llu, epoch %llu\n", hostname, p, version, epoch); } } @@ -2984,6 +3442,7 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_ldap_variables(char **host, uint16 *host = hostname; *port = p; *ip_address = ip_addr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with ldap_variables version %llu, epoch %llu\n", hostname, p, version, epoch); proxy_info("Cluster: detected peer %s:%d with ldap_variables version %llu, epoch %llu\n", hostname, p, version, epoch); } } @@ -3041,6 +3500,7 @@ void ProxySQL_Cluster_Nodes::get_peer_to_sync_proxysql_servers(char **host, uint *host = hostname; *port = p; *ip_address = ip_addr; + proxy_debug(PROXY_DEBUG_CLUSTER, 5, "Detected peer %s:%d with proxysql_servers version %llu, epoch %llu\n", hostname, p, version, epoch); proxy_info("Cluster: detected peer %s:%d with proxysql_servers version %llu, epoch %llu\n", hostname, p, version, epoch); } } @@ -3064,14 +3524,15 @@ SQLite3_result * ProxySQL_Cluster_Nodes::stats_proxysql_servers_checksums() { //unsigned long long curtime = monotonic_time(); for( std::unordered_map::iterator it = umap_proxy_nodes.begin(); it != umap_proxy_nodes.end(); ) { ProxySQL_Node_Entry * node = it->second; - ProxySQL_Checksum_Value_2 * vals[6]; + ProxySQL_Checksum_Value_2 * vals[7]; vals[0] = &node->checksums_values.admin_variables; vals[1] = &node->checksums_values.mysql_query_rules; vals[2] = &node->checksums_values.mysql_servers; vals[3] = &node->checksums_values.mysql_users; vals[4] = &node->checksums_values.mysql_variables; vals[5] = &node->checksums_values.proxysql_servers; - for (int i=0; i<6 ; i++) { + vals[6] = &node->checksums_values.mysql_servers_v2; + for (int i=0; i<7 ; i++) { ProxySQL_Checksum_Value_2 *v = vals[i]; char **pta=(char **)malloc(sizeof(char *)*colnum); pta[0]=strdup(node->get_hostname()); @@ -3097,6 +3558,9 @@ SQLite3_result * ProxySQL_Cluster_Nodes::stats_proxysql_servers_checksums() { case 5: pta[2]=strdup((char *)"proxysql_servers"); break; + case 6: + pta[2]=strdup((char*)"mysql_servers_v2"); + break; default: break; } @@ -3221,6 +3685,7 @@ vector> get_module_checksums(ProxySQL_N res.push_back({"mysql_users", &entry->checksums_values.mysql_users}); res.push_back({"mysql_variables", &entry->checksums_values.mysql_variables}); res.push_back({"proxysql_servers", &entry->checksums_values.proxysql_servers}); + res.push_back({"mysql_servers_v2", &entry->checksums_values.mysql_servers_v2}); return res; } @@ -3854,7 +4319,8 @@ cluster_metrics_map = std::make_tuple( ProxySQL_Cluster::ProxySQL_Cluster() : proxysql_servers_to_monitor(NULL) { pthread_mutex_init(&mutex,NULL); pthread_mutex_init(&update_mysql_query_rules_mutex,NULL); - pthread_mutex_init(&update_mysql_servers_mutex,NULL); + pthread_mutex_init(&update_runtime_mysql_servers_mutex,NULL); + pthread_mutex_init(&update_mysql_servers_v2_mutex, NULL); pthread_mutex_init(&update_mysql_users_mutex,NULL); pthread_mutex_init(&update_proxysql_servers_mutex,NULL); pthread_mutex_init(&update_mysql_variables_mutex,NULL); @@ -3872,6 +4338,7 @@ ProxySQL_Cluster::ProxySQL_Cluster() : proxysql_servers_to_monitor(NULL) { cluster_mysql_servers_save_to_disk = true; cluster_mysql_users_save_to_disk = true; cluster_proxysql_servers_save_to_disk = true; + cluster_mysql_servers_sync_algorithm = 1; init_prometheus_counter_array(cluster_metrics_map, this->metrics.p_counter_array); init_prometheus_gauge_array(cluster_metrics_map, this->metrics.p_gauge_array); } diff --git a/lib/ProxySQL_GloVars.cpp b/lib/ProxySQL_GloVars.cpp index d095361675..195eba791e 100644 --- a/lib/ProxySQL_GloVars.cpp +++ b/lib/ProxySQL_GloVars.cpp @@ -405,6 +405,11 @@ uint64_t ProxySQL_GlobalVariables::generate_global_checksum() { myhash.Update(v->checksum,strlen(v->checksum)); myhash.Update(&v->version,sizeof(v->version)); } + v = &checksums_values.mysql_servers_v2; + if (v->version) { + myhash.Update(v->checksum, strlen(v->checksum)); + myhash.Update(&v->version, sizeof(v->version)); + } v = &checksums_values.mysql_users; if (v->version) { myhash.Update(v->checksum,strlen(v->checksum)); diff --git a/lib/debug.cpp b/lib/debug.cpp index 806cbea2bc..7d236f6a0f 100644 --- a/lib/debug.cpp +++ b/lib/debug.cpp @@ -474,6 +474,7 @@ void init_debug_struct() { GloVars.global.gdbg_lvl[PROXY_DEBUG_QUERY_STATISTICS].name=(char *)"debug_query_statistics"; GloVars.global.gdbg_lvl[PROXY_DEBUG_RESTAPI].name=(char *)"debug_restapi"; GloVars.global.gdbg_lvl[PROXY_DEBUG_MONITOR].name=(char *)"debug_monitor"; + GloVars.global.gdbg_lvl[PROXY_DEBUG_CLUSTER].name=(char *)"debug_cluster"; for (i=0;i module_sync_payloads { { update_mysql_servers, - "mysql_servers", + "mysql_servers_v2", "admin-cluster_mysql_servers_diffs_before_sync", }, { @@ -2156,6 +2156,7 @@ int main(int, char**) { std::make_tuple("admin-cluster_mysql_variables_save_to_disk" , "true" ), std::make_tuple("admin-cluster_proxysql_servers_diffs_before_sync" , "4" ), std::make_tuple("admin-cluster_proxysql_servers_save_to_disk" , "true" ), + std::make_tuple("admin-cluster_mysql_servers_sync_algorithm" , "1" ), // std::make_tuple("admin-cluster_username" , "" ), Known issue, can't clear // std::make_tuple("admin-cluster_password" , "" ), Known issue, can't clear // std::make_tuple("admin-debug" , "false" ), Should not be synced diff --git a/test/tap/tests/test_cluster_sync_config/test_cluster_sync-t.cnf b/test/tap/tests/test_cluster_sync_config/test_cluster_sync-t.cnf index 8a51f47dc9..51b163f34e 100644 --- a/test/tap/tests/test_cluster_sync_config/test_cluster_sync-t.cnf +++ b/test/tap/tests/test_cluster_sync_config/test_cluster_sync-t.cnf @@ -17,6 +17,7 @@ admin_variables= cluster_mysql_users_diffs_before_sync=3 cluster_admin_variables_diffs_before_sync=3 cluster_proxysql_servers_diffs_before_sync=3 + cluster_sync_interfaces=false } mysql_variables= diff --git a/test/tap/tests/test_cluster_sync_config/test_cluster_sync_nomonitor/.gitignore b/test/tap/tests/test_cluster_sync_config/test_cluster_sync_nomonitor/.gitignore new file mode 100644 index 0000000000..25cc06f021 --- /dev/null +++ b/test/tap/tests/test_cluster_sync_config/test_cluster_sync_nomonitor/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory + +# Except this file +!.gitignore \ No newline at end of file diff --git a/test/tap/tests/test_cluster_sync_config/test_cluster_sync_withmonitor/.gitignore b/test/tap/tests/test_cluster_sync_config/test_cluster_sync_withmonitor/.gitignore new file mode 100644 index 0000000000..25cc06f021 --- /dev/null +++ b/test/tap/tests/test_cluster_sync_config/test_cluster_sync_withmonitor/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory + +# Except this file +!.gitignore \ No newline at end of file diff --git a/test/tap/tests/test_cluster_sync_mysql_servers-t.cpp b/test/tap/tests/test_cluster_sync_mysql_servers-t.cpp new file mode 100644 index 0000000000..8d258710fe --- /dev/null +++ b/test/tap/tests/test_cluster_sync_mysql_servers-t.cpp @@ -0,0 +1,1076 @@ +/** + * @file test_cluster_sync_mysql_servers-t.cpp + * @brief Checks that ProxySQL mysql_server and mysql_server_v2 is properly syncing. + * @details Checks the sync of the following tables: + * - 'mysql_servers_v2' + * - 'mysql_servers' + * + * + * Test Cluster Isolation: + * ---------------------- + * For guaranteeing that this test doesn't invalidate the configuration of a running ProxySQL cluster and + * that after the test, the previous valid configuration is restored, the following actions are performed: + * + * 1. The Core nodes from the current cluster configuration are backup. + * 2. Primary (currently tested instance) is removed from the Core nodes. + * 3. A sync wait until all core nodes have performed the removal of primary is executed. + * 4. Now Primary is isolated from the previous cluster, tests can proceed. Primary is setup to hold itself + * in its 'proxysql_servers' as well as the target spawned replica. + * 5. After the tests recover the primary configuration and add it back to the Core nodes from Cluster: + * - Recover the previous 'mysql_servers' from disk, and load them to runtime, discarding any previous + * config performed during the test. + * - Insert the primary back into a Core node from cluster and wait for all nodes to sync including it. + * - Insert into the primary the previous backup Core nodes from Cluster and load to runtime. + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include +#ifndef SPOOKYV2 +#include "SpookyV2.h" +#define SPOOKYV2 +#endif +#include "tap.h" +#include "command_line.h" +#include "utils.h" + +#define MYSQL_QUERY__(mysql, query) \ + do { \ + if (mysql_query(mysql, query)) { \ + fprintf(stderr, "File %s, line %d, Error: %s\n", \ + __FILE__, __LINE__, mysql_error(mysql)); \ + goto cleanup; \ + } \ + } while(0) + +// GLOBAL TEST PARAMETERS +const uint32_t SYNC_TIMEOUT = 10; +const uint32_t CONNECT_TIMEOUT = 10; +const uint32_t R_NOMONITOR_PORT = 96061; +const uint32_t R_WITHMONITOR_PORT = 96062; + +const std::string t_debug_query = "mysql -u%s -p%s -h %s -P%d -C -e \"%s\""; + +using mysql_server_tuple = std::tuple; +using replication_hostgroups_tuple = std::tuple; + +/** + * @brief Computes the checksum for the resultset, excluding records labeled as 'OFFLINE_HARD', instead of checking each row individually. + * + * @param resultset mysql_servers + * + */ +uint64_t mysql_servers_raw_checksum(MYSQL_RES* resultset) { + if (resultset == nullptr) { return 0; } + + uint64_t num_rows = mysql_num_rows(resultset); + if (num_rows == 0) { return 0; } + + MYSQL_FIELD* fields = mysql_fetch_fields(resultset); + uint32_t num_fields = mysql_num_fields(resultset); + uint32_t status_idx = 0; + + for (uint32_t i = 0; i < num_fields; i++) { + if (strcmp(fields[i].name, "status") == 0) { + status_idx = i; + } + } + + SpookyHash myhash {}; + myhash.Init(19,3); + + while (MYSQL_ROW row = mysql_fetch_row(resultset)) { + for (uint32_t i = 0; i < num_fields; i++) { + if (strcmp(row[status_idx], "OFFLINE_HARD") == 0) { + continue; + } + + if (row[i]) { + if (strcmp(fields[i].name, "status") == 0) { + if (strcmp(row[i], "ONLINE") == 0 || strcmp(row[i], "SHUNNED") == 0) { + myhash.Update("0", strlen("0")); + } else { + myhash.Update("2", strlen("1")); + } + } else { + // computing 'strlen' is required see @details + myhash.Update(row[i], strlen(row[i])); + } + } else { + myhash.Update("", 0); + } + } + } + + // restore the initial resulset index + mysql_data_seek(resultset, 0); + + uint64_t res_hash = 0, hash2 = 0; + myhash.Final(&res_hash, &hash2); + + return res_hash; +} + + +/** + * @brief Helper function to verify that the sync of a table (or variable) have been performed. + * + * @param r_proxy_admin An already opened connection to ProxySQL. + * @param queries Queries to be executed that should return a **non-zero** value after the sync has taken place. + * @param sync_timeout Timeout for the sync to happen. + * + * @return EXIT_SUCCESS in case of success, otherwise: + * - '-1' if a query against Admin fails to be performed (failure is logged). + * - '-2' if timeout expired without sync being completed. + */ +int sync_checker(MYSQL* r_proxy_admin, const std::vector& queries, uint32_t sync_timeout) { + bool not_synced_query = false; + uint waited = 0; + + while (waited < sync_timeout) { + not_synced_query = false; + + // Check that all the entries have been synced + for (const auto& query : queries) { + int q_res = mysql_query(r_proxy_admin, query.c_str()); + if (q_res != EXIT_SUCCESS) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(r_proxy_admin)); + return -1; + } + + MYSQL_RES* proxysql_servers_res = mysql_store_result(r_proxy_admin); + MYSQL_ROW row = mysql_fetch_row(proxysql_servers_res); + int row_value = atoi(row[0]); + mysql_free_result(proxysql_servers_res); + + if (row_value == 0) { + not_synced_query = true; + break; + } + } + + if (not_synced_query) { + waited += 1; + sleep(1); + } else { + break; + } + } + + if (not_synced_query) { + return -2; + } else { + return EXIT_SUCCESS; + } +} + +int check_nodes_sync( + const CommandLine& cl, const std::vector& core_nodes, const std::string& check_query, uint32_t sync_timeout +) { + for (const auto& node : core_nodes) { + const std::string host { node[0] }; + const int port = std::stol(node[1]); + + MYSQL* c_node_admin = mysql_init(NULL); + if (!mysql_real_connect(c_node_admin, host.c_str(), cl.admin_username, cl.admin_password, NULL, port, NULL, 0)) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(c_node_admin)); + return EXIT_FAILURE; + } + + int not_synced = sync_checker(c_node_admin, { check_query }, sync_timeout); + if (not_synced != EXIT_SUCCESS) { + const std::string err_msg { "Node '" + host + ":" + std::to_string(port) + "' sync timed out" }; + fprintf(stderr, "File %s, line %d, Error: `%s`\n", __FILE__, __LINE__, err_msg.c_str()); + return EXIT_FAILURE; + } + } + + return EXIT_SUCCESS; +} + +int insert_mysql_servers_records(MYSQL* proxy_admin, const std::vector& insert_mysql_servers_values, + const std::vector& insert_replication_hostgroups_values) { + + MYSQL_QUERY(proxy_admin, "DELETE FROM mysql_servers"); + MYSQL_QUERY(proxy_admin, "DELETE FROM mysql_replication_hostgroups"); + + // Configure 'mysql_servers' and check sync with NULL comments + const char* t_insert_mysql_servers = + "INSERT INTO mysql_servers (" + " hostgroup_id, hostname, port, gtid_port, status, weight, compression, max_connections," + " max_replication_lag, use_ssl, max_latency_ms, comment" + ") VALUES (%d, '%s', %d, %d, '%s', %d, %d, %d, %d, %d, %d, '%s')"; + + const char* t_mysql_replication_hostgroups = + "INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type) VALUES (%d,%d,'%s')"; + + for (auto const& values : insert_mysql_servers_values) { + std::string insert_mysql_servers_hostgroup_query; + string_format( + t_insert_mysql_servers, + insert_mysql_servers_hostgroup_query, + std::get<0>(values), + std::get<1>(values).c_str(), + std::get<2>(values), + std::get<3>(values), + std::get<4>(values).c_str(), + std::get<5>(values), + std::get<6>(values), + std::get<7>(values), + std::get<8>(values), + std::get<9>(values), + std::get<10>(values), + std::get<11>(values).c_str() + ); + + // Insert the new mysql_servers hostgroups values + MYSQL_QUERY(proxy_admin, insert_mysql_servers_hostgroup_query.c_str()); + } + + for (auto const& values : insert_replication_hostgroups_values) { + std::string insert_mysql_replication_hostgroups_query; + string_format( + t_mysql_replication_hostgroups, + insert_mysql_replication_hostgroups_query, + std::get<0>(values), + std::get<1>(values), + std::get<2>(values).c_str() + ); + + // Insert the new mysql_replication_hostgroups values + MYSQL_QUERY(proxy_admin, insert_mysql_replication_hostgroups_query.c_str()); + } + + MYSQL_QUERY(proxy_admin, "LOAD MYSQL SERVERS TO RUNTIME"); + + return EXIT_SUCCESS; +} + +int wait_for_node_sync(MYSQL* r_proxy_admin, uint64_t master_checksum, const std::string& table) { + uint waited = 0; + bool not_synced = false; + const std::string& query = "SELECT * FROM " + table; + + while (waited < SYNC_TIMEOUT) { + not_synced = false; + + MYSQL_QUERY(r_proxy_admin, query.c_str()); + MYSQL_RES* mysql_servers_res = mysql_store_result(r_proxy_admin); + auto replica_checksum = mysql_servers_raw_checksum(mysql_servers_res); + mysql_free_result(mysql_servers_res); + + if (replica_checksum != master_checksum) { + not_synced = true; + diag("Waiting for '%s' to be synced", table.c_str()); + } + + if (not_synced) { + waited += 1; + sleep(1); + } else { + break; + } + } + + if (not_synced) { + diag("'wait_for_node_sync' timeout for query '%s'", query.c_str()); + } + + return not_synced; +}; + +int check_mysql_servers_sync( + const CommandLine& cl, MYSQL* proxy_admin, MYSQL* r_proxy_withmonitor_admin, MYSQL* r_proxy_nomonitor_admin +) { + std::string print_master_mysql_servers_hostgroups; + string_format(t_debug_query, print_master_mysql_servers_hostgroups, cl.admin_username, cl.admin_password, cl.host, cl.admin_port, "SELECT * FROM mysql_servers"); + std::string print_master_runtime_mysql_servers_hostgroups; + string_format(t_debug_query, print_master_runtime_mysql_servers_hostgroups, cl.admin_username, cl.admin_password, cl.host, cl.admin_port, "SELECT * FROM runtime_mysql_servers"); + + std::string print_nomonitor_replica_mysql_servers_hostgroups; + string_format(t_debug_query, print_nomonitor_replica_mysql_servers_hostgroups, "radmin", "radmin", cl.host, R_NOMONITOR_PORT, "SELECT * FROM mysql_servers"); + std::string print_nomonitor_replica_runtime_mysql_servers_hostgroups; + string_format(t_debug_query, print_nomonitor_replica_runtime_mysql_servers_hostgroups, "radmin", "radmin", cl.host, R_NOMONITOR_PORT, "SELECT * FROM runtime_mysql_servers"); + std::string print_nomonitor_replica_disk_mysql_servers_hostgroups; + string_format(t_debug_query, print_nomonitor_replica_disk_mysql_servers_hostgroups, "radmin", "radmin", cl.host, R_NOMONITOR_PORT, "SELECT * FROM disk.mysql_servers"); + + std::string print_withmonitor_replica_mysql_servers_hostgroups; + string_format(t_debug_query, print_withmonitor_replica_mysql_servers_hostgroups, "radmin", "radmin", cl.host, R_WITHMONITOR_PORT, "SELECT * FROM mysql_servers"); + std::string print_withmonitor_replica_runtime_mysql_servers_hostgroups; + string_format(t_debug_query, print_withmonitor_replica_runtime_mysql_servers_hostgroups, "radmin", "radmin", cl.host, R_WITHMONITOR_PORT, "SELECT * FROM runtime_mysql_servers"); + std::string print_withmonitor_replica_disk_mysql_servers_hostgroups; + string_format(t_debug_query, print_withmonitor_replica_disk_mysql_servers_hostgroups, "radmin", "radmin", cl.host, R_WITHMONITOR_PORT, "SELECT * FROM disk.mysql_servers"); + + + std::string variable_val; + + // get mysql_servers sync algorithm value + int g_err = get_variable_value(proxy_admin, "admin-cluster_mysql_servers_sync_algorithm", variable_val); + if (g_err) { return EXIT_FAILURE; } + const int cluster_sync_mysql_servers_algorithm = atoi(variable_val.c_str()); + + // get monitor enabled variable value + g_err = get_variable_value(proxy_admin, "mysql-monitor_enabled", variable_val); + if (g_err) { return EXIT_FAILURE; } + + bool monitor_enabled = false; + if (strcasecmp(variable_val.c_str(), "true") == 0 || strcasecmp(variable_val.c_str(), "1") == 0) { + monitor_enabled = true; + } + + // get save-to-disk variable value + g_err = get_variable_value(proxy_admin, "admin-cluster_mysql_servers_save_to_disk", variable_val); + if (g_err) { return EXIT_FAILURE; } + bool save_to_disk_value = false; + if (strcasecmp(variable_val.c_str(), "true") == 0 || strcasecmp(variable_val.c_str(), "1") == 0) { + save_to_disk_value = true; + } + + // get read_only interval variable value + g_err = get_variable_value(proxy_admin, "mysql-monitor_read_only_interval", variable_val); + if (g_err) { return EXIT_FAILURE; } + const long monitor_read_only_interval = std::stol(variable_val); + + // get read_only timeout variable value + g_err = get_variable_value(proxy_admin, "mysql-monitor_read_only_timeout", variable_val); + if (g_err) { return EXIT_FAILURE; } + const long monitor_read_only_timeout = std::stol(variable_val); + + diag("Checking mysql_servers_sync status " + "[admin-cluster_mysql_servers_sync_algorithm:'%d', " + "mysql-monitor_enabled:'%s', " + "admin-cluster_mysql_servers_save_to_disk:'%s'" + "]...", cluster_sync_mysql_servers_algorithm, (monitor_enabled ? "true" : "false"), (save_to_disk_value ? "true" : "false")); + + std::cout << "MASTER 'MYSQL SERVERS' TABLE BEFORE SYNC:" << std::endl; + system(print_master_mysql_servers_hostgroups.c_str()); + std::cout << std::endl; + + // Wait till read_only actions have been performed + uint64_t wait = monitor_read_only_interval + monitor_read_only_timeout; + usleep((wait * 1000) * 2); + + std::cout << "MASTER 'RUNTIME MYSQL SERVERS' TABLE BEFORE SYNC:" << std::endl; + system(print_master_runtime_mysql_servers_hostgroups.c_str()); + std::cout << std::endl; + + uint64_t master_mysql_servers_checksum = 0; + uint64_t master_runtime_mysql_servers_checksum = 0; + + // fetch master mysql_servers resultset and compute it's hash + { + MYSQL_QUERY(proxy_admin, "SELECT * FROM mysql_servers"); + MYSQL_RES* mysql_servers_res = mysql_store_result(proxy_admin); + master_mysql_servers_checksum = mysql_servers_raw_checksum(mysql_servers_res); + mysql_free_result(mysql_servers_res); + } + + // fetch master runtime_mysql_servers resultset and compute it's hash + { + MYSQL_QUERY(proxy_admin, "SELECT * FROM runtime_mysql_servers"); + MYSQL_RES* mysql_servers_res = mysql_store_result(proxy_admin); + master_runtime_mysql_servers_checksum = mysql_servers_raw_checksum(mysql_servers_res); + mysql_free_result(mysql_servers_res); + } + + // This comment is exclusively for this TAP test + // If monitor is enabled, records of runtime_mysql_servers and mysql_servers should not match + if (monitor_enabled == true) { + ok(master_mysql_servers_checksum != master_runtime_mysql_servers_checksum, "'runtime_mysql_servers' and 'mysql_servers' should not match."); + } else { + ok(master_mysql_servers_checksum == master_runtime_mysql_servers_checksum, "'runtime_mysql_servers' and 'mysql_servers' should match."); + } + // + + // SYNCH CHECK + bool not_synced_query = false; + + if (save_to_disk_value == false) { + + if (cluster_sync_mysql_servers_algorithm == 1) { + // Algo: 1 [Sync mysql_servers_v2 and runtime_mysql_servers] + + // Replica [WITHMONITOR] mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_mysql_servers_checksum, "mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_mysql_servers_checksum, "mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [WITHMONITOR] runtime_mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_runtime_mysql_servers_checksum, "runtime_mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'RUNTIME MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_runtime_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'runtime_mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] runtime_mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_runtime_mysql_servers_checksum, "runtime_mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'RUNTIME MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_runtime_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'runtime_mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + } else if (cluster_sync_mysql_servers_algorithm == 2) { + // Algo: 2 [Sync mysql_servers_v2 only] + + // Replica [WITHMONITOR] mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_mysql_servers_checksum, "mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_mysql_servers_checksum, "mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [WITHMONITOR] runtime_mysql_servers for both the replica and master will be identical. + // Reason: Replica [WITHMONITOR] has monitoring checks enabled and will generate identical runtime_mysql_servers records. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_runtime_mysql_servers_checksum, "runtime_mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'RUNTIME MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_runtime_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'runtime_mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] runtime_mysql_servers will be identical to master mysql_servers. + // Reason: Replica [NOMONITOR] has monitoring checks disabled, so runtime_mysql_servers will be identical to mysql_servers records. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_mysql_servers_checksum, "runtime_mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'RUNTIME MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_runtime_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'runtime_mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + } else if (cluster_sync_mysql_servers_algorithm == 3) { + // Algo: 3 [If the command line includes the argument "-M", Algorithm 1 will be selected. + // If "-M" is not provided, then Algorithm 2 will be chosen.] + + // Replica [WITHMONITOR] mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_mysql_servers_checksum, "mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] mysql_servers for both the replica and master will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_mysql_servers_checksum, "mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [WITHMONITOR] runtime_mysql_servers for both the replica and master will be identical. + // Reason: Algorithm 2 will be selected [Sync mysql_servers_v2]. After read_only_action, + // the runtime_mysql_servers for replica becomes identical to master. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_runtime_mysql_servers_checksum, "runtime_mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'RUNTIME MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_runtime_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'runtime_mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] runtime_mysql_servers for both the replica and master will be identical. + // Reason: Algorithm 1 will be selected [Sync mysql_servers_v2 and runtime_mysql_servers] + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_runtime_mysql_servers_checksum, "runtime_mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'RUNTIME MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_runtime_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'runtime_mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + } + } else { //save_to_disk_value is true + + // If Algorithm 1 is selected, the runtime_mysql_servers data will be saved to disk. + // If Algorithm 2 is selected, the mysql_servers data will be saved to disk. + // However, for Algorithm 3, the data saved to disk will depend on the algorithm that is chosen based on the -M argument. + + if (cluster_sync_mysql_servers_algorithm == 1) { + + // Replica [WITHMONITOR] disk.mysql_servers and master runtime_mysql_servers will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_runtime_mysql_servers_checksum, "disk.mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_disk_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'disk.mysql_servers' should be identical to 'runtime_mysql_servers'."); + std::cout << std::endl; + + // Replica [NOMONITOR] disk.mysql_servers and master runtime_mysql_servers will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_runtime_mysql_servers_checksum, "disk.mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_disk_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'disk.mysql_servers' should be identical to 'runtime_mysql_servers'."); + std::cout << std::endl; + + } else if (cluster_sync_mysql_servers_algorithm == 2) { + + // Replica [WITHMONITOR] disk.mysql_servers and master mysql_servers will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_mysql_servers_checksum, "disk.mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_disk_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'disk.mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] disk.mysql_servers and master mysql_servers will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_mysql_servers_checksum, "disk.mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_disk_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'disk.mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + } else if (cluster_sync_mysql_servers_algorithm == 3) { + + // Replica [WITHMONITOR] disk.mysql_servers and master mysql_servers will be identical. + not_synced_query = wait_for_node_sync(r_proxy_withmonitor_admin, master_mysql_servers_checksum, "disk.mysql_servers"); + std::cout << "REPLICA [WITHMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_withmonitor_replica_disk_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'disk.disk.mysql_servers' with NULL comments should be synced."); + std::cout << std::endl; + + // Replica [NOMONITOR] disk.mysql_servers and master runtime_mysql_servers will be identical. + not_synced_query = wait_for_node_sync(r_proxy_nomonitor_admin, master_runtime_mysql_servers_checksum, "disk.mysql_servers"); + std::cout << "REPLICA [NOMONITOR] 'MYSQL SERVERS' TABLE AFTER SYNC:" << std::endl; + system(print_nomonitor_replica_disk_mysql_servers_hostgroups.c_str()); + ok(not_synced_query == false, "'disk.mysql_servers' with NULL comments should be synced."); + } + } + + diag("Checking mysql_servers_sync status " + "[admin-cluster_mysql_servers_sync_algorithm:'%d', " + "mysql-monitor_enabled:'%s', " + "admin-cluster_mysql_servers_save_to_disk:'%s'" + "]... Done", cluster_sync_mysql_servers_algorithm, (monitor_enabled ? "true" : "false"), (save_to_disk_value ? "true" : "false")); + + return EXIT_SUCCESS; +} + +/** + * @brief Assumes that 'proxysql_servers' holds at least the one entry required for this test. + * @details It's assumed that primary ProxySQL is part of a Cluster. + */ +int update_proxysql_servers(const CommandLine& cl, MYSQL* admin) { + const char update_proxysql_servers_t[] { + "UPDATE proxysql_servers SET comment='%s' WHERE hostname='%s' and port=%d" + }; + + cfmt_t update_servers { + cstr_format(update_proxysql_servers_t, std::to_string(time(NULL)).c_str(), cl.host, cl.admin_port) + }; + MYSQL_QUERY_T(admin, update_servers.str.c_str()); + MYSQL_QUERY_T(admin, "LOAD PROXYSQL SERVERS TO RUNTIME"); + + return EXIT_SUCCESS; +} + +int setup_config_file(const CommandLine& cl, uint32_t r_port, const std::string& config_filename) { + const std::string& workdir = std::string(cl.workdir); + const std::string& t_fmt_config_file = workdir + "test_cluster_sync_config/test_cluster_sync-t.cnf"; + const std::string& fmt_config_file = workdir + "test_cluster_sync_config/test_cluster_sync_" + config_filename + "/test_cluster_sync.cnf"; + const std::string& datadir_path = workdir + "test_cluster_sync_config/test_cluster_sync_" + config_filename; + + // Prepare the configuration file + config_t cfg {}; + + config_init(&cfg); + + if (!config_read_file(&cfg, t_fmt_config_file.c_str())) { + fprintf(stderr, "%s:%d - %s\n", config_error_file(&cfg), config_error_line(&cfg), config_error_text(&cfg)); + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - Error reading config file."); + config_destroy(&cfg); + return -1; + } + + config_setting_t* r_datadir = config_lookup(&cfg, "datadir"); + if (r_datadir == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - 'datadir' setting not found."); + return -1; + } + + config_setting_t* r_admin_vars = config_lookup(&cfg, "admin_variables"); + if (r_admin_vars == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - 'admin_variables' setting not found."); + return -1; + } + + config_setting_t* r_mysql_ifaces = config_setting_get_member(r_admin_vars, "mysql_ifaces"); + if (r_mysql_ifaces == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - 'mysql_ifaces' setting not found."); + return -1; + } + + int r_ifaces_res = config_setting_set_string(r_mysql_ifaces, std::string { "0.0.0.0:" + std::to_string(r_port) }.c_str()); + if (r_ifaces_res == CONFIG_FALSE) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - Error while trying to set the values for 'mysql_ifaces'."); + return -1; + } + + config_setting_t* p_servers = config_lookup(&cfg, "proxysql_servers"); + if (p_servers == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - 'proxysql_servers' setting not found."); + return -1; + } + + int r_datadir_res = config_setting_set_string(r_datadir, datadir_path.c_str()); + if (r_datadir_res == CONFIG_FALSE) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - Error while trying to set the 'datadir' value."); + return -1; + } + + // Get first group settings + config_setting_t* r_pserver_group = config_setting_get_elem(p_servers, 0); + if (r_pserver_group == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - 'proxysql_servers' doesn't contains first group."); + return -1; + } + config_setting_t* r_pserver_hostname = config_setting_get_member(r_pserver_group, "hostname"); + config_setting_t* r_pserver_port = config_setting_get_member(r_pserver_group, "port"); + + // Check the group members + if (r_pserver_hostname == nullptr || r_pserver_port == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - 'proxysql_servers' doesn't contains the necessary group members."); + return -1; + } + + int fhost_res = config_setting_set_string(r_pserver_hostname, cl.host); + int fport_res = config_setting_set_int(r_pserver_port, cl.admin_port); + + if (fhost_res == CONFIG_FALSE || fport_res == CONFIG_FALSE) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Invalid config file - Error while trying to set the values from env variables."); + return -1; + } + + // Write the new config file + if (config_write_file(&cfg, fmt_config_file.c_str()) == CONFIG_FALSE) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, "Config file - Error while trying to write the new config file."); + return -1; + } + + config_destroy(&cfg); + + return 0; +} + +int launch_proxysql_replica(const CommandLine& cl, uint32_t r_port, const std::string config_filename, bool monitor_enabled, + const std::atomic& save_proxy_stderr) { + + const std::string& workdir = std::string(cl.workdir); + const std::string& replica_stderr = workdir + "test_cluster_sync_config/test_cluster_sync_" + config_filename + "/cluster_sync_node_stderr.txt"; + const std::string& proxysql_db = workdir + "test_cluster_sync_config/test_cluster_sync_" + config_filename + "/proxysql.db"; + const std::string& stats_db = workdir + "test_cluster_sync_config/test_cluster_sync_" + config_filename + "/proxysql_stats.db"; + const std::string& fmt_config_file = workdir + "test_cluster_sync_config/test_cluster_sync_" + config_filename + "/test_cluster_sync.cnf"; + + // Setup the config file using the env variables in 'CommandLine' + if (setup_config_file(cl, r_port, config_filename)) { + return EXIT_FAILURE; + } + + const std::string& proxy_binary_path = workdir + "../../../src/proxysql"; + const std::string& proxy_command = proxy_binary_path + " -f " + (monitor_enabled == false ? "-M" : "") + " -c " + fmt_config_file + " > " + replica_stderr + " 2>&1"; + + diag("Launching replica ProxySQL [%s] via 'system' with command : `%s`", config_filename.c_str(), proxy_command.c_str()); + int exec_res = system(proxy_command.c_str()); + + ok(exec_res == 0, "proxysql cluster node [%s] should execute and shutdown nicely. 'system' result was: %d", config_filename.c_str(), exec_res); + + // In case of error place in log the reason + if (exec_res || save_proxy_stderr.load()) { + if (exec_res) { + diag("LOG: Proxysql cluster node [%s] execution failed, logging stderr into 'test_cluster_sync_node_stderr_%s.txt", config_filename.c_str(), config_filename.c_str()); + } else { + diag("LOG: One of the tests failed to pass, logging stderr 'test_cluster_sync_node_stderr_%s.txt", config_filename.c_str()); + } + } + + remove(proxysql_db.c_str()); + remove(stats_db.c_str()); + + return EXIT_SUCCESS; +} + +int get_read_only_value(const std::string& host, uint16_t port, const std::string& username, const std::string& password, + int* read_only_val) { + + // check is mysql server has read_only value 0 + MYSQL* mysqldb = mysql_init(NULL); + + // Initialize connections + if (!mysqldb) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(mysqldb)); + return EXIT_FAILURE; + } + + // Connnect to local proxysql + if (!mysql_real_connect(mysqldb, host.c_str(), username.c_str(), password.c_str(), NULL, port, NULL, 0)) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(mysqldb)); + mysql_close(mysqldb); + return EXIT_FAILURE; + } + + const int rc_query = mysql_query(mysqldb,"SELECT @@global.read_only read_only"); + + if (rc_query == 0) { + MYSQL_RES *result = mysql_store_result(mysqldb); + MYSQL_ROW row; + + while ((row = mysql_fetch_row(result))) { + + if (row[0]) { + *read_only_val = static_cast(std::strtoul(row[0], NULL, 10)); + } + } + + mysql_free_result(result); + } + + mysql_close(mysqldb); + + return EXIT_SUCCESS; +} + +std::vector> queries = { + { + "SET mysql-monitor_read_only_interval=200", // setting read_only variables + "SET mysql-monitor_read_only_timeout=100", + "SET mysql-monitor_enabled='true'", // enabling monitor + "LOAD MYSQL VARIABLES TO RUNTIME", + "UPDATE global_variables SET variable_value='1' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'",// setting admin-cluster_mysql_servers_sync_algorithm to 1 -> fetch mysql_servers_v2 and runtime_mysql_servers + "LOAD ADMIN VARIABLES TO RUNTIME" + }, + { + "UPDATE global_variables SET variable_value='2' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type) VALUES (999,998,'read_only')", // adding dummy data so replica nodes can sync after algorithm change from 1 to 2. + "LOAD MYSQL SERVERS TO RUNTIME" + }, + { + "UPDATE global_variables SET variable_value='3' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "DELETE FROM mysql_replication_hostgroups WHERE writer_hostgroup=999 AND reader_hostgroup=998 AND check_type='read_only'", // deleting dummy data so replica nodes can sync after algorithm change from 2 to 3. + "LOAD MYSQL SERVERS TO RUNTIME" + }, + { + "SET mysql-monitor_enabled='false'", + "LOAD MYSQL VARIABLES TO RUNTIME", + "LOAD MYSQL SERVERS TO RUNTIME", // to regenerate runtime_mysql_servers + "UPDATE global_variables SET variable_value='1' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME" + }, + { + "UPDATE global_variables SET variable_value='2' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type) VALUES (999,998,'read_only')",// adding dummy data so replica nodes can sync after algorithm change from 1 to 2. + "LOAD MYSQL SERVERS TO RUNTIME" + }, + { + "UPDATE global_variables SET variable_value='3' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "DELETE FROM mysql_replication_hostgroups WHERE writer_hostgroup=999 AND reader_hostgroup=998 AND check_type='read_only'", // deleting dummy data so replica nodes can sync after algorithm change from 2 to 3. + "LOAD MYSQL SERVERS TO RUNTIME" + }, + { + // save to disk + "SET mysql-monitor_enabled='true'", + "LOAD MYSQL VARIABLES TO RUNTIME", + "UPDATE global_variables SET variable_value='true' WHERE variable_name='admin-cluster_mysql_servers_save_to_disk'", // setting admin-cluster_mysql_servers_save_to_disk to true + "UPDATE global_variables SET variable_value='1' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type) VALUES (997,996,'read_only')", // adding dummy data so replica nodes can sync after algorithm change from 1 to 2. + "LOAD MYSQL SERVERS TO RUNTIME" + }, + { + "UPDATE global_variables SET variable_value='2' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "DELETE FROM mysql_replication_hostgroups WHERE writer_hostgroup=997 AND reader_hostgroup=996 AND check_type='read_only'", // deleting dummy data so replica nodes can sync after algorithm change from 2 to 3. + "LOAD MYSQL SERVERS TO RUNTIME" + }, + { + "UPDATE global_variables SET variable_value='3' WHERE variable_name='admin-cluster_mysql_servers_sync_algorithm'", + "LOAD ADMIN VARIABLES TO RUNTIME", + "INSERT INTO mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup, check_type) VALUES (997,996,'read_only')",// adding dummy data so replica nodes can sync after algorithm change from 1 to 2. + "LOAD MYSQL SERVERS TO RUNTIME" + } +}; + +int main(int, char**) { + + CommandLine cl; + std::atomic save_proxy_stderr(false); + + if (cl.getEnv()) { + diag("Failed to get the required environmental variables."); + return EXIT_FAILURE; + } + + plan( 1 + 1 // replica instances + + 1 // confirming mysql server 127.0.0.1:13306 is a writer + + (6 * 5) // calling check_mysql_servers_sync 7 times, 5 differnt checks in each call + + (3 * 3) + + 1 + 1 // shutting down replica instances + ); + + MYSQL* proxy_admin = mysql_init(NULL); + + // Initialize connections + if (!proxy_admin) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(proxy_admin)); + return EXIT_FAILURE; + } + + // Connnect to local proxysql + if (!mysql_real_connect(proxy_admin, cl.host, cl.admin_username, cl.admin_password, NULL, cl.admin_port, NULL, 0)) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(proxy_admin)); + return EXIT_FAILURE; + } + + const std::string t_update_proxysql_servers { + "INSERT INTO proxysql_servers (hostname, port, weight, comment) VALUES ('%s', %d, 0, 'proxysql')" + }; + + std::string update_proxysql_servers; + string_format(t_update_proxysql_servers, update_proxysql_servers, cl.host, cl.admin_port); + + // 1. Backup the Core nodes from current cluster configuration + MYSQL_QUERY(proxy_admin, "DROP TABLE IF EXISTS proxysql_servers_sync_test_backup_2687"); + MYSQL_QUERY(proxy_admin, "CREATE TABLE proxysql_servers_sync_test_backup_2687 AS SELECT * FROM proxysql_servers"); + + // 2. Remove primary from Core nodes + MYSQL_QUERY(proxy_admin, "DELETE FROM proxysql_servers WHERE hostname=='127.0.0.1' AND PORT==6032"); + MYSQL_QUERY(proxy_admin, "LOAD PROXYSQL SERVERS TO RUNTIME"); + MYSQL_QUERY(proxy_admin, "SELECT hostname,port FROM proxysql_servers"); + MYSQL_RES* my_res = mysql_store_result(proxy_admin); + std::vector core_nodes { extract_mysql_rows(my_res) }; + mysql_free_result(my_res); + + // 3. Wait for all Core nodes to sync (confirm primary out of core nodes) + std::string check_no_primary_query {}; + string_format( + "SELECT CASE COUNT(*) WHEN 0 THEN 1 ELSE 0 END FROM proxysql_servers WHERE hostname=='%s' AND port==%d", + check_no_primary_query, cl.host, cl.admin_port + ); + + int check_res = check_nodes_sync(cl, core_nodes, check_no_primary_query, SYNC_TIMEOUT); + if (check_res != EXIT_SUCCESS) { return EXIT_FAILURE; } + + // 4. Remove all current servers from primary instance (only secondary sync matters) + MYSQL_QUERY(proxy_admin, "DELETE FROM proxysql_servers"); + MYSQL_QUERY(proxy_admin, update_proxysql_servers.c_str()); + MYSQL_QUERY(proxy_admin, "LOAD PROXYSQL SERVERS TO RUNTIME"); + + // disable admin-cluster_mysql_servers_save_to_disk before executing replicas + MYSQL_QUERY(proxy_admin, "UPDATE global_variables SET variable_value='false' WHERE variable_name='admin-cluster_mysql_servers_save_to_disk'"); // setting admin-cluster_mysql_servers_save_to_disk to false + MYSQL_QUERY(proxy_admin, "LOAD ADMIN VARIABLES TO RUNTIME"); + + // cleaning old records + MYSQL_QUERY(proxy_admin, "DELETE FROM mysql_servers"); + MYSQL_QUERY(proxy_admin, "DELETE FROM mysql_replication_hostgroups"); + MYSQL_QUERY(proxy_admin, "LOAD MYSQL SERVERS TO RUNTIME"); + + // Launch proxysql with cluster config and monitor feature disabled + std::thread proxysql_replica_nomonitor_thd(launch_proxysql_replica, std::ref(cl), R_NOMONITOR_PORT, "nomonitor", false, std::ref(save_proxy_stderr)); + + // Launch proxysql with cluster config - with -M commandline + std::thread proxysql_replica_withmonitor_thd(launch_proxysql_replica, std::ref(cl), R_WITHMONITOR_PORT, "withmonitor", true, std::ref(save_proxy_stderr)); + + MYSQL* r_proxysql_nomonitor_admin = NULL; + MYSQL* r_proxysql_withmonitor_admin = NULL; + { + // Waiting for proxysql to be ready + conn_opts_t conn_opts_nomonitor {}; + conn_opts_nomonitor.host = cl.host; + conn_opts_nomonitor.user = "radmin"; + conn_opts_nomonitor.pass = "radmin"; + conn_opts_nomonitor.port = R_NOMONITOR_PORT; + + // connect to proxsqyl replica [nomonitor] + r_proxysql_nomonitor_admin = wait_for_proxysql(conn_opts_nomonitor, CONNECT_TIMEOUT); + + // Once the thread is spanwed we should always go to cleanup to wait + ok(r_proxysql_nomonitor_admin != nullptr, "New instance of proxysql [nomonitor] with cluster config should be properly spawned."); + + if (r_proxysql_nomonitor_admin == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(r_proxysql_nomonitor_admin)); + goto cleanup; + } + + conn_opts_t conn_opts_withmonitor {}; + conn_opts_withmonitor.host = cl.host; + conn_opts_withmonitor.user = "radmin"; + conn_opts_withmonitor.pass = "radmin"; + conn_opts_withmonitor.port = R_WITHMONITOR_PORT; + + // connect to proxsqyl replica [nomonitor] + r_proxysql_withmonitor_admin = wait_for_proxysql(conn_opts_withmonitor, CONNECT_TIMEOUT); + + // Once the thread is spanwed we should always go to cleanup to wait + ok(r_proxysql_withmonitor_admin != nullptr, "New instance of proxysql [withmonitor] with cluster config should be properly spawned."); + + if (r_proxysql_withmonitor_admin == nullptr) { + fprintf(stderr, "File %s, line %d, Error: %s\n", __FILE__, __LINE__, mysql_error(r_proxysql_withmonitor_admin)); + goto cleanup; + } + + int read_only_val = -1; + int result = get_read_only_value("127.0.0.1", 13306, "root", "root", &read_only_val); + if (result != EXIT_SUCCESS) { + fprintf(stderr, "File %s, line %d, Error: `%s`\n", __FILE__, __LINE__, "Fetching read_only value from mysql server failed."); + goto cleanup; + } + + // For thorough testing of synchronization under all possible scenarios, it is necessary for + // the MySQL server at 127.0.0.1:13306 to function as a writer. + ok(read_only_val == 0, "MySQL Server '127.0.0.1:13306' should function as a writer"); + + const std::vector insert_mysql_servers_values { + std::make_tuple(1, "127.0.0.1", 13306, 12, "ONLINE", 1, 1, 1000, 300, 1, 200, ""), // this server has read_only value 0 (writer) + std::make_tuple(2, "127.0.0.1", 13307, 13, "OFFLINE_SOFT", 2, 1, 500, 300, 1, 200, ""), + std::make_tuple(3, "127.0.0.1", 13308, 14, "OFFLINE_HARD", 2, 1, 500, 300, 1, 200, ""), + std::make_tuple(4, "127.0.0.1", 13309, 15, "SHUNNED", 1, 0, 500, 300, 1, 200, "") + }; + + const std::vector insert_replication_hostgroups_values { + std::make_tuple(0, 1, "read_only") // Here we are assigning the hostgroup to the reader, and read-only actions will creating a new entry in hostgroup 0. + }; + + // Inserting new records into 'mysql_servers' and 'mysql_replication_hostgroups'. + result = insert_mysql_servers_records(proxy_admin, insert_mysql_servers_values, insert_replication_hostgroups_values); + + if (result != EXIT_SUCCESS) { + fprintf(stderr, "File %s, line %d, Error: `%s`\n", __FILE__, __LINE__, "Failed to insert records in mysql_servers table."); + goto cleanup; + } + + for (const auto& pre_queries : queries) { + + for (const std::string& query : pre_queries) { + MYSQL_QUERY__(proxy_admin, query.c_str()); + usleep(1000000); + } + sleep(2); + + result = check_mysql_servers_sync(cl, proxy_admin, r_proxysql_withmonitor_admin, r_proxysql_nomonitor_admin); + if (result != EXIT_SUCCESS) { + fprintf(stderr, "File %s, line %d, Error: `%s`\n", __FILE__, __LINE__, "Checking mysql servers sync records failed."); + goto cleanup; + } + } + } + +cleanup: + // In case of test failing, save the stderr output from the spawned proxysql instance + if (tests_failed() != 0) { + save_proxy_stderr.store(true); + } + + if (r_proxysql_nomonitor_admin) { + int mysql_timeout = 2; + + mysql_options(r_proxysql_nomonitor_admin, MYSQL_OPT_CONNECT_TIMEOUT, &mysql_timeout); + mysql_options(r_proxysql_nomonitor_admin, MYSQL_OPT_READ_TIMEOUT, &mysql_timeout); + mysql_options(r_proxysql_nomonitor_admin, MYSQL_OPT_WRITE_TIMEOUT, &mysql_timeout); + mysql_query(r_proxysql_nomonitor_admin, "PROXYSQL SHUTDOWN"); + mysql_close(r_proxysql_nomonitor_admin); + } + + if (r_proxysql_withmonitor_admin) { + int mysql_timeout = 2; + + mysql_options(r_proxysql_withmonitor_admin, MYSQL_OPT_CONNECT_TIMEOUT, &mysql_timeout); + mysql_options(r_proxysql_withmonitor_admin, MYSQL_OPT_READ_TIMEOUT, &mysql_timeout); + mysql_options(r_proxysql_withmonitor_admin, MYSQL_OPT_WRITE_TIMEOUT, &mysql_timeout); + mysql_query(r_proxysql_withmonitor_admin, "PROXYSQL SHUTDOWN"); + mysql_close(r_proxysql_withmonitor_admin); + } + + proxysql_replica_nomonitor_thd.join(); + proxysql_replica_withmonitor_thd.join(); + + // Recover primary ProxySQL MySQL and ProxySQL servers + diag("RESTORING: Recovering primary configuration..."); + + { + // Recover previous MySQL servers and generate a newer checksum for primary + MYSQL_QUERY(proxy_admin, "LOAD MYSQL SERVERS FROM DISK"); + MYSQL_QUERY(proxy_admin, "LOAD MYSQL SERVERS TO RUNTIME"); + + // Insert primary into another Core node config and wait for replication + diag("RESTORING: Inserting primary back into Core nodes"); + bool recovered_servers_st = false; + + std::string insert_query {}; + string_format( + "INSERT INTO proxysql_servers (hostname,port,weight,comment) VALUES ('%s',%d,0,'proxysql')", + insert_query, cl.host, cl.admin_port + ); + + for (const auto& row : core_nodes) { + const std::string host { row[0] }; + const int port = std::stol(row[1]); + MYSQL* c_node_admin = mysql_init(NULL); + + diag("RESTORING: Inserting into node '%s:%d'", host.c_str(), port); + + if (!mysql_real_connect(c_node_admin, host.c_str(), cl.admin_username, cl.admin_password, NULL, port, NULL, 0)) { + const std::string err_msg { + "Connection to core node failed with '" + std::string { mysql_error(c_node_admin) } + "'. Retrying..." + }; + fprintf(stderr, "File %s, line %d, Error: `%s`\n", __FILE__, __LINE__, err_msg.c_str()); + mysql_close(c_node_admin); + continue; + } + + int my_rc = mysql_query(c_node_admin, insert_query.c_str()); + if (my_rc == EXIT_SUCCESS) { + mysql_query(c_node_admin, "LOAD PROXYSQL SERVERS TO RUNTIME"); + break; + } else { + const std::string err_msg { + "Insert primary into node failed with: '" + std::string { mysql_error(c_node_admin) } + "'" + }; + fprintf(stderr, "File %s, line %d, Error: `%s`\n", __FILE__, __LINE__, err_msg.c_str()); + } + } + + // Wait for sync after primary insertion into Core node + std::string check_for_primary {}; + string_format( + "SELECT COUNT(*) FROM proxysql_servers WHERE hostname=='%s' AND port==%d", check_no_primary_query, + cl.host, cl.admin_port + ); + + // Wait for the other nodes to sync ProxySQL servers to include Primary + int check_res = check_nodes_sync(cl, core_nodes, check_no_primary_query, SYNC_TIMEOUT); + if (check_res != EXIT_SUCCESS) { return EXIT_FAILURE; } + + // Recover the old ProxySQL servers from backup in primary + MYSQL_QUERY(proxy_admin, "DELETE FROM proxysql_servers"); + MYSQL_QUERY(proxy_admin, "INSERT INTO proxysql_servers SELECT * FROM proxysql_servers_sync_test_backup_2687"); + MYSQL_QUERY(proxy_admin, "DROP TABLE proxysql_servers_sync_test_backup_2687"); + MYSQL_QUERY(proxy_admin, "LOAD PROXYSQL SERVERS TO RUNTIME"); + } + + mysql_close(proxy_admin); + + return exit_status(); +}