Skip to content

Commit

Permalink
Document wsrep_sync_wait database option
Browse files Browse the repository at this point in the history
  • Loading branch information
@yhabteab
yhabteab committed Mar 27, 2024
1 parent eea3ea5 commit 7f14d28
Show file tree
Hide file tree
Showing 3 changed files with 48 additions and 15 deletions.
8 changes: 8 additions & 0 deletions config.example.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ database:
# Database password.
password: CHANGEME

# List of low-level database options that can be set to influence some Icinga DB internal default behaviours.
options:
# Enforce Galera cluster nodes to perform strict cluster-wide causality checks before executing
# specific SQL queries determined by the number you provided.
# Defaults to 7.
# See https://icinga.com/docs/icinga-db/latest/doc/03-Configuration/#galera-cluster
# wsrep_sync_wait: 7

# Connection configuration for the Redis server where Icinga 2 writes its configuration, state and history items.
# This is the same connection as configured in the 'icingadb' feature of the corresponding Icinga 2 node.
# High availability setups require a dedicated Redis server per Icinga 2 node and
Expand Down
49 changes: 36 additions & 13 deletions doc/03-Configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,26 @@ This is also the database used in
[Icinga DB Web](https://icinga.com/docs/icinga-db-web) to view and work with the data.
In high availability setups, all Icinga DB instances must write to the same database.

| Option | Description |
|----------|--------------------------------------------------------------------------------------------------------|
| type | **Optional.** Either `mysql` (default) or `pgsql`. |
| host | **Required.** Database host or absolute Unix socket path. |
| port | **Optional.** Database port. By default, the MySQL or PostgreSQL port, depending on the database type. |
| database | **Required.** Database name. |
| user | **Required.** Database username. |
| password | **Optional.** Database password. |
| tls | **Optional.** Whether to use TLS. |
| cert | **Optional.** Path to TLS client certificate. |
| key | **Optional.** Path to TLS private key. |
| ca | **Optional.** Path to TLS CA certificate. |
| insecure | **Optional.** Whether not to verify the peer. |
| Option | Description |
|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| type | **Optional.** Either `mysql` (default) or `pgsql`. |
| host | **Required.** Database host or absolute Unix socket path. |
| port | **Optional.** Database port. By default, the MySQL or PostgreSQL port, depending on the database type. |
| database | **Required.** Database name. |
| user | **Required.** Database username. |
| password | **Optional.** Database password. |
| tls | **Optional.** Whether to use TLS. |
| cert | **Optional.** Path to TLS client certificate. |
| key | **Optional.** Path to TLS private key. |
| ca | **Optional.** Path to TLS CA certificate. |
| insecure | **Optional.** Whether not to verify the peer. |
| options | **Optional.** List of low-level database options that can be set to influence some Icinga DB internal default behaviours. See [database options](#database-options) for details. |

### Database Options

| Option | Description |
|-----------------|---------------------------------------------------------------------------------------------------------------------------------|
| wsrep_sync_wait | **Optional.** Enforce [Galera cluster](#galera-cluster) nodes to perform strict cluster-wide causality checks. Defaults to `7`. |

## Logging Configuration

Expand Down Expand Up @@ -91,3 +98,19 @@ allowing to keep this information for longer with a smaller storage footprint.

A duration string is a sequence of decimal numbers and a unit suffix, such as `"20s"`.
Valid units are `"ms"`, `"s"`, `"m"` and `"h"`.

### Galera Cluster

Icinga DB might not function properly or even crash occasionally when it is used with [Galera cluster](https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/) setups.
To avoid such unexpected crashes and reading/writing inconsistent data, Icinga DB sets the [wsrep_sync_wait](https://mariadb.com/kb/en/galera-cluster-system-variables/#wsrep_sync_wait)
system variable for all its database connections. Consequently, strict cluster-wide causality checks are enforced before
executing specific SQL queries, which are determined by the value set in the `wsrep_sync_wait` system variable. By default,
Icinga DB sets this to `7`, which includes `READ, UPDATE, DELETE, INSERT, REPLACE` query types and is usually sufficient.
Unfortunately, this also has the downside that every single Icinga DB query will be blocked until the cluster nodes
resynchronise their states after each executed query, and may result in degraded performance.

However, this does not necessarily have to be the case if, for instance, Icinga DB is only allowed to connect to a single
cluster node at a time. This is the case when a load balancer does not randomly route connections to all the nodes evenly,
but always to the same node until it fails, or if your database cluster nodes have a virtual IP address fail over
assigned. In such situations, you can set the `wsrep_sync_wait` system variable to `0` in the `/etc/icingadb/config.yml` file to disable
it entirely, as Icinga DB doesn't have to wait for cluster synchronisation then.
6 changes: 4 additions & 2 deletions pkg/icingadb/db.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,10 @@ type Options struct {
// The default is 2^13, which in our tests showed the best performance in terms of execution time and parallelism.
MaxRowsPerTransaction int `yaml:"max_rows_per_transaction" default:"8192"`

// WsrepSyncWait defines which kinds of SQL statements catch up all pending sync between nodes first, see:
// https://mariadb.com/kb/en/galera-cluster-system-variables/#wsrep_sync_wait
// WsrepSyncWait enforces Galera cluster nodes to perform strict cluster-wide causality checks
// before executing specific SQL queries determined by the number you provided.
// Please refer to the below link for a detailed description.
// https://icinga.com/docs/icinga-db/latest/doc/03-Configuration/#galera-cluster
WsrepSyncWait int `yaml:"wsrep_sync_wait" default:"7"`
}

Expand Down

0 comments on commit 7f14d28

Please sign in to comment.