- Backing Up the Mailbox Server
- Backup Methods
- Directory Structure for Backup Files
- Backup and Restore Using the Administration Console
- Backup and Restore Using the Command Line Interface
- Backing up using the Standard Method
- Aborting a Full Backup in Progress
- Backup using the Auto-Grouped Method
- Backup Options
- Managing Disk Space for Backups
- Restoring Data
- General Steps for Disaster Recovery
- Using snapshots to Backup and Restore
- Notes on Ephemeral Data
|
Note
|
Starting with Zimbra 8.8, there are two versions of this feature, Standard and New Generation (NG) versions. Zimbra 8.7 and earlier included only the Standard version, which is explained below. To use and enable the NG version of this feature with Zimbra 8.8 and later, refer to the specific NG chapter later in this Guide. |
|
Warning
|
Standard Backup (aka Classic Backup) is deprecated and will be removed in the next release following Zimbra 8.8.12. Installations that have not yet migrated to new Network Edition NG Modules or ZSP are encouraged to do so. |
{product-name} includes a configurable backup manager that resides on every {product-name} server and performs both backup and restore functions. You do not have to stop the {product-name} server in order to run the backup process.
This chapter describes how data is backed up and restored and how to use the CLI tools to backup or restore your {product-name} mailbox server. In addition, this chapter also provides information and general guidelines for disaster recovery.
{product-name} includes a configurable backup manager that resides
on every {product-name} server and performs both backup and
restore functions. You do not have to stop the {product-name}
server in order to run the backup process. The backup manager can be
used to restore a single user, rather than having to restore the entire
system in the event that one user’s mailbox becomes corrupted. Full and
incremental backups are saved in /opt/zimbra/backup.
Each Zimbra mailbox server generates redo logs that contain current and archived transactions processed by the message store server since the last incremental backup.
When the server is restored, after the backed up files are fully restored, any redo logs in the archive and the current redo log in use are replayed to bring the system to the point before the failure.
When the current redo log file size reaches 100MB, the current redo log rolls over to an archive directory. At that point, the server starts a new redo log. All uncommitted transactions from the previous redo log are preserved. In the case of a crash, when the server restarts, the current redo logs are read to re-apply any uncommitted transactions.
When an incremental backup is run, the redo logs are moved from the archive to the backup directory.
Two backup methods are available:
-
The standard backup method is appropriate for enterprise deployments where full backups are run during non-working days.
-
The auto-grouped backup method is recommended for large {product-name} environments where running a full backup of all accounts at one time would take too long.
The standard backup method runs a weekly full backup and daily incremental backups. A full backup process backs up all the information needed to restore mailboxes, including the LDAP directory server, database, index directory, and message directory for each mailbox.
When backing up shared messages, if a file representing a message already exists in the backup, it flags this object as such and does not copy its content again.
An incremental backup process backs up the LDAP data and gathers all the redo logs written since the last incremental backup. If the incremental backup process finds no previous full backup for a mailbox, a full backup is performed on that mailbox.
Incremental backups move the redo logs to the backup directory. The redo logs are a journal of every activity that has taken place. They contain a full copy of all messages delivered, as well as metadata such as tags, contacts, and conversations.
These backup files can be used to restore the complete mailbox server or individual mailboxes so that account and message data is completely restored.
The LDAP directory is backed up as part of either the full or incremental backup process. All accounts, domains, servers, COS, and other data are backed up.
Each mailbox server generates redo logs that contain every transaction processed by that server. If an unexpected shutdown occurs to the server, the redo logs are used for the following:
-
To ensure that no uncommitted transactions remain, the server reads the current redo log upon startup and re-executes and completes any uncommitted transactions.
-
To recover data written since the last full backup in the event of a server failure.
When the server is restored, after the backed up files are fully restored, any redo logs in the archive and the current redo log in use are replayed to bring the system to the point before the failure.
The Zimbra MTA is not backed up, as the data is only on the server for a very short time.
Custom configurations — such as mailboxd’s `jetty/etc/*.xml — are not backed up.
A backup report is sent to the admin mailbox when full and incremental backups are performed. This report shows the success or failure of the backup and includes information about when the backup started and ended, the number of accounts backed up and redo log sequence range.
If the backup failed, additional error information is included.
The auto-grouped backup method runs a full backup for a different group of mailboxes at each scheduled backup. The auto-grouped backup method is designed for very large {product-name} environments where backing up all accounts can take a long time. Because auto-grouped backups combine full and incremental backup functions, there is no need for incremental backups. Each auto-grouped session runs a full backup of the targeted group of mailboxes. It is not recommended to run auto-grouped backups manually since they are scheduled from the CLI and run automatically at the scheduled times.
The backup destination is known as a backup target. To the backup
system, it is a path in the file system of the mail server. The Zimbra
default backup directory is /opt/zimbra/backup.
The backup directory structure created by the standard backup process is shown in Standard Backup directory structure. You can run regularly scheduled backups to the same target area without overwriting previous backup sessions.
The accounts.xml file lists all accounts that are in all the backups combined. For each account, this file shows the account ID, the email address, and the label of the latest full backup for that account. If you save your backup sessions to another location, you must also save the latest accounts.xml file to that location. The accounts.xml file is used to look up the latest full Backup for an account during restore. If the accounts.xml file is missing you must specify the backup label to restore from.
The redo log directory is located at /opt/zimbra/redolog/redo.log. When
the current redo log file size reaches 100MB, the current redo log rolls
over to an archive directory, /opt/zimbra/redolog/archive. At this
point the server starts a new redo log. All uncommitted transactions
from the previous redo log are preserved. In the case of a crash, when
the server restarts, the current redo logs are read to re-apply any
uncommitted transactions.
Redo operations are time critical, therefore a directory move is
performed instead of a copy-then-delete function. This directory move
can only be performed if the source and destination paths are on the
same file system volume. In other words, the redo log and redo-archive
log must be on the same file system volume because the archive files are
a subdirectory of the redo log file system.
All incremental and auto-grouped backup sessions must be saved to the same directory as all the redo logs must be found in the same backup target. Standard full backup sessions can use a different target directory.
|
Default root of backups |
|
List of all accounts, each with email file address, Zimbra ID, and latest full backup label. The accounts.xml maintains the mapping of email addresses to their current zimbraIds and also the most recent full backup for each account. |
|
Root of backup sessions. |
|
A full backup directory. The timestamp for a session is the backup start time in GMT, including milliseconds. GMT is used rather than local time to preserve visual ordering across daylight savings transitions. |
|
Metadata about this backup label for a full or incremental session, such as start and stop times. |
|
Contains message files that are shared among accounts in this backup. |
|
Global database tables and |
|
Database schema information for global tables. Each table dump file has a
|
|
Copy of |
|
Database table data dump. |
|
LDAP dumps. |
|
Each account’s data is saved in a subdirectory of this. |
|
Root for each account. |
|
Metadata about this account’s backup. |
|
Account’s LDAP information, including aliases, identities, data sources, distribution lists, etc. |
|
If this is present, this files links to ldap.xml of the most recent incremental backup. |
|
Account-specific database table dumps. |
|
Database schema information for this account’s tables. |
|
Database table data dump. |
|
Contains blob files. |
|
Contains Lucene index files. |
|
An incremental backup directory. This directory is similar to the full backup directory schema and includes these metafiles. |
|
|
|
|
|
|
|
Note
|
For auto-grouped backups, the directory structure saves the redo log files to the full backup session. There are no incremental backup sessions. |
Many of the backup and restore procedures can be run directly from the Administration Console. In the Navigation pane, Monitoring>Backup lists each of the servers.
Backups can be configured from the Administration Console as a global settings configuration and as a server-specific configuration. Server settings override global settings.
In the global settings, you can configure the email addresses to receive notification about the results of the backup. The default is to send the notification to the admin account.
For Auto-grouped, you configure the number of groups to divide the backups into.
The standard backup is the default and is automatically scheduled. You do
not need to make any additional changes. But when running the auto-grouped
backup you must manually configure the backup schedule. To do so, access
the CLI and follow the steps under
Schedule Auto-Group Backups
to run zmschedulebackup -D to set the default
schedule for auto-grouped backups.
Throttling option with auto-grouped backup. The auto-grouped backup method automatically backs up mailboxes that have never been backed up when the next backup is scheduled. This might not be the best option every time a full backup is required on all mailboxes, such as immediately after massive mailbox migrations or after a major upgrade. Enabling Throttle automatic backups limits the mailbox count in a daily backup to T/N. This breaks the constraint of backing up all mailboxes in N days, but it helps backup to finish during off hours.
When all mailboxes are backed up at least once, disable throttling:
zmprov mcf zimbraBackupAutoGroupedThrottled TRUEThe Zimbra backup and restore procedures can be run as CLI commands.
The following utilities are provided to create backup schedules, perform full and incremental backups, restore the mail server, or restore the LDAP server.
-
zmschedulebackup— This command is used to schedule full backups, incremental backups, and deletion of old backups. -
zmbackup— This command executes full or incremental backup of the {product-name} mailbox server. This is run on a live server, while themailboxdprocess and the mailbox server are running. This command also has an option to manually delete old backups when they are no longer needed. -
zmbackupabort— This command stops a full backup that is in process. -
zmbackupabort -r— This command stops an ongoing restore. -
zmbackupquery— This command lists the information about ongoing and completed backups, including labels and dates. -
zmrestore— This command restores a backup to a running {product-name} mailbox server. -
zmrestoreoffline— This command restores the {product-name} mail server when themailboxdprocess is stopped. -
zmrestoreldap— This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data.
Refer to Appendix A: Command Line Utilities for usage and definitions for each of these commands.
When you initiate a backup, you can issue the command from the same server being backed up, run the command remotely and specify the target server on the command line, or use the Administration Console to start a backup session.
When {product-name} was installed, the backup schedule for the standard method of full and incremental backups was added to the crontab. Under the default schedule, the full backup is scheduled for 1:00 a.m., every Saturday. The incremental backups are scheduled for 1:00 a.m., Sunday through Friday.
By default, backups older than a month are deleted every night at 12 a.m.
You can change the backup schedule using the zmschedulebackup command.
Specify the fields as follows, separate each field with a blank space:
-
minute — 0 through 59
-
hour — 0 through 23
-
day of the month — 1 through 31
-
month — 1 through 12
-
day of the week — 0 through 7 (0 or 7 is Sunday, or use names)
Type an asterisk (*) in the fields you are not using.
zmschedulebackup options-
Replace the existing full backup, incremental backup and delete backup schedule. When you use
-R, the complete backup schedule is replaced. If you use this command, remember to set the deletion schedule, if you want backup sessions to be scheduled for automatic deletion. This example replaces the existing schedule to have full backups run on Sunday at 1 a.m., incremental backups to run Monday through Saturday at 1 a.m., and old backups deleted at 12:00 a.m. every day.zmschedulebackup -R f "0 1 * * 7" i "0 1 * * 1-6" d 1m "0 0 * * *"
-
Add an additional full backup time to your current schedule. This example adds a full backup on Thursday at 1 a.m.
zmschedulebackup -A f "0 1 * * 4" -
Review your backup schedule. The schedule is displayed.
zmschedulebackup -q
-
Save the schedule command to a text file. This would allow you to easily recreate the same schedule after a reinstall or upgrade
zmschedulebackup -s
|
Note
|
To return backups to the default schedule use the command zmschedulebackup -D.
|
The default backup schedule is displayed similarly to the following example:
0 1 * * 6 /opt/zimbra/bin/zmbackup -f - all
0 1* * 0-5 /opt/zimbra/bin/zmbackup -i
0 0 * * * /opt/zimbra/bin/zmbackup -del 1mRead as follows:
0 1 * * * 6 /opt/zimbra/bin/zmbackup -f - all0 1* * 0-5 /opt/zimbra/bin/zmbackup -i0 0 * * * /opt/zimbra/bin/zmbackup -del 1mEach crontab entry contains six fields that appear in this order:
| Field | |||||
|---|---|---|---|---|---|
1 |
2 |
3 |
4 |
5 |
6 |
0 |
1 |
* |
* |
6 |
|
-
minute (0-59 allowed)
-
hour (0-23)
-
day of the month (1-31)
-
month (1-12 or names)
-
day of the week (0-7 or names allowed, with both 0 and 7 representing Sunday
-
string to be executed
|
Note
|
The asterisk character works as a wild card, representing every occurrence of the field’s value. |
- Admin Console:
-
Home → Configure → Global Settings → Backup/Restore
You can add additional recipient addresses or change the notification email address in the Administration Console.
The full backup process goes through the following steps to back up the mailbox, the database, the indexes, and the LDAP directory:
-
Backs up the global system data including system tables and
localconfig.xml. -
Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
-
Places the account’s mailbox in maintenance mode to temporarily block mail delivery and user access to that mailbox.
-
Backs up the mailbox.
-
Creates MariaDB dump for all data related to that mailbox.
-
Backs up the message directory for that mailbox.
-
Creates a backup of the index directory for that mailbox.
-
-
Returns that account’s mailbox to active mode and moves on to the next one.
-
Backs up the LDAP directory.
A full backup is usually run asynchronously. When you begin the full
backup, the label of the ongoing backup process is immediately
displayed. The backup continues in the background. You can use the
zmbackupquery command to check the status of the running backup at any
time.
Backup files are saved as zip files without compression. To change the
default zip option, see Appendix A: Command Line Utilities,
zmbackup section.
Incremental backups are run using the CLI command, zmbackup. The process for incremental backup is as follows:
-
Backs up the global system data including system tables and
localconfig.xml. -
Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
-
Moves the archive redo logs, created since the last backup, to the
<backup_target>/redologsfolder.Archived logs that are less than an hour old at the time of incremental backup are copied to the backup and are not deleted. These redologs are deleted one hour after the backup. The interval is set by the
localconfigkeybackup_archived_redolog_keep_time. The default is 3600 seconds.If no full backup for this account is found, the backup process performs a full backup on this account, even if only an incremental backup was specified.
-
Backs up the LDAP directory.
Use the zmbackup command to perform the following backup operations:
-
Perform a manual backup of all mailboxes on server<1>:
zmbackup -f -s server1.domain.com -a all
-
Perform a manual, incremental backup of all mailboxes on server1 since last full backup
zmbackup -i -s server1.domain.com -a all
-
Perform a manual, full backup of only user1’s mailbox on server1
zmbackup -f -s server1.domain.com -a [email protected]
You can delete backup sessions either by label or by date.
-
Deleting by label deletes that session and all backup sessions before that session.
-
Deleting by date deletes all backup session prior to the specified date.
For example, zmbackup -del 7d deletes backups older than 7 days
from now. You can specify day (d), month (m), or year (y).
Each full or incremental backup is a backup session.
Each backup session is labeled with date and time. For example, the
label full-20210712.155951.123 says this is a backup from July 12,
2021 at 3:59:51.123.
|
Note
|
The times set in the session label are GMT, not the local time. GMT is used rather than local time to preserve visual ordering across daylight savings transitions. |
Use the zmbackupquery command to find full backup sessions.
-
To find a specific full backup session:
zmbackupquery -lb full-20210712.155951.123
-
To find a full backup sessions since a specific date:
zmbackupquery --type full --from "2021/01/01 12:45:45"
-
To find all full backup sessions in the backup directory:
zmbackupquery --type full
-
To find the best point in time to restore for an account specify a time window
zmbackupquery -a [email protected] --type full --from "2021/07/05 12:01:15" --to "2021/07/12 17:01:45"
|
Note
|
If a backup session is interrupted because the server crashes during backup
(not aborted), the interrupted backup session is saved as a temporary
session. The temporary backup session is put in
<backup_target>/sessions_tmp folder. You can use the rm command to delete the
directory.
|
-
Before you can abort a backup, you must know the backup session label. This label is displayed when
zmbackupfirst starts. If you do not know the full backup label, usezmbackupqueryto find the label. -
Use the
zmbackupabortcommand to stop a backup that is in progress. The backup is immediately stopped and becomes a partially successful backup.-
Stop the backup, if you know the label name
-
zmbackupabort -lb full-20210712.155951.123 -s server1
-
Stop the backup, if you do not know the label
zmbackupquery zmbackupabort -s server1 -lb full-20210712.155951.123
The auto-grouped backup method is configured either from the Administration Console or from the CLI.
- Admin Console:
-
Home → Configure → Global Settings → Backup/Restore or
Home → Configure → Servers → server → Backup/Restore
Set the backup method in the global configuration, and you can override the configuration on a per server basis if you do not want a particular server to use the auto-grouped backup method.
To set up auto-grouped backup, you modify LDAP attributes with the zmprov command:
zmprov mcf <ldap_attribute> <arg>You can also set the attributes at the server level using zmprov ms.
The following LDAP attributes are modified:
-
zimbraBackupMode— Set it to be Auto-Grouped. The default is Standard. -
zimbraBackupAutoGroupedInterval— Set this to the interval in either days or weeks that backup sessions should run for a group. The default is1d. Backup intervals can be 1 or more days, entered asxd(e.g.1d); or 1 or more weeks, entered asxw(e.g.1w). -
zimbraBackupAutoGroupedNumGroups— This is the number of groups to spread mailboxes over. The default is 7 groups.
You must configure the auto-group backup schedule.
Run zmschedulebackup -D to set the default schedule for
auto-grouped backups based on your zimbraBackupAutoGroupedInterval
setting.
One group is backed up each interval. The auto-grouped backup automatically adjusts for changes in the number of mailboxes on the server. Each backup session backs up the following:
-
All mailboxes that have never been backed up before. These are newly provisioned mailboxes.
-
All mailboxes that have not been backed within the number of scheduled backup days. For example, if backups are scheduled to run over six days, mailboxes that have not been backed up in the past 5 days are backed up.
-
More mailboxes, the oldest backup first. This is done so that the daily auto-grouped backup load is balanced.
For example, if you configured the auto-grouped backup interval to be daily (1d) and the number of groups to be 7, the first time auto-grouped backup runs, all accounts are backed up. After the initial backup, auto-grouped backup runs again the next day. This time accounts that have been newly provisioned and a percentage of accounts close to one-seventh of the total are backed up again. Accounts with the oldest backup date are backed up first. The backup continues with newly provisioned accounts and approximately one-seventh of accounts being backed up daily over seven days.
When backing up shared messages, if a file representing a message already exists in the backup, it flags this object as such and does not copy its content again.
Backup files are saved as zip files without compression. To change the
default zip option, see Appendix A: Command Line Utilities,
zmbackup section.
These backup files can be used to restore the complete {product-name} system or individual mailboxes so that account and message data is completely restored. Archived redo logs are moved to the backup session as part of the full backup. When the server is restored from an auto-grouped backup, redo logs are replayed to bring the system to the point before the failure.
The backup process can be configured to selectively backup content and to back up the MariaDB database.
You can configure these backup options so that search indexes, blobs, and HSM blobs are not backed up during a full backup session.
-
zimbraBackupSkipSearchIndex— Default is FALSE. If set to TRUE, the search index is not backed up. The mailbox will have to be reindexed after restoring from a backup without the search index. -
zimbraBackupSkipBlobs— Default is FALSE. If this is set to TRUE, blobs are not backed up. This might be useful for getting a quicker backup of just database data when the blobs reside on fault-tolerant storage. This configuration applies to all blobs, those on the primary volumes as well as secondary (HSM) volumes. -
zimbraBackupSkipHsmBlobs— Default is FALSE. If this is set to TRUE, blobs on HSM volumes are not backed up. Set this ifzimbraBackupSkipBlobsis FALSE but you want to skip blobs on HSM volumes.
You can configure {product-name} backups to run mysqldump to
back up your MariaDB database during backup sessions. When this is enabled,
a mysqldump backup runs with each full, incremental, and auto-grouped
backup.
The mysqldump is a backup of your MariaDB database at a specific time.
Data changes that occur later than the dump file are written to the
binary log. To recover to a specific point in time, binary logging must
be enabled. See the Zimbra wiki article, MariaDB Backup and Restore at
https://wiki.zimbra.com/wiki/MySQL_Backup_and_Restore.
The MariaDB dump files are gzipped and placed in the backup target
directory, or to /opt/zimbra/backup, if no directory is specified.
These files can be quite large. Make sure that the free disk space is at least three times greater than the actual MariaDB database file for each MariaDB database backup file that is saved.
-
Enable
mysqldumpto run automatically with your backups, type
zmlocalconfig edit mysql_backup_retention=<N>
N is the number of copies of the MariaDB database backups that are retained.
|
Note
|
To restore a MariaDB database, contact Zimbra support for assistance. |
Backup sessions fail if the target disk does not have enough space. All data backed up in the backup session is discarded and deleted.
You can choose to receive notification when your disk might not have enough space to complete the backup
Configuring the zimbraBackupMinFreeSpace attribute helps you
manage running backup session by notifying you.
Set the value for attribute zimbraBackupMinFreeSpace to the amount of
free space required on the backup target disk before a backup session is
run. If the disk has less space than the value set in the attribute, the
back up session will not run and an email notification is sent to the
administrator.
|
Note
|
If you are also backing up the MariaDB database, make sure you set the value
large enough to include the myslqdump file size.
|
The value for this attribute can be specified as a percentage of the
total disk space, for example 25%, or as a number of bytes, for example
300MB, 50GB, etc. The default value is 0, meaning the check is disabled
and backup is always allowed to start.
The attribute can be set globally or by server.
-
As global:
zmprov mcf zimbraBackupMinFreeSpace <value>
-
By server:
zmprov ms <zmhostname> zimbraBackupMinFreeSpace <value>
Backup sessions run if the free disk space is at least the value you set. If your backup file is larger than the value, the backup session fails. You should monitor the size of the backup files and adjust the attribute value if the backup requires more space than the configured value.
Three types of restore procedures can be run:
-
The
zmrestorecommand is used to restore the mailboxes while the {product-name} mailbox server is running. -
The
zmrestoreofflinecommand is used to restore the mailbox server when just themailboxdprocess it is stopped. This command is run for disaster recovery. -
The
zmrestoreldapcommand is used to restore the content of the LDAP directory server.
The restore process allows all accounts or individual accounts to be specified.
The zmrestore process goes through the following steps to restore the mailbox, the database, the indexes, and the LDAP directory.
-
Retrieves specified accounts to be restored, or specify all for all accounts that have been backed up.
-
Iterates through each mailbox:
-
Deletes the mailbox on the server to clear any existing data
-
Restores the last full backup of the MariaDB data, the index directory, and the message directory for that mailbox
-
Replays redo logs in all incremental backups since last full backup
-
Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server
-
Replays the current redo log
-
|
Note
|
Accounts are restored, even if the account exceeds its quota. The next time the user performs an action that affects quota, they receive a warning that they have exceeded their quota. |
|
Important
|
Users using the Zimbra Connector for Microsoft Outlook must perform an initial sync on the Outlook client when they log on after the Zimbra server is restored. |
Examples
Including last full backup and any incremental backups since last full backup
zmrestore -a allzmrestore -a [email protected]The following restore options affect redo log replay. If you do not specify one of these options, all redo logs since the full backup you’re restoring from are replayed
|
Important
|
After you perform any of the following point-in-time restores, you should immediately run a complete backup for those accounts to avoid future restore problems with those accounts. |
A restore that is run using any of the following options is a point-in-time restore:
-
-restoreToTime<arg> - Replay the redo logs until the time specified. -
-restoreToIncrLabel<arg> - Replay redo logs up to and including this incremental backup. -
-restoreToRedoSeq<arg> - Replay up to and including this redo log sequence. -
-br- Replays the redo logs in backup only, therefore excluding archived and current redo logs of the system. -
-rf- Restores to the full backup only. This does not include any incremental backups at all.
Restore stops at the earliest possible point in time if more than one point in time restore options are specified.
zmrestore -a [email protected] <arg>Two common ways to write the <timearg> are
-
"YYYY/MM/DD hh:mm:ss" -
YYYYMMDD.hhmmss
zmrestore -rf --a allzmrestore -ra -a [email protected]A prefix is prepended to the original account names
zmrestore -ca -a [email protected] -pre restoreThe result from the above example would be an account called [email protected].
localconfig.xmlzmrestore -sys--contineOnError (-c) to the command so that the restore process continues if an error is encountered.zmrestore -a all -cWhen -c is designated, accounts that could not be restored are displayed
when the restore process is complete.
Can also be used to restore deleted accounts
zmrestore -a [email protected]zmrestore -a [email protected] -skipDeletedAccountsWhen the mailbox is restored it will contain messages that were deleted. This is useful if users use POP and remove messages from the server
zmrestore -a [email protected] --skipDeletes|
Note
|
When the latest point in time is requested, do not add a backup label
(-lb). Without specifying the label, the most recent full backup before the
requested point is automatically used as the starting point.
|
The zmbackupabort -r command interrupts a restore that is in process.
The restore process stops after the current account finishes being
restored. The command displays a message showing which accounts were not
restored.
To stop the restore type:
zmbackupabort -rThe offline restore process can only be run when the$1`mailboxd` server is not running. In general, offline restore is run under the following circumstances:
-
Certain components of the Zimbra server are corrupted, and the server cannot be started. For example, the data in LDAP or the database are corrupted.
-
A disaster requires the Zimbra software to be reinstalled on the server.
The offline restore must be run before the {product-name} mailbox store server is started to keep the redo logs in sequence.
In a disaster recovery when the Zimbra software is reinstalled, if mailboxd
is started before the backup files are restored, the mail server would
begin to accept email messages and perform other activities, producing redo
logs in the process. Since the pre-disaster data have not been restored to
the server, the redo logs would be out of sequence. Once mailboxd is
running, it would be too late to restore the pre-disaster data.
The offline restore process goes through the following steps.
-
Specified accounts to be restored are retrieved. If the command-line does not specify any mailbox address, the list of all mailboxes on the specified mail host is retrieved from Zimbra LDAP directory server.
-
Iterates through each mailbox:
-
Deletes the mailbox on the server to clear any existing data
-
Restores the last full backup of the MariaDB data, the index directory, and the message directory for that mailbox
-
Replays redo logs in all incremental backups since last full backup
-
Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server
-
Replays the current redo log
-
Use the zmrestore command to restore one or more selected accounts. In the event that a user’s mailbox has become corrupted, you might want to restore that user from the last full and incremental backup sets.
-
For each account to be restored, put the account into maintenance mode
zmprov ma <account> zimbraAccountStatus maintenance
Maintenance mode prevents the delivery of new emails during the restore. Otherwise, the emails would be overwritten during the restore process.
-
Run the
zmrestorecommand to restore the accountszmrestore -a [email protected]
-
For each account that was restored, put the account back into active mode
zmprov ma <account> zimbraAccountStatus active
|
Important
|
If a user account is restored and the COS that the account was assigned no longer exists, the default COS is assigned to the account. |
When you restore from a full backup, you can exclude the search index and blobs.
-
Search index — If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
zmrestore -a <all|account> --exclude-search-index
-
Blobs — This is a useful option when all blobs for the mailbox being restored already exist.
zmrestore <all or account>|--exclude-blobs
-
HSM-blobs — This is useful when all HSM blobs for the mailbox being restored already exist.
zmrestore <all or account> --exclude-hsm-blobs
In a disaster recovery where you need to restore the entire system, restore the LDAP directory server first.
The zmrestoreldap command restores the global LDAP data including COS,
distribution lists, etc. You can restore the complete LDAP server, which
recreates the entire schema or you can restore specific accounts. You
specify the session to restore. The restore command has to be run on the
LDAP server being restored.
Examples
zmrestoreldap -lbszmrestoreldap -lb full20211130135236zmrestoreldap -lb full20211130135236 -a [email protected] [email protected]Use the following steps to restore a mailbox store server in a general disaster scenario involving multiple machines.
-
Restore the LDAP directory server to a known good state before doing anything with the mailbox store server.
-
Put all mailboxes into maintenance mode to prevent mail delivery and user login while restoring the mailboxes.
-
Stop the mailbox store server if it is running.
-
Reinstall the {product-name} software on the mailbox server, if necessary.
-
Restore mailboxes.
-
Start the {product-name} server.
-
Put all {product-name} mailboxes back in active mode.
-
Run a full backup of the server.
When your system unexpectedly stops and then restarts on startup, the server searches the redo log for uncommitted transactions and replays any that it finds. Replaying the redo logs brings the system to a consistent state.
If a complete machine failure occurs, use the following steps to restore to a new server.
|
Important
|
The {product-name} version you install on the new server must be the same version as installed on the old server. The server can have a different operating system. |
The new server hardware must meet the requirements described in the Installation Prerequisites section of the {product-name} Single Server Installation guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide.
You do the following to restore to a new server:
-
Prepare the new server.
-
Block client access to the old server’s IP address with firewall rules.
-
Mount any volumes that were in use on the older server.
-
Delete the MariaDB data that is set up in the initial installation of {product-name}.
-
Copy the backup files to the new server.
-
Run
zmrestoreldapto restore the global LDAP data. -
Run
zmrestoreofflineto restore account data from the backup sessions. -
Prepare and run a new backup.
Two scenarios for disaster recovery are the server has died and the {product-name} files cannot be accessed, or {product-name} is still running, but the server hardware needs to be replaced.
If the server is not running:
-
Block client access to the server IP address with firewall rules.
-
Find the latest full {product-name} backup session to use.
If the server is still running, to prepare the move to the new server:
-
Block client access to the server’s IP address with firewall rules.
-
Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.
-
Run
zmcontrol stop, to stop {product-name}. In order to restore to the most current state, no new mail should be received after the last incremental backup has run. -
Change the hostname and IP address on the old server to something else. Do not turn off the server.
Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that {product-name} is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the {product-name} installation guide for more information about preparing the server. Before you begin to install {product-name}, note the information you need from the old server including admin account name and password, LDAP, Amavis, and Postfix passwords, spam training, and non-spam training user account names, exact domain name, and the global document account name.
|
Note
|
Make sure the computer time is set to the same time as the old server. Verify that the old hostname and MX DNS records resolve to the new server. |
-
Copy the {product-name} License.xml file to a directory on the new server. You cannot complete the {product-name} installation if the license is not on the new server.
-
Run
./install.shand follow the directions in the installation guide to install {product-name}. Make sure that you configure the same domain, hostname, passwords as on the old server. During Zimbra Collaboration install, the following settings must be changed to match the original server settings:-
Zimbra LDAP Server — For Domain to create, identify the same default domain as on the old server.
-
Zimbra Mailbox Server — An administrator’s account is automatically created.
-
Make sure that the account name for Admin user to create is the same name as on the original server.
-
Set the admin password to be the same as on the old server.
-
Set the LDAP password to be the same as on the old server.
-
Set the Postfix user and Amavis user passwords to be the same as on the old server
-
Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the old server.
-
Global Document Account — This account name is automatically generated and is usually named wiki. If you changed this, change the Global Document Account name to be the same account name as on the original server.
-
-
Change any other settings on the new server to match the configuration on the original server.
-
In the main menu, set the default backup schedule and the automatic starting of servers after the configuration is complete to NO.
-
-
Stop the new server
zmcontrol stop
-
If the old server had additional storage volumes configured, mount the additional volumes now.
-
Delete the MariaDB data and reinitialize an empty data directory. If you do not do this,
zmrestoreofflinewill have errors. Aszimbra, type:rm -rf /opt/zimbra/db/data/* /opt/zimbra/libexec/zmmyinitThe MariaDB service is now running.
-
Copy all the files in the
/backupfolder from the old server or from an archive location to/opt/zimbra/backup. -
Restore the LDAP.
zmrestoreldap -lb <latest_label>
If you are restoring a large number of accounts, you might run a command such as the UNIX command,
nohup, so that the session does not terminate before the restore is complete.NoteTo find the LDAP session label to restore, type zmrestoreldap –lbs. -
Ensure that the following services are running before attempting to execute
zmrestoreoffline.-
mysqld(MariaDB) -
slapd(OpenLDAP)zmcontrol start
-
-
Ensure that the following services are stopped before attempting to execute
zmrestoreoffline.-
mailboxdzmmailboxdctl stop
-
Because some {product-name} services are running at this
point, type zmconvertctl start. This is required before running
zmrestoreoffline.
-
Sync your LDAP password from backup directory to the new production servers LDAP config.
zmlocalconfig -f -e zimbra_ldap_password=<password>
-
Start the offline restore after stopping
mailboxd.zmmailboxdctl stop zmrestoreoffline -sys -a all -c -br
You might run a command such as
nohuphere also. To watch the progress, tail/opt/zimbra/log/mailbox.log.NoteUse –con the command line so that accounts are restored, even if some accounts encounter errors during the offline restore process. -
Because some {product-name} services are running at this point, type
zmcontrol stopto stop all services. -
Remove any old backup sessions because these sessions are no longer valid.
rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*
-
Start {product-name}.
zmcontrol start
-
Run a full backup.
zmbackup -f -a all
-
Remove the firewall rules and allow client access to the new server.
The restoration steps are similar for most server failures you may encounter. If a failure occurs, review the disaster recovery section to understand the process and then follow the steps below for the specific type of failure.
-
Reinstall the LDAP server. See the {product-name} Installation guide.
-
Find the label for the LDAP session to restore. Run the
zmrestoreldap - lb <label>command, with no arguments to restore all accounts, domains, servers, COS, etc. for the LDAP server. -
Make sure that all accounts are in active mode. From the command line, type
zmprov ma zimbraAccountStatus active
-
If a partition becomes corrupted, replace the failed disk.
-
To restore the latest full and incremental backup files, run
zmrestore -a all
The zmrestore process automatically retrieves the list of all mailboxes on the specified mail host from the backup date and iterates through each mailbox to restore the mailboxes to the last known good state.
If the redo log becomes unreadable, the mailboxd service stops and
cannot restart. If this happens, inspect the hardware and software to
find the source of the problem before proceeding.
Without the latest redo log, the Zimbra mailbox server cannot be returned to the most current state. The Zimbra mailbox data can be restored to the latest archived redo log state. A new redo log for current transactions is created after the Zimbra mailbox server is restored.
|
Important
|
The mailboxd service must not be running, and all accounts must be in
maintenance mode before beginning.
|
-
Put all accounts into maintenance mode.
zmprov md <domain> zimbraDomainStatus maintenance
-
With the
mailboxdservice not running, typezmrestoreoffline
The offline restore process begins by retrieving the list of all mailboxes on the specified mail host from the backup.
The offline restore than iterates through each mailbox to:
-
Delete the mailboxes on the server
-
Restore the last full backup from the backup area
-
Restore all incremental backups for that mailbox in order, since the last full backup. This involves replaying the redo logs from the backup target area
-
Replay all archived redo logs
Because the redo log for current transactions is not available, the mailbox server is returned to the state of the last archived redo log.
-
-
After the offline restore is complete, start {product-abbrev}.
zmcontrol startup
-
When the Zimbra mailbox server is up, run a full backup of the Zimbra server. The full backup must be run immediately to have the latest data backed up because the latest redo log is not available.
The localconfig.xml file, located in the /opt/zimbra/conf folder,
includes the core Zimbra server configuration, such as paths and passwords,
This file is backed up in full and incremental backups. When you run an
incremental or full restore, the backed-up version of the localconfig.xml
is renamed localconfig.xml.restore and is copied to the /opt/zimbra/conf
directory.
If you have made changes since the last backup, you might need to replace
the localconfig.xml file with the restored copy. Compare these files, and
if the .restore file has the latest local configuration data, delete the
localconfig.xml file and rename the file with the .restore extension to
localconfig.xml.
You can backup and restore a server using the snapshot feature provided by the storage layer rather than using Zimbra’s backup and restore feature. Using snapshots, you can maintain a standby site and reroute users to the standby site to keep operations running if the primary site fails.
Snapshots are taken for all volumes of data and are transferred to the
standby site periodically. Data volumes that are backed up using
snapshots include MariaDB, blobs, Lucene index, and redologs.
When the primary site is down, the zmplayredo command is used to bring
consistency to the snapshots and to reapply any changes in data to
minimize data loss across volumes
There are four volumes of data:
-
MariaDB
-
Blob
-
Lucene index
-
Redologs
Sets of snapshots are taken every hour and transferred to the remote standby site. However, all snapshots are not taken at one instant and could be a second to a minute apart from each other. Also, snapshots of redologs may be taken more frequently. The sequence of events could look like:
8:00:00 - snapshot mysql 8:00:01 - snapshot blob 8:00:02 - snapshot index 8:00:03 - snapshot redolog 8:05:00 - transfer the snapshot set to remote site completed ... 8:15:00 - snapshot redolog 8:15:05 - transfer of redolog snapshot to remote site completed ... 8:30:00 - snapshot redolog 8:30:05 - transfer of redolog snapshot to remote site completed ... 8:35:00 - primary site fails
On the remote site, there are snapshots from the 8:00 set of data as well as subsequent snapshots of the redologs. They all have to be brought together so that the most recent information is available on the standby site once users are rerouted to it.
You can now run the zmplayredo command to replay changes from 8:00:00.
zmplayredo --fromTime "2021/10/17 08:00:00:000"All data is brought forward to the current time and the standby site is set up and running. Data from 8:30:00 to 8:35:00 is lost but that is expected when the restore process is being carried out.
As of ZCS 8.8, ephemeral data is not backed up as part of the backup process. Since auth tokens are ephemeral attributes, the implication is that clients accessing accounts restored after deletion will need to re-authenticate; auth tokens generated prior to the backup will no longer work.
If SSDB is used as the ephemeral backend, a backup will not include any ephemeral attributes.
Note: This section does not detail how to deploy and administer an SSDB server. For that information please see section SSDB Configuration Options.
Backing up the data stored in SSDB (if so configured) is done as follows:
ssdb-dump -h localhost -p 8888 -o /tmp/ephemeral-backup-<date>Note: If running in master / slave configuration the ssdb-dump should be run on the master.
ssdb-dump - SSDB backup command
Copyright (c) 2012-2015 ssdb.io
recv begin...
received 1 entry(s)
received 10 entry(s)
received 100 entry(s)
received 1000 entry(s)
received 10000 entry(s)
received 100000 entry(s)
received 200000 entry(s)
received 300000 entry(s)
received 400000 entry(s)
received 400021 entry(s)
recv end
total dumped 400021 entry(s)
Compactions
Level Files Size(MB) Time(sec) Read(MB) Write(MB)
--------------------------------------------------
2 1 7 0 0 7
compacting data...
Compactions
Level Files Size(MB) Time(sec) Read(MB) Write(MB)
--------------------------------------------------
2 2 10 0 0 10
backup has been made to folder: /tmp/ephemeral-backup-<date>Restoring ephemeral data to SSDB from a backup can only be done with a backup from an SSDB server.
Restoration can be done in one of two ways:
-
import into a running server
-
override of existing data