Skip to content

Latest commit

 

History

History
1570 lines (1259 loc) · 50.9 KB

backuprestore.adoc

File metadata and controls

1570 lines (1259 loc) · 50.9 KB

Backup and Restore

Table of Contents

{product-name} includes a configurable backup manager that resides on every {product-name} server and performs both backup and restore functions. You do not have to stop the {product-name} server in order to run the backup process.

This chapter describes how data is backed up and restored and how to use the CLI tools to backup or restore your {product-name} mailbox server. In addition, this chapter also provides information and general guidelines for disaster recovery.

Backing Up the Mailbox Server

{product-name} includes a configurable backup manager that resides on every {product-name} server and performs both backup and restore functions. You do not have to stop the {product-name} server in order to run the backup process. The backup manager can be used to restore a single user, rather than having to restore the entire system in the event that one user’s mailbox becomes corrupted. Full and incremental backups are saved in in opt/zimbra/backup.

Redo Log

Each Zimbra mailbox server generates redo logs that contain current and archived transactions processed by the message store server since the last incremental backup.

When the server is restored, after the backed up files are fully restored, any redo logs in the archive and the current redo log in use are replayed to bring the system to the point before the failure.

When the current redo log file size reaches 100MB, the current redo log rolls over to an archive directory. At that point, the server starts a new redo log. All uncommitted transactions from the previous redo log are preserved. In the case of a crash, when the server restarts, the current redo logs are read to re-apply any uncommitted transactions.

When an incremental backup is run, the redo logs are moved from the archive to the backup directory.

Backup Methods

Two backup methods are available:

  • The standard backup method is appropriate for enterprise deployments where full backups are run during non-working days.

  • The auto-grouped backup method is recommended for large {product-name} environments where running a full backup of all accounts at one time would take too long.

Standard Backup

The standard backup method runs a weekly full backup and daily incremental backups. A full backup process backs up all the information needed to restore mailboxes, including the LDAP directory server, database, index directory, and message directory for each mailbox.

When backing up shared messages, if a file representing a message already exists in the backup, it flags this object as such and does not copy its content again.

An incremental backup process backs up the LDAP data and gathers all the redo logs written since the last incremental backup. If the incremental backup process finds no previous full backup for a mailbox, a full backup is performed on that mailbox.

Incremental backups move the redo logs to the backup directory. The redo logs are a journal of every activity that has taken place. They contain a full copy of all messages delivered, as well as metadata such as tags, contacts, and conversations.

These backup files can be used to restore the complete mailbox server or individual mailboxes so that account and message data is completely restored.

The LDAP directory is backed up as part of either the full or incremental backup process. All accounts, domains, servers, COS, and other data are backed up.

Each mailbox server generates redo logs that contain every transaction processed by that server. If an unexpected shutdown occurs to the server, the redo logs are used for the following:

  • To ensure that no uncommitted transactions remain, the server reads the current redo log upon startup and re-executes and completes any uncommitted transactions.

  • To recover data written since the last full backup in the event of a server failure.

When the server is restored, after the backed up files are fully restored, any redo logs in the archive and the current redo log in use are replayed to bring the system to the point before the failure.

The Zimbra MTA is not backed up, as the data is only on the server for a very short time.

Custom configurations—such as mailboxd’s server.xml—are not backed up.

Backup Notifcation

A backup report is sent to the admin mailbox when full and incremental backups are performed. This report shows the success or failure of the backup and includes information about when the backup started and ended, the number of accounts backed up and redo log sequence range.

If the backup failed, additional error information is included.

Auto-Grouped Backup Method

The auto-grouped backup method runs a full backup for a different group of mailboxes at each scheduled backup. The auto-grouped backup method is designed for very large {product-name} environments where backing up all accounts can take a long time. Because auto-grouped backups combine full and incremental backup functions, there is no need for incremental backups. Each auto-grouped session runs a full backup of the targeted group of mailboxes. It is not recommended to run auto-grouped backups manually since they are scheduled from the CLI and run automatically at the scheduled times.

Directory Structure for Backup Files

The backup destination is known as a backup target. To the backup system, it is a path in the file system of the mail server. The Zimbra default backup directory is /opt/zimbra/backup.

The backup directory structure created by the standard backup process is shown in Standard Backup directory structure. You can run regularly scheduled backups to the same target area without overwriting previous backup sessions.

The accounts.xml file lists all accounts that are in all the backups combined. For each account, this file shows the account ID, the email address, and the label of the latest full backup for that account. If you save your backup sessions to another location, you must also save the latest accounts.xml file to that location. The accounts.xml file is used to look up the latest full Backup for an account during restore. If the accounts.xml file is missing you must specify the backup label to restore from.

The redo log directory is located at /opt/zimbra/redolog/redo.log. When the current redo log file size reaches 100MB, the current redo log rolls over to an archive directory, /opt/zimbra/redolog/archive. At this point the server starts a new redo log. All uncommitted transactions from the previous redo log are preserved. In the case of a crash, when the server restarts, the current redo logs are read to re-apply any uncommitted transactions.

Redo operations are time critical, therefore a directory move is performed instead of a copy-then-delete function. This directory move can only be performed if the source and destination paths are on the same file system volume. In other words, the redo log and redo -archive log must be on the same file system volume because the archive files are a subdirectory of the redo log file system.

All incremental and auto-grouped backup sessions must be saved to the same directory as all the redo logs must be found in the same backup target. Standard full backup sessions can use a different target directory.

Table 1. Standard Backup directory structure

/opt/zimbra/backup

Default root of backups

accounts.xml/

List of all accounts, each with email file address, Zimbra ID, and latest full backup label. The accounts.xml maintains the mapping of email addresses to their current zimbraIds and also the most recent full backup for each account.

sessions/

Root of backup sessions.

full-<timestamp>/

A full backup directory. The timestamp for a session is the backup start time in GMT, including milliseconds. GMT is used rather than local time to preserve visual ordering across daylight savings transitions.

session.xml

Metadata about this backup label for full or incremental session, such as start and stop times.

shared_blobs/

Contains message files that are shared among accounts in this backup.

sys/

Global database tables and localconfig.

db_schema.xml

Database schema information for global tables. Each table dump file has a .csv format.

localconfig.xml

Copy of /opt/zimbra/conf/localconfig.xml at the time of the backup.

<table name>.dat

Database table data dump.

LDAP/ldap.bak

LDAP dumps.

accounts/

Each account’s data is saved in a subdirectory of this.

<.../zimbraId>/

Root for each account.

meta.xml

Metadata about this account’s backup.

ldap.xml

Account’s LDAP information, including aliases, identities, data sources, distribution lists, etc.

ldap_latest.xml

If this is present, this files links to ldap.xml of the most recent incremental backup.

db/

Account-specific database table dumps.

db_schema.xml

Database schema information for this account’s tables.

<table name>.dat

Database table data dump.

blobs/

Contains blob files.

index/

Contains Lucene index files.

incr-<timestamp>

An incremental backup directory. This directory is similar to the full backup directory schema and includes these meta files.

session.xml

sys/db_schema.xml

accounts/..../<zimbraID>/ldap.xml

incr-<timestamp> does not include accounts/.../<zimbraId>/db/db_schema.xml because incremental backup does not dump account tables.
Note
 For auto-grouped backups, the directory structure saves the redo log files to the full backup session. There are no incremental backup sessions.

Backup and Restore Using the Administration Console

Many of the backup and restore procedures can be run directly from the Administration Console. In the Navigation pane, Monitoring>Backup lists each of the servers.

Configure Backup from the Admin Console

Backups can be configured from the Administration Console as a global settings configuration and as a server-specific configuration. Server settings override global settings.

In the global settings, you can configure the email addresses to receive notification about the results of the backup. The default is to send the notification to the admin account.

For Auto-grouped, you configure the number of groups to divide the backups into.

The standard backup is the default and is automatically scheduled. You do not need to make any additional changes. But when running the auto-grouped backup you must manually configure the backup schedule. To do so, access the CLI and follow the steps under Schedule Auto-Group Backups to run zmschedulebackup -D to set the default schedule for auto-grouped backups.

Throttling option with auto-grouped backup. The auto-grouped backup methodautomatically backs up mailboxes that have never been backed up when the next backup is scheduled. This might not be the best option every time a full backup is required on all mailboxes, such as immediately after massive mailbox migrations or after a major upgrade. Enabling Throttle automatic backups limits the mailbox count in a daily backup to T/N. This breaks theconstraint of backing up all mailboxes in N days, but it helps backup to finish during off hours.

CLI

When all mailboxes are backed up at least once, disable throttling:

zmprov mcf zimbraBackupAutoGroupedThrottled TRUE

Backup and Restore Using the Command Line Interface

The Zimbra backup and restore procedures can be run as CLI commands.

CLI

The following utilities are provided to create backup schedules, perform full and incremental backups, restore the mail server, or restore the LDAP server.

  • zmschedulebackup. This command is used to schedule full backups, incremental backups, and deletion of old backups.

  • zmbackup. This command executes full or incremental backup of the mail server. This is run on a live server, while the mailboxd process and the mailbox server are running. This command also has an option to manually delete old backups when they are no longer needed.

  • zmbackupabort. This command stops a full backup that is in process.

  • zmbackupabort -r. This command stops an ongoing restore.

  • zmbackupquery. This command lists the information about ongoing and completed backups, including labels and dates.

  • zmrestore. This command executes a full or incremental restore to the Zimbra mail server. The zmrestore command is performed on a server that is running.

  • zmrestoreoffline. This command restores the Zimbra mail server when the mailboxd process is stopped.

  • zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data.

Refer to Appendix A Command Line Utilities for usage and definitions for each of these commands.

Backing up using the Standard Method

When you initiate a backup, you can issue the command from the same server being backed up, run the command remotely and specify the target server on the command line, or use the Administration Console to start a backup session.

Scheduling a Standard Backup

When {product-name} was installed, the backup schedule for the standard method of full and incremental backups was added to the crontab. Under the default schedule, the full backup is scheduled for 1:00 a.m., every Saturday. The incremental backups are scheduled for 1:00 a.m., Sunday through Friday.

By default, backups older than a month are deleted every night at 12 a.m.

You can change the backup schedule using the zmschedulebackup command.

Specify the fields as follows, separate each field with a blank space:

  • minute — 0 through 59

  • hour — 0 through 23

  • day of month — 1 through 31

  • month — 1 through 12

  • day of week — 0 through 7 (0 or 7 is Sunday, or use names)

Type an asterisk (*) in the fields you are not using.

Example of zmschedulebackup options

  • Replace the existing full backup, incremental backup and delete backup schedule. When you use -R, the complete backup schedule is replaced. If you use this command, remember to set the delete schedule, if you want backup sessions to be scheduled for automatic deletion. This example replaces the existing schedule to have full backups run on Sunday at 1 a.m., incremental backups to run Monday through Saturday at 1 a.m., and old backups deleted at 12:00 a.m. every day.

    zmschedulebackup -R f “0 1 * * 7” i “0 1 * * 1-6” d 1m “0 0 * * *”

  • Add an additional full backup time to your current schedule. This example adds a full backup on Thursday at 1 a.m.

    zmschedulebackup -A f “0 1 * * 4”

  • Review your backup schedule. The schedule is displayed.

    zmschedulebackup -q

  • Save the schedule command to a text file. This would allow you to easily recreate the same schedule after reinstall or upgrade

    zmschedulebackup -s
Note
 If you change the default schedule and want to return to it, enter the command zmschedulebackup -D.

Default Standard Backup Schedule

The default backup schedule is displayed similarly to the following example:

Example 1. Default Backup Schedule
0 1 * * 6 /opt/zimbra/bin/zmbackup -f - all
0 1* * 0-5 /opt/zimbra/bin/zmbackup -i
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m

Read as follows:

The full backup runs on 1 a.m. on Saturdays
0 1 * * * 6 /opt/zimbra/bin/zmbackup -f - all
An incremental backup runs at 1 a.m. from Sunday through Friday
0 1* * 0-5 /opt/zimbra/bin/zmbackup -i
Backup sessions are deleted at midnight 1 month after they were created.
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
How to read the crontable

Each crontab entry contains six fields that appear in this order:

Field

1

2

3

4

5

6

0

1

*

*

6

/opt/zimbra/bin/zmbackup -f -all

  1. minute (0-59 allowed)

  2. hour (0-23)

  3. day of month (1-31)

  4. month (1-12 or names)

  5. day of week (0-7 or names allowed, with both 0 and 7 representing Sunday

  6. string to be executed

Note
 The asterisk character works as a wild card, representing every occurrence of the field’s value.

You can add additional recipient addresses or change the notification email address in the Administration Console Global Settings>Backup/Restore page.

Full Backup Process

The full backup process goes through the following steps to backup the mailbox, the database, the indexes, and the LDAP directory:

  1. Backs up the global system data including system tables and the local config.xml file.

  2. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.

  3. Places the account’s mailbox in maintenance mode to temporarily block mail delivery and user access to that mailbox.

  4. Backs up the mailbox.

    1. Creates MySQL dump for all data related to that mailbox.

    2. Backs up the message directory for that mailbox.

    3. Creates a backup of the index directory for that mailbox.

  5. Returns that account’s mailbox to active mode and moves on to the next one.

  6. Backs up the LDAP directory.

Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time.

Backup files are saved as zip files without compression. To change the default zip option, see Appendix A Command Line Interface, zmbackup section.

Incremental Backup Process

Incremental backups are run using the CLI command, zmbackup. The process for incremental backup is as follows:

  1. Backs up the global system data including system tables and the local config.xml.

  2. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.

  3. Moves the archive redo logs, created since the last backup, to the <backup target>/redologs directory.

    Archived logs that are less than an hour old at the time of incremental backup are copied to the backup and are not deleted. These redologs are deleted one hour after the backup. The interval is set by the localconfig key backup_archived_redolog_keep_time. The default is 3600 seconds.

    If no full backup for this account is found, the backup process performs a full backup on this account, even if only an incremental backup was specified.

  4. Backs up the LDAP directory.

Performing Manual Backups

CLI

Use the zmbackup command to perform the following backup operations:

  • Perform a manual back up all mailboxes on server<1>:

zmbackup -f -s server1.domain.com -a all

  • Perform a manual, incremental backup of all mailboxes on server1 since last full backup

zmbackup -i -s server1.domain.com -a all

  • Perform a manual, full backup of only user1’s mailbox on server1

zmbackup -f -s server1.domain.com -a [email protected]

Deleting Backup Sessions

You can delete backup sessions either by label or by date.

  • Deleting by label deletes that session and all backup sessions before that session.

  • Deleting by date deletes all backup session prior to the specified date.

CLI

For example, zmbackup -del 7d deletes backups older than 7 days from now. You can specify day (d), month (m), or year (y).

Finding a Specific Backups

Each full or incremental backup is a backup session.

Each backup session is labeled with date and time. For example, the label full-20070712.155951.123 says this is a backup from July 12, 2007 at 3:59:51.123.

Note
 The times set in the session label are GMT, not the local time. GMT is used rather than local time to preserve visual ordering across daylight savings transitions.
CLI

Use the zmbackupquery command to find full backup sessions.

  • To find a specific full backup session:

zmbackupquery -lb full-20070712.155951.123

  • To find a full backup sessions since a specific date:

zmbackupquery --type full --from “2007/01/01 12:45:45”

  • To find all full backup sessions in the backup directory:

zmbackupquery --type full

  • To find the best point in time to restore for an account specify a time window

zmbackupquery -a [email protected] --type full --from “2007/ 07/05 12:01:15” --to “2007/07/12 17:01:45”
Note
 If a backup session is interrupted because the server crashes during backup (not aborted), the interrupted backup session is saved as a temporary session. The temporary backup session is put in <backup target>/sessions_tmp directory. You can use the rm command to delete the directory.

Aborting a Full Backup in Progress

  1. Before you can abort a backup, you must know the backup session label. This label is displayed when zmbackup first starts. If you do not know the full backup label, use zmbackupquery to find the label.

  2. Use the zmbackupabort command to stop a backup that is in progress. The backup is immediately stopped and becomes a partially successful backup.

    • Stop the backup, if you know the label name

zmbackupabort -lb full-20070712.155951.123 -s server1

  • Stop the backup, if you do not know the label

zmbackupquery
zmbackupabort -s server1 -lb full-20070712.155951.123

Back up using the Auto-Grouped Method

The auto-grouped backup method is configured either from the Administration Console or from the CLI.

Admin Console

Configure> Global Settings, Backup/Restore page, or, to set it by servers:

Configure> Servers <server> Backup/Restore page.

Configuring Auto-Grouped Backup from the CLI

Set the backup method in the global configuration, and you can override the configuration on a per server basis if you do not want a particular server to use the auto-grouped backup method.

CLI

To set up auto-grouped backup, you modify LDAP attributes with the zmprov command:

zmprov mcf <ldap_attribute> <arg>

You can also set the attributes at the server level using zmprov ms.

The following LDAP attributes are modified:

  • zimbraBackupMode. Set it to be Auto-Grouped. The default is Standard.

  • zimbraBackupAutoGroupedInterval. Set this to the interval in either days or weeks that backup sessions should run for a group. The default is 1d. Backup intervals can be 1 or more days, entered as xd (1d); or 1 or more weeks, entered as xw (1w).

  • zimbraBackupAutoGroupedNumGroups. This is the number of groups to spread mailboxes over. The default is 7 groups.

Schedule Auto-Group Backups

You must configure the auto-group backup schedule.

CLI

Run zmschedulebackup -D to set the default schedule for auto-grouped backups based on your zimbraBackupAutoGroupedInterval setting.

One group is backed up each interval. The auto- grouped backup automatically adjusts for changes in the number of mailboxes on the server. Each backup session backs up the following:

  • All mailboxes that have never been backed up before. These are newly provisioned mailboxes.

  • All mailboxes that have not been backed within the number of scheduled backup days. For example, if backups are scheduled to run over six days, mailboxes that have not been backed up in the past 5 days are backed up.

  • More mailboxes, the oldest backup first. This is done so that the daily auto-grouped backup load is balanced.

    For example, if you configured the auto-grouped backup interval to be daily (1d) and the number of groups to be 7, the first time auto-grouped backup runs, all accounts are backed up. After the initial backup, auto-grouped backup runs again the next day. This time accounts that have been newly provisioned and a percentage of accounts close to one-seventh of the total are backed up again. Accounts with the oldest backup date are backed up first. The backup continues with newly provisioned account and approximately one-seventh of accounts being backed up daily over seven days.

When backing up shared messages, if a file representing a message already exists in the backup, it flags this object as such and does not copy its content again.

Backup files are saved as zip files without compression. To change the default zip option, see Appendix A Command Line Interface, zmbackup section.

These backup files can be used to restore the complete {product-name} system or individual mailboxes so that account and message data is completely restored. Archived redo logs are moved to the backup session as part of the full backup. When the server is restored from an auto-grouped backup, redo logs are replayed to bring the system to the point before the failure.

Backup Options

The backup process can be configured to selectively back up content and to backup the MySQL database.

Backup Up content Options

You can configure these backup options so that search indexes, blobs, and HSM blobs are not backed up during a full backup session.

  • zimbraBackupSkipSearchIndex. Default is FALSE. If set to TRUE, search index is not backed up. The mailbox will have to be reindexed after restoring from a backup without the search index.

  • zimbraBackupSkipBlobs. The default is FALSE. If this is set to TRUE, blobs are not backed up. This might be useful for getting a quicker backup of just database data when the blobs reside on a fault-tolerant storage. This configuration applies to all blobs, those on the primary volumes as well as secondary (HSM) volumes.

  • zimbraBackupSkipHsmBlobs. The default is false. If this is set to true, blobs on HSM volumes are no backed up. Set this if zimbraBackupSkipBlobs is false but you want to skip blobs on HSM volumes.

Back Up the MySQL Database

You can configure {product-name} backups to run mysqldump to backup your MySQL database during backup sessions. When this is enabled, a mysqldump backup runs with each full, incremental, and auto-grouped backup.

The mysqldump is a backup of your MySQL database at a specific time. Data changes that occur later than the dump file are written to the binary log.To recover to a specific point in time, binary logging must be enabled. See the Zimbra wiki article, MySQL Backup and Restore at http://wiki.zimbra.com/wiki/ MySQL_Backup_and_Restore.

The MySQL dump files are gzipped and placed in the backup target directory, or to /opt/zimbra/backup, if no directory is specified.

These files can be quite large. Make sure that the free disk space is at least three times greater than the actual mySQL database file for each mySQL database backup file that is saved.

  • Enable mysqldump to run automatically with your backups, type

zmlocalconfig edit mysql_backup_retention=<N>.

N is the number of copies of the mySQL database backups that areretained.

Note
 To restore a MySQL database, contact Zimbra support for assistance.

Managing Disk Space for Backups

Backup sessions fail if the target disk does not have enough space. All data backed up in the backup session is discarded and deleted.

You can choose to receive notification when your disk might not have enough space to complete the backup

CLI

Configuring the zimbraBackupMinFreeSpace attribute helps you manage running backup session by notifying you.

Set the value for attribute zimbraBackupMinFreeSpace to the amount of free space required on the backup target disk before a backup session is run. If the disk has less space than the value set in the attribute, the backup session will not run and an email notification is sent to the administrator.

Note
 If you are also backing up the MySQL database, make sure you set the value large enough to include the myslqdump file size.

The value for this attribute can be specified as a percentage of the total disk space, for example 25%, or as number of bytes, for example 300MB, 50GB, etc. The default value is 0, meaning the check is disabled and backup is always allowed to start.

CLI

The attribute can be set globally or by server.

  • As global:

zmprov mcf zimbraBackupMinFreeSpace <value>

  • By server:

zmprov ms <server hostname> zimbraBackupMinFreeSpace <value>

Backup sessions run if the free disk space is at least the value you set. If your backup file is larger than the value, the backup session fails. You should monitor the size of the backup files and adjust the attribute value if the backup require more space than the configured value.

Restoring Data

Three types of restore procedures can be run:

CLI

The zmrestore command is used to restore the mailboxes while the {product-name} mailbox server is running.

  • zmrestoreoffline — to restore the mail server when the mail server is down. This command is run for disaster recovery.

  • zmrestoreldap — to restore the content of the LDAP directory server.

The restore process allows all accounts or individual accounts to be specified.

Restore Process

The zmrestore process goes through the following steps to restore the mailbox, the database, the indexes, and the LDAP directory.

  1. Retrieves specified accounts to be restored, or specify all for all accounts that have been backed up.

  2. Iterates through each mailbox:

    1. Deletes the mailbox on the server to clear any existing data

    2. Restores the last full backup of the MySQL data, the index directory, and the message directory for that mailbox

    3. Replays redo logs in all incremental backups since last full backup

    4. Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server

    5. Replays the current redo log

Note
 Accounts are restored, even if the account exceeds its quota. The next time the user performs an action that affects quota, they receive a warning that they have exceeded their quota.
Important
 Users using the Zimbra Connector for Microsoft Outlook must perform an initial sync on the Outlook client when they log on after the Zimbra server is restored.

Examples

Example 2. Perform a full restore of all accounts on server1

Including last full backup and any incremental backups since last full backup

zmrestore -a all
Example 3. Perform a single account restore on server1
zmrestore -a [email protected]
Example 4. Restore to a specific point in time (PIT)

The following restore options affect redo log replay. If you do not specify one of these options, all redo logs since the full backup you’re restoring from are replayed

Important
 After you perform any of the following point-in-time restores,you should immediately run a complete backup for those accounts to avoid future restore problems with those accounts.

A restore that is run using any of the following options is a point-in-time restore:

  • -restoreToTime <arg> - Replay the redo logs until the time specified.

  • -restoreToIncrLabel <arg> - Replay redo logs up to and including this incremental backup.

  • -restoreToRedoSeq <arg> - Replay up to and including this redo log sequence.

  • -br - Replays the redo logs in backup only, therefore excluding archived and current redo logs of the system.

  • -rf - Restores to the full backup only. This does not include any incremental backups at all.

Example 5. Specify an exact time, the incremental backup label, or the redo log sequence to restore to.

Restore stops at the earliest possible point in time if more than one point in time restore options are specified.

zmrestore -a [email protected] <arg>

Two common ways to write the <timearg> are

  • “YYYY/MM/DD hh:mm:ss”

  • YYYYMMDD.hhmmss

Example 6. Perform an incremental restore only to last full backup, excluding incremental backups since then, for all accounts
zmrestore -rf --a all
Example 7. Restore mailbox and LDAP data for an account
zmrestore -ra -a [email protected]
Example 8. Restore to a new target account

A prefix is prepended to the original account names

zmrestore -ca -a [email protected] -pre restore

The result from the above example would be an account called [email protected].

Example 9. Restore system tables in the database (db) and the local config
zmrestore -sys
Example 10. Include --contineOnError (-c) to the command so that the restore process continues if an error is encountered.
zmrestore -a all -c

When -c is designated, accounts that could not be restored are displayed when the restore process is complete.

Example 11. Restore a specific account

Can also be used to restore deleted accounts

zmrestore -a [email protected]
Example 12. Avoid restoring accounts that were deleted
zmrestore -a [email protected] -skipDeletedAccounts
Example 13. Restore a mailbox, but exclude all the delete operations that were in the redolog replay

When the mailbox is restored it will contain messages that were deleted. This is useful if users use POP and remove messages from the server

zmrestore -a [email protected] --skipDeletes
Note
 When the latest point in time is requested, do not add a backup label (-lb). Without specifying the label, the most recent full backup before the requested point is automatically used as the starting point.

Stop a Restore Process

The zmbackupabort -r command interrupts a restore that is in process. The restore process stops after the current account finishes being restored. The command displays a message showing which accounts were not restored.

To stop the restore type:

zmbackupabort -r

Restore Mailboxes When Mail Server Is Down

The offline restore process can only be run when the mailboxd server is not running. In general, offline restore is run under the following circumstances:

  • Certain components of the Zimbra server are corrupted, and the server cannot be started. For example, the data in LDAP or the database are corrupted.

  • A disaster requires the Zimbra software to be reinstalled on the server.

The offline restore must be run before the {product-name} mailbox store server is started to keep the redo logs in sequence.

In a disaster recovery when the Zimbra software is reinstalled, if mailboxd is started before the backup files are restored, the mail server would begin to accept email messages and perform other activities, producing redo logs in the process. Since the pre-disaster data have not been restored to the server, the redo logs would be out of sequence. Once mailboxd is running, it would be too late to restore the pre-disaster data.

The offline restore process goes through the following steps.

  1. Specified accounts to be restored are retrieved. If the command-line does not specify any mailbox address, the list of all mailboxes on the specified mail host are retrieved from Zimbra LDAP directory server.

  2. Iterates through each mailbox:

    1. Deletes the mailbox on the server to clear any existing data

    2. Restores the last full backup of the MySQL data, the index directory, and the message directory for that mailbox

    3. Replays redo logs in all incremental backups since last full backup

    4. Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server

    5. Replays the current redo log

Restore All Accounts

  1. Restore all accounts on server1 when mailboxd is stopped

    zmrestoreoffline -a all

  2. Start mailboxd after the offline restore is complete

    zmcontrol startup

Restore Individual Accounts on a Live System

Use the zmrestore command to restore one or more selected accounts. In the event that a user’s mailbox has become corrupted, you might want to restore that user from the last full and incremental backup sets.

  1. For each account to be restored, put the account into maintenance mode

    zmprov ma <account> zimbraAccountStatus maintenance

    Maintenance mode prevents delivery of new emails during the restore. Otherwise, the emails would be overwritten during the restore process.

  2. Run the zmrestore command to restore the accounts

  3. For each account that was restored, put the account back into active mode

    zmprov ma <account> zimbraAccountStatus active
Important
 If a user account is restored and the COS that the account was assigned no longer exists, the default COS is assigned to the account.

Exclude Items from a Restore

When you restore from a full backup, you can exclude the search index and blobs.

  • Search index. If you do not restore the search index data, the mailbox willhave to be reindexed after the restore.

    zmrestore <all or account> --exclude-search-index

  • Blobs. This is a useful option when all blobs for the mailbox being restoredalready exists.

    zmrestore <all or account>|--exclude-blobs

  • HSM-blobs. This is useful when all HSM blobs for the mailbox beingrestored already exists.

    zmrestore <all or account> --exclude-hsm-blobs

Restore the LDAP Server

In a disaster recovery where you need to restore the entire system, restore the LDAP directory server first.

The zmrestoreldap command restores the global LDAP data including COS, distribution lists, etc. You can restore the complete LDAP server, which recreates the entire schema or you can restore specific accounts. You specify the session to restore. The restore command has to be run on the LDAP server being restored.

Examples

Example 14. Find the LDAP session labels
zmrestoreldap -lbs
Example 15. Restore the complete LDAP directory server
zmrestoreldap -lb full20061130135236
Example 16. Restore LDAP data for specific accounts
zmrestoreldap -lb full20061130135236 -a [email protected] [email protected]

General Steps for Disaster Recovery

Use the following steps to restore a mailbox store server in a general disaster scenario involving multiple machines.

Preparation

  1. Restore the LDAP directory server to a known good state before doing anything with the mailbox store server.

  2. Put all mailboxes into maintenance mode to prevent mail delivery and user login while restoring the mailboxes.

  3. Stop the mailbox store server if it is running.

Recovery

  1. Reinstall the {product-name} software on the mailbox server, if necessary.

  2. Restore mailboxes.

  3. Start the {product-name} server.

  4. Put all {product-name} mailboxes back in active mode.

  5. Run a full backup of the server.

Crash Recovery Server Startup

When your system unexpectedly stops and then restarts on startup, the server searches the redo log for uncommitted transactions and replays any that it finds. Replaying the redo logs brings the system to a consistent state.

Restore the {product-name}

If a complete machine failure occurs, use the following steps to restore to a new server.

Important
 The {product-name} version you install on the new server must be the same version as installed on the old server. The server can have a different operating system.

The new server hardware must meet the requirements described in the Installation Prerequisites section of the {product-name} Single Server Installation guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide.

You do the following to restore to a new server:

  1. Prepare the new server.

  2. Block client access to the old server’s IP address with firewall rules.

  3. Mount any volumes that were in use on the older server.

  4. Delete the MySQL data that is set up in the initial installation of {product-name}.

  5. Copy the backup files to the new server.

  6. Run zmrestoreldap to restore the global LDAP data.

  7. Run zmrestoreoffline to restore account data from the backup sessions.

  8. Prepare and run a new backup.

Old Server Status

Two scenarios for disaster recovery are the server has died and the {product-name} files cannot be accessed, or {product-name} is still running, but the server hardware needs to be replaced.

If the server is not running:

  1. Block client access to the server IP address with firewall rules.

  2. Find the latest full {product-name} backup session to use.

If server is still running, to prepare the move to the new server:

  1. Block client access to the server’s IP address with firewall rules.

  2. Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.

  3. Run zmcontrol stop, to stop {product-name}. In order to restore to the most current state, no new mail should be received after the last incremental backup has run.

  4. Change the hostname and IP address on the old server to something else. Do not turn off the server.

Install {product-abbrev} on a New Server

Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that {product-name} is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the {product-name} installation guide for more information about preparing the server. Before you begin to install {product-name}, note the information you need from the old server including: admin account name and password, LDAP, Amavis, and Postfix passwords, spam training and non-spam training user account names, exact domain name, and the global document account name.

Note
 Make sure the computer time is set to the same time as the old server. Verify that the old hostname and MX DNS records resolve to the new server.

  1. Copy the {product-name} License.xml file to a directory on the new server. You cannot complete the {product-name} installation if the license is not on the new server.

  2. Run ./install.sh and follow the directions in the installation guide to install {product-name}. Make sure that you configure the same domain, hostname, passwords as on the old server. During Zimbra Collaboration install, the following settings must be changed to match the original server settings:

    1. Zimbra LDAP Server. For Domain to create, identify the same defaultdomain as on the old server.

    2. Zimbra Mailbox Server. An administrator’s account is automaticallycreated.

      • Make sure that the account name for Admin user to create is the same name as on the original server.

      • Set the admin password to be the same as on the old server.

      • Set the LDAP password to be the same as on the old server.

      • Set the Postfix user and Amavis user passwords to be the same as on the old server

      • Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names onthe old server.

      • Global Document Account – This account name is automatically generated and is usually named wiki. If you changed this, change the Global Document Account name to be the same account name as on the original server.

    3. Change any other settings on the new server to match the configuration on the original server.

    4. In the main menu, set the default backup schedule and the automatic starting of servers after the configuration is complete to NO.

Restoring a Backup to a New Server

  1. Stop the new server

    zmcontrol stop

  2. If the old server had additional storage volumes configured, mount the additional volumes now.

  3. Delete the mysql data and reinitialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. As zimbra, type

    rm -rf /opt/zimbra/db/data/* /opt/zimbra/libexec/zmmyinit

    The mySQL service is now running.

  4. Copy all the files in the /backup directory from the old server or from an archive location to /opt/zimbra/backup.

  5. Restore the LDAP.

    zmrestoreldap -lb <latest_label>

    If you are restoring a large number of accounts, you might run a command such as the UNIX command, nohup, so that the session does not terminate before the restore is complete.

    Note
    		 To find the LDAP session label to restore, type zmrestoreldap –lbs.

  6. Because some {product-name} services are running at this point, type zmconvertctl start. This is required before running zmrestoreoffline.

  7. Sync your LDAP password from backup directory to the new production servers LDAP config.

    zmlocalconfig -f -e zimbra_ldap_password=<password>

  8. Start the offline restore.

    zmrestoreoffline -sys -a all -c -br.

    You might run a command such as nohup here also. To watch the progress, tail /opt/zimbra/log/mailbox.log.

    Note
    		 Use –c on the command line so that accounts are restored, even if some accounts encounter errors during the offline restore process.

  9. Because some {product-name} services are running at this point, type zmcontrol stop to stop all services.

  10. Remove any old backup sessions because these sessions are no longer valid.

    rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*

  11. Start {product-name}.

    zmcontrol start

  12. Run a full backup.

    zmbackup -f -a all

  13. Remove the firewall rules and allow client access to the new server.

Restoring from Different Failure Scenarios

The restoration steps are similar for most server failures you may encounter. If a failure occurs, review the disaster recovery section to understand the process and then follow the steps below for the specific type of failure.

Restore When LDAP is Corrupted

  1. Reinstall the LDAP server. See the {product-name} Installation guide.

  2. Find the label for the LDAP session to restore. Run the zmrestoreldap - lb <label> command, with no arguments to restore all accounts, domains, servers, COS, etc. for the LDAP server.

  3. Make sure that all accounts are in active mode. From the command line, type zmprov ma zimbraAccountStatus active

Restore After Replacing Corrupted Partitions

  1. If a partition becomes corrupted, replace the failed disk.

  2. To restore the latest full and incremental backup files, run

    zmrestore -a all

    The zmrestore process automatically retrieves the list of all mailboxes on the specified mail host from the backup date and iterates through each mailbox to restore the mailboxes to the last known good state.

Restore After Corrupted or Unreadable Redo Log

If the redo log becomes unreadable, the mailboxd service stops and cannot restart. If this happens, inspect the hardware and software to find the source of the problem before proceeding.

Without the latest redo log, the Zimbra mailbox server cannot be returned to the most current state. The Zimbra mailbox data can be restored to the latest archived redo log state. A new redo log for current transactions is created after the Zimbra mailbox server is restored.

Important
 The mailboxd service must not be running, and all accounts must be in maintenance mode before beginning.

  1. Put all accounts into maintenance mode.

    zmprov md <domain> zimbraDomainStatus maintenance

  2. With the mailboxd service not running, type

    zmrestoreoffline

    The offline restore process begins by retrieving the list of all mailboxes on the specified mail host from the backup.

    The offline restore than iterates through each mailbox to:

    • Delete the mailboxes on the server

    • Restore the last full backup from the backup area

    • Restore all incremental backups for that mailbox in order, since the last full backup. This involves replaying the redo logs from the backup target area

    • Replay all archived redo logs

    Because the redo log for current transactions is not available, the mailbox server is returned to the state of the last archived redo log.

  3. After the offline restore is complete, start {product-abbrev}.

    zmcontrol startup

  4. When the Zimbra mailbox server is up, run a full backup of the Zimbra server. The full backup must be run immediately to have the latest data backed up, because the latest redo log is not available.

Change Local Configuration Files After Restoring Zimbra

The localconfig.xml file, located in the /opt/zimbra/conf directory, includes the core Zimbra server configuration, such as paths and passwords, This file is backed up in full and incremental backups. When you run an incremental or full restore, the backed-up version of the localconfig.xml is renamed localconfig.xml.restore and is copied to the /opt/zimbra/conf directory.

If you have made changes since the last backup, you might need to replace the localconfig.xml file with the restored copy. Compare these files, and if the .restore file has the latest local configuration data, delete the localconfig.xmlfile and rename the file with the .restore extension to localconfig.xml.

Using snapshots to Backup and Restore

You can back up and restore a server using the snapshot feature provided by the storage layer rather than using Zimbra’s backup and restore feature. Using snapshots, you can maintain a standby site if the primary site fails and reroute users to the standby site to keep operations running.

Snapshots are taken for all volumes of data and are transferred to the standby site periodically. Data volumes that are backed up using snapshots include mysql, blobs, lucene index, and redologs.

When the primary site is down, the zmplayredo command is used to bring consistency to the snapshots and to reapply any changes in data to minimize data loss across volumes

There are four volumes of data:

  • Mysql

  • Blob

  • Lucene index

  • Redologs

Sets of snapshots are taken every hour and transferred to the remote standby site. However, all snapshots are not taken at one instant and could be a second to a minute apart from each other. Also, snapshots of redologs may be taken more frequently. The sequence of events could look like:

8:00:00 - snapshot mysql

8:00:01 - snapshot blob

8:00:02 - snapshot index

8:00:03 - snapshot redolog

8:05:00 - transfer the snapshot set to remote site completed

…​

8:15:00 - snapshot redolog

8:15:05 - transfer of redolog snapshot to remote site completed

…​

8:30:00 - snapshot redolog

8:30:05 - transfer of redolog snapshot to remote site completed

…​

8:35:00 - primary site fails

On the remote site, there are snapshots from the 8:00 set of data as well as subsequent snapshots of the redologs. They all have to be brought together so that the most recent information is available on the standby site once users are rerouted to it.

You can now run the zmplayredo command to replay changes from 8:00:00.

zmplayredo --fromTime ‘2008/10/17 08:00:00:000*’*

All data is brought forward to the current time and the standby site is set up and running. Data from 8:30:00 to 8:35:00 is lost but that is expected when the restore process is being carried out.