Skip to content

Commit

Permalink
Add more details for COSI for AWS (neo4j#1574)
Browse files Browse the repository at this point in the history
Co-authored-by: NataliaIvakina <[email protected]>
  • Loading branch information
renetapopova and NataliaIvakina authored May 7, 2024
1 parent 3a97864 commit bd7eb48
Show file tree
Hide file tree
Showing 9 changed files with 230 additions and 87 deletions.
36 changes: 26 additions & 10 deletions modules/ROOT/pages/backup-restore/aggregate.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -96,32 +96,48 @@ Consult Neo4j support before use.

[NOTE]
====
As of Neo4j 5.19, the `--from-path=<path>` option can also load backup artifacts from AWS S3 URIs.
Neo4j uses link:https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html[default AWS credentials] to access AWS S3 URIs.
From Neo4j 5.19, the `--from-path=<path>` option can also load backup artifacts from AWS S3 URIs.
For more information, see <<aggregate-backup-aws-s3-example>>.
====


[[aggregate-backup-example]]
== Examples

.An example of how to perform aggregation of a set of backups located in a given folder for the `neo4j` database.
=== Aggregating a backup chain located in a given folder

The following is an example of how to perform aggregation of a set of backups located in a given folder for the `neo4j` database:

[source,shell]
----
bin/neo4j-admin database aggregate-backup --from-path=/mnt/backups/ neo4j
----
The command first looks inside the `/mnt/backups/` directory for a backup chain for the database `neo4j`. If found, it is then aggregated into a single backup artifact.

.An example of how to perform aggregation of a set of backups located in a given folder for the `neo4j` database in an AWS S3 URI.
[source,shell]
----
bin/neo4j-admin database aggregate-backup --from-path=s3://mnt/backups/ neo4j
----
=== Aggregating a backup chain identified using a given backup file

The following is an example of how to perform aggregation of a set of backups identified using a given backup file for the `neo4j` database:

.An example of how to perform aggregation of a set of backups identified using a given backup file for the `neo4j` database.
[source,shell]
----
bin/neo4j-admin database aggregate-backup --from-path=/mnt/backups/neo4j-2022-10-18T13-00-07.backup
----
The command checks the `/mnt/backups/` directory for a backup chain including the file _neo4j-2022-10-18T13-00-07.backup_, for the database `neo4j`.
If found, it is then aggregated into a single backup artifact.
This option is only available in Neo4j 5.2 and later.
This option is only available in Neo4j 5.2 and later.

[[aggregate-backup-aws-s3-example]]
=== Aggregating a backup chain located in an AWS S3 storage

The following is an example of how to perform aggregation of a set of backups located in a given folder in an AWS S3 storage.

include::partial$/aws-s3-overrides.adoc[]

include::partial$/aws-s3-credentials.adoc[]

. Then, use the following command to aggregate the backup chain located in the given folder in the AWS S3 storage:
+
[source,shell]
----
bin/neo4j-admin database aggregate-backup --from-path=s3://myBucket/myDirectory/myBackup.backup neo4j
----
24 changes: 19 additions & 5 deletions modules/ROOT/pages/backup-restore/offline-backup.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ neo4j-admin database dump [-h] [--expand-commands]
Dump a database into a single-file archive.
The archive can be used by the load command.
`<to-path>` should be a directory (in which case a file called _<database>.dump_ will be created), or `--to-stdout` can be supplied to use standard output.
If neither `--to-path` or `--to-stdout` is supplied `server.directories.dumps.root` setting will be used as a destination.
If neither `--to-path` or `--to-stdout` is supplied `server.directories.dumps.root` setting will be used as a destination.
It is not possible to dump a database that is mounted in a running Neo4j server.

=== Parameters
Expand Down Expand Up @@ -96,11 +96,13 @@ The `neo4j-admin database dump` command has the following options:
[NOTE]
====
As of Neo4j 5.19, the `--to-path=<path>` option can also dump databases to AWS S3 URIs.
Neo4j uses link:https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html[default AWS credentials] to access AWS S3 URIs.
For more information, see <<offline-backup-aws-s3-example>>.
====

[[offline-backup-example]]
== Example
== Examples

=== Dump the default database `neo4j` to a local directory

The following is an example of how to create a dump of the default database `neo4j` using the `neo4j-admin database dump` command.
The target directory _/dumps/neo4j_ must exist before running the command and the database must be offline.
Expand All @@ -112,11 +114,23 @@ bin/neo4j-admin database dump <database> --to-path=/full/path/to/dumps/

The command creates a file called _<database>.dump_ where `<database>` is the database specified in the command.

The following is an example of how to create a dump in an AWS S3 URI, as of Neo4j 5.19.
[[offline-backup-aws-s3-example]]
=== Dump a database to a folder located in an AWS S3 storage

_This feature is available from Neo4j 5.19._

To dump a database to an AWS S3 storage, you can use the `--to-path` option with an S3 URI.
For example, use the following steps.

include::partial$/aws-s3-overrides.adoc[]

include::partial$/aws-s3-credentials.adoc[]

. Run `neo4j-admin database dump` command to dump your database to an AWS S3 storage:
+
[source, shell, role="nocopy"]
----
bin/neo4j-admin database dump <database> --to-path=s3://full/path/to/aws-s3-dumps/
bin/neo4j-admin database dump <database> --to-path=s3://myBucket/myDirectory/
----

[NOTE]
Expand Down
47 changes: 32 additions & 15 deletions modules/ROOT/pages/backup-restore/online-backup.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,8 @@ If you want to force a full backup, use `FULL`.

[NOTE]
====
As of Neo4j 5.19, the `--to-path=<path>` option can also create backups in AWS S3 URIs.
Neo4j uses link:https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html[default AWS credentials] to access AWS S3 URIs.
From Neo4j 5.19, the `--to-path=<path>` option can also create backups in AWS S3 storage.
For more information, see <<online-backup-aws-s3>>.
====

[[backup-command-exit-codes]]
Expand Down Expand Up @@ -253,7 +253,7 @@ For example, if your current database has a `Total mapped size` of `128GB` as pe
====

[[online-backup-resources]]
=== Computational resources configurations
=== Computational resource configurations

Transaction log files::
The xref:database-internals/transaction-logs.adoc[transaction log files], which keep track of recent changes, are rotated and pruned based on a provided configuration.
Expand Down Expand Up @@ -334,44 +334,61 @@ If a name pattern is used for the database together with multiple target servers
The following are examples of how to perform a backup of a single database and multiple databases.
The target directory _/mnt/backups/neo4j_ must exist before calling the command and the database(s) must be online.

.Perform a backup of a single database
====
=== Back up a single database

You do not need to use the `--type` option to specify the type of backup.
By default, the type is automatically determined based on the existing backups.

[source, shell,role=nocopy noplay]
----
bin/neo4j-admin database backup --to-path=/path/to/backups/neo4j neo4j
----
====

.Perform a forced full backup of a single database.
====

=== Perform a forced full backup of a single database.

If you want to force a full backup after several differential backups, you can use the `--type=full` option.

[source, shell,role=nocopy noplay]
----
bin/neo4j-admin database backup --type=full --to-path=/path/to/backups/neo4j neo4j
----
====

.Use `neo4j-admin database backup` to back up multiple databases
====

=== Back up multiple databases

To back up several databases that match database a pattern you can use name globbing.
For example, to backup all databases that start with *n* from your three-node cluster, run:

[source, shell,role=nocopy noplay]
----
bin/neo4j-admin database backup --from=192.168.1.34:6362,192.168.1.35:6362,192.168.1.36:6362 --to-path=/mnt/backups/neo4j --pagecache=4G "n*"
----
====

.Use `neo4j-admin database backup` to back up a list of databases
====
=== Back up a list of databases

To back up several databases by name, you can provide a list of database names.

[source, shell,role=nocopy noplay]
----
neo4j-admin database backup --from=192.168.1.34:6362,192.168.1.35:6362,192.168.1.36:6362 --to-path=/mnt/backups/neo4j --pagecache=4G "test*" "neo4j"
----
====

[[online-backup-aws-s3]]
=== Back up a database to an AWS S3 storage

_This feature is available from Neo4j 5.19._

To back up a database to an AWS S3 storage, you can use the `--to-path` option with an S3 URI.
For example, use the following steps.

include::partial$/aws-s3-overrides.adoc[]

include::partial$/aws-s3-credentials.adoc[]

. Run the `neo4j-admin database backup` command to dump your database to the AWS S3 storage:
+
[source, shell, role="nocopy"]
----
bin/neo4j-admin database backup <database> --to-path=s3://myBucket/myDirectory/
----
39 changes: 27 additions & 12 deletions modules/ROOT/pages/backup-restore/restore-backup.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,9 @@ Usage of this option is only allowed if the `--from-path` parameter points to ex

[NOTE]
====
As of Neo4j 5.19, the `--from-path=<path>` option can also load backups from AWS S3 URIs.
Neo4j uses link:https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html[default AWS credentials] to access AWS S3 URIs.
====
From Neo4j 5.19, the `--from-path=<path>` option can also load backups from AWS S3 URIs.
For more information, see <<restore-backup-aws-s3>>.
====

[[restore-backup-example]]
== Examples
Expand Down Expand Up @@ -153,10 +153,12 @@ That means, if you restore `neo4j-2023-06-29T14-50-45.backup`, your database wil

=== Restore a database backup

This example assumes that you want to restore your data in a new database, called `mydatabase`.
The following examples assume that you want to restore your data in a new database, called `mydatabase`.
If you want to replace an existing database, you need to stop it first, and add the option `--overwrite-destination=true` to the restore command.

. Restore a database backup by running the following command:
==== Restore a full backup

Restore a database backup by running the following command:
+
[source, shell,role=nocopy noplay]
----
Expand All @@ -165,13 +167,6 @@ bin/neo4j-admin database restore --from-path=/path/to/backups/neo4j-2023-05-05T1
+
The `--from-path=` argument must contain the path to the last backup of a chain, in this case, `neo4j-2023-06-29T14-51-33.backup`.
+
As of Neo4j 5.19, `from-path=<path>` can alternatively load an AWS S3 URI:
+
[source, shell,role=nocopy noplay]
----
bin/neo4j-admin database restore --from-path=s3:///path/to/backups/neo4j-2023-05-05T11-26-38.backup mydatabase
----
+
[TIP]
====
If you want to restore several databases at once, you must stop them first and then you can alter the command by specifying a comma-separated list of paths to backup artifacts, and remove the `<database>` parameter.
Expand Down Expand Up @@ -219,6 +214,26 @@ For example, `--restore-until=123`.
CREATE DATABASE mydatabase;
----

[[restore-backup-aws-s3]]
=== Restore a backup located in an AWS S3 storage

_This feature is available from Neo4j 5.19._

You can restore a database backup located in an AWS S3 storage, you can use the `--from-path` option with an S3 URI.
For example, use the following steps.

include::partial$/aws-s3-overrides.adoc[]

include::partial$/aws-s3-credentials.adoc[]

. Run the `neo4j-admin database restore` command to restore the database located in your AWS S3 storage:
+
[source, shell,role=nocopy noplay]
----
bin/neo4j-admin database restore --from-path=s3://myBucket/myDirectory/myBackup.backup mydatabase
----


=== Restore a database backup in a cluster

To restore a database backup in a cluster, designate one of the servers to be used as a seeder, and restore the database backup on that server.
Expand Down
46 changes: 31 additions & 15 deletions modules/ROOT/pages/backup-restore/restore-dump.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -99,31 +99,19 @@ If `--info` is specified, then the database is not loaded, but information (i.e.

[NOTE]
====
As of Neo4j 5.19, the `--from-path=<path>` option can also load databases from AWS S3 URIs.
Neo4j uses link:https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html[default AWS credentials] to access AWS S3 URIs.
From Neo4j 5.19, the `--from-path=<path>` option can also load databases from AWS S3 URIs.
For more information, see <<restore-dump-example>>.
====

[[restore-dump-example]]
== Example

The following is an example of how to load the dump of the `neo4j` database (_neo4j.dump_) created in the section xref:backup-restore/offline-backup.adoc#offline-backup-example[Back up an offline database], using the `neo4j-admin database load` command.
The following are examples of how to load the dump of the `neo4j` database (_neo4j.dump_) created in the section xref:backup-restore/offline-backup.adoc#offline-backup-example[Back up an offline database], using the `neo4j-admin database load` command.
When replacing an existing database, you have to shut it down before running the command.
The `--overwrite-destination` option is required because you are replacing an existing database.

If you are not replacing an existing database, you must create the database (using `CREATE DATABASE` against the `system` database) after the load operation finishes.

[source,shell, role="nocopy"]
----
bin/neo4j-admin database load --from-path=/full-path/data/dumps <database> --overwrite-destination=true
----

The following is an example of how to load the dump from an AWS S3 URI, as of Neo4j 5.19.

[source,shell, role="nocopy"]
----
bin/neo4j-admin database load --from-path=s3://full/path/to/aws-s3-dumps/ <database> --overwrite-destination=true
----

[TIP]
====
The command looks for a file called _<database>.dump_ where `<database>` is the database specified in the command.
Expand All @@ -142,3 +130,31 @@ For more information, see xref:clustering/databases.adoc#cluster-seed[Seed a clu
`neo4j-admin database load` cannot be applied to xref:database-administration/composite-databases/manage-composite-databases.adoc[Composite databases].
It must be run directly on the databases that are associated with that Composite database.
====

=== Load a dump from a local directory

You can load a dump from a local directory using the following command:

[source,shell, role="nocopy"]
----
bin/neo4j-admin database load --from-path=/full-path/data/dumps <database> --overwrite-destination=true
----

=== Load a dump from an AWS S3 storage

_This feature is available from Neo4j 5.19._

You can load a dump from an AWS S3 storage using the following steps.

include::partial$/aws-s3-overrides.adoc[]

include::partial$/aws-s3-credentials.adoc[]

. Run the `neo4j-admin database load` command to load a dump from your AWS S3 storage:
+
[source,shell, role="nocopy"]
----
bin/neo4j-admin database load --from-path=s3://myBucket/myDirectory/ <database> --overwrite-destination=true
----


29 changes: 23 additions & 6 deletions modules/ROOT/pages/tools/neo4j-admin/consistency-checker.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,8 @@ Value can be plain numbers, like `10000000` or e.g. `20G` for 20 gigabytes, or e
[NOTE]
====
As of Neo4j 5.19, the `--from-path=<path>` option can also check artifacts in AWS S3 URIs.
Neo4j uses link:https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html[default AWS credentials] to access AWS S3 URIs.
====
For more information, see <<check-database-from-aws-s3-uris>>.
====


== Output
Expand All @@ -137,8 +137,8 @@ The location of the report file is the current working directory, or as specifie

== Example

.Run the consistency checker
====
=== Run the consistency checker

Note that the database must be stopped first.

----
Expand Down Expand Up @@ -189,10 +189,27 @@ Run with the `--from-path` option to check the consistency of a backup or a dump
bin/neo4j-admin database check --from-path=<directory-with-backup-or-dump> neo4j
----

====

[NOTE]
====
`neo4j-admin database check` cannot be applied to xref:database-administration/composite-databases/manage-composite-databases.adoc[Composite databases].
It must be run directly on the databases that are associated with that Composite database.
====

[[check-database-from-aws-s3-uris]]
=== Check a backup stored in an AWS S3 storage

_This feature is available from Neo4j 5.19._

To check data stored in an AWS S3 storage, you can use the `--from-path` option with S3 URIs.
For example, use the following steps.

include::partial$/aws-s3-overrides.adoc[]

include::partial$/aws-s3-credentials.adoc[]

. Run the `bin/neo4j-admin database check` command to check the consistency of your database from the AWS S3 storage:
+
[source, shell, role=noplay]
----
bin/neo4j-admin database check <database> --from-path=s3://myBucket/myDirectory/myBackup.backup
----
Loading

0 comments on commit bd7eb48

Please sign in to comment.