Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[linstor] Improvement of LINSTOR database backup management #187

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 17 additions & 30 deletions docs/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,18 +267,20 @@ linstor --help

## How do I restore LINSTOR DB from backup?

The backups of LINSTOR resources are stored in secrets as CRD YAML files and have a segmented format. Backup occurs automatically on a schedule.
The backups of LINSTOR resources are stored in Custom Resources `replicatedstoragemetadatabackups.storage.deckhouse.io` and have a segmented format. Backup occurs automatically on a schedule.

An example of a correctly formatted backup looks like this:

```shell
linstor-20240425074718-backup-0 Opaque 1 28s sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-1 Opaque 1 28s sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-2 Opaque 1 28s sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-completed Opaque 0 28s <none>
sds-replicated-volume-daily-backup-20241112130501-backup-0 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112130501-backup-1 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112140501-backup-0 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-daily-backup-20241112140501-backup-1 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-weekly-backup-20241112130400-backup-0 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
sds-replicated-volume-weekly-backup-20241112130400-backup-1 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
```

The backup is stored in encoded segments in secrets of the form linstor-%date_time%-backup-{0..2}, where the secret of the form linstor-%date_time%-backup-completed contains no data and serves as a marker for a successfully completed backup process.
The backup is stored in encoded segments in Custom Resources `replicatedstoragemetadatabackups.storage.deckhouse.io` of the form `sds-replicated-volume-{daily|weekly}-backup-%date_time%-backup-{0..2}`.

### Restoration Process

Expand All @@ -292,33 +294,18 @@ BACKUP_NAME="linstor_db_backup"
Check for the presence of backup copies:

```shell
kubectl -n $NAMESPACE get secrets --show-labels
kubectl get rsmb --show-labels
```

Example command output:

```shell
linstor-20240425072413-backup-0 Opaque 1 33m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072413
linstor-20240425072413-backup-1 Opaque 1 33m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072413
linstor-20240425072413-backup-2 Opaque 1 33m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072413
linstor-20240425072413-backup-completed Opaque 0 33m <none>
linstor-20240425072510-backup-0 Opaque 1 32m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072510
linstor-20240425072510-backup-1 Opaque 1 32m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072510
linstor-20240425072510-backup-2 Opaque 1 32m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072510
linstor-20240425072510-backup-completed Opaque 0 32m <none>
linstor-20240425072634-backup-0 Opaque 1 31m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072634
linstor-20240425072634-backup-1 Opaque 1 31m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072634
linstor-20240425072634-backup-2 Opaque 1 31m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072634
linstor-20240425072634-backup-completed Opaque 0 31m <none>
linstor-20240425072918-backup-0 Opaque 1 28m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072918
linstor-20240425072918-backup-1 Opaque 1 28m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072918
linstor-20240425072918-backup-2 Opaque 1 28m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072918
linstor-20240425072918-backup-completed Opaque 0 28m <none>
linstor-20240425074718-backup-0 Opaque 1 10m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-1 Opaque 1 10m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-2 Opaque 1 10m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-completed Opaque 0 10m <none>

sds-replicated-volume-daily-backup-20241112130501-backup-0 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112130501-backup-1 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112140501-backup-0 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-daily-backup-20241112140501-backup-1 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-weekly-backup-20241112130400-backup-0 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
sds-replicated-volume-weekly-backup-20241112130400-backup-1 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
```

Each backup has its own label with the creation time. Choose the desired one and copy its label into an environment variable.
Expand All @@ -331,7 +318,7 @@ LABEL_SELECTOR="sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074
Create a temporary directory to store archive parts:
```shell
TMPDIR=$(mktemp -d)
echo "Временный каталог: $TMPDIR"
echo "Temporary directory: $TMPDIR"
```

Next, create an empty archive and combine the secret data into one file:
Expand All @@ -346,7 +333,7 @@ MOBJECTS=$(kubectl get rsmb -l "$LABEL_SELECTOR" --sort-by=.metadata.name -o jso

for MOBJECT in $MOBJECTS; do
echo "Process: $MOBJECT"
kubectl get rsmb "$MOBJECT" -o jsonpath="{.data}" | base64 --decode >> "$COMBINED"
kubectl get rsmb "$MOBJECT" -o jsonpath="{.spec.data}" | base64 --decode >> "$COMBINED"
done
```

Expand Down
47 changes: 17 additions & 30 deletions docs/FAQ_RU.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,18 +267,20 @@ linstor --help

## Как восстановить БД LINSTOR из бэкапа?

Резервные копии ресурсов LINSTOR лежат в секретах в виде CRD yaml файлов, и имеют сегментированный формат.
Резервные копии ресурсов LINSTOR хранятся в Custom Resources `replicatedstoragemetadatabackups.storage.deckhouse.io` и имеют сегментированный формат.
Резервное копирование происходит автоматически по расписанию.

Пример корректно сформированного бэкапа выглядит следующим образом:
```shell
linstor-20240425074718-backup-0 Opaque 1 28s sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-1 Opaque 1 28s sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-2 Opaque 1 28s sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-completed Opaque 0 28s <none>
sds-replicated-volume-daily-backup-20241112130501-backup-0 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112130501-backup-1 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112140501-backup-0 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-daily-backup-20241112140501-backup-1 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-weekly-backup-20241112130400-backup-0 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
sds-replicated-volume-weekly-backup-20241112130400-backup-1 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
```

Резервная копия хранится закодированными сегментами в секретах вида `linstor-%date_time%-backup-{0..2}`, секрет вида `linstor-%date_time%-backup-completed` не содержит данных, и служит маркером корректно отработавшего процесса резервного копирования.
Резервная копия хранится закодированными сегментами в секретах вида `sds-replicated-volume-{daily|weekly}-backup-%date_time%-backup-{0..2}`.

### Процесс восстановления резервной копии

Expand All @@ -293,33 +295,18 @@ BACKUP_NAME="linstor_db_backup"
Проверьте наличие резервных копий

```shell
kubectl -n $NAMESPACE get secrets --show-labels
kubectl get rsmb --show-labels
```

Пример вывода команды

```shell
linstor-20240425072413-backup-0 Opaque 1 33m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072413
linstor-20240425072413-backup-1 Opaque 1 33m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072413
linstor-20240425072413-backup-2 Opaque 1 33m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072413
linstor-20240425072413-backup-completed Opaque 0 33m <none>
linstor-20240425072510-backup-0 Opaque 1 32m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072510
linstor-20240425072510-backup-1 Opaque 1 32m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072510
linstor-20240425072510-backup-2 Opaque 1 32m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072510
linstor-20240425072510-backup-completed Opaque 0 32m <none>
linstor-20240425072634-backup-0 Opaque 1 31m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072634
linstor-20240425072634-backup-1 Opaque 1 31m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072634
linstor-20240425072634-backup-2 Opaque 1 31m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072634
linstor-20240425072634-backup-completed Opaque 0 31m <none>
linstor-20240425072918-backup-0 Opaque 1 28m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072918
linstor-20240425072918-backup-1 Opaque 1 28m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072918
linstor-20240425072918-backup-2 Opaque 1 28m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425072918
linstor-20240425072918-backup-completed Opaque 0 28m <none>
linstor-20240425074718-backup-0 Opaque 1 10m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-1 Opaque 1 10m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-2 Opaque 1 10m sds-replicated-volume.deckhouse.io/linstor-db-backup=20240425074718
linstor-20240425074718-backup-completed Opaque 0 10m <none>

sds-replicated-volume-daily-backup-20241112130501-backup-0 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112130501-backup-1 97m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130501
sds-replicated-volume-daily-backup-20241112140501-backup-0 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-daily-backup-20241112140501-backup-1 37m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112140501
sds-replicated-volume-weekly-backup-20241112130400-backup-0 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
sds-replicated-volume-weekly-backup-20241112130400-backup-1 98m completed=true,sds-replicated-volume.deckhouse.io/sds-replicated-volume-db-backup=20241112130400
```

Каждая резервная копия имеет свой label со временем создания.
Expand Down Expand Up @@ -347,14 +334,14 @@ MOBJECTS=$(kubectl get rsmb -l "$LABEL_SELECTOR" --sort-by=.metadata.name -o jso

for MOBJECT in $MOBJECTS; do
echo "Process: $MOBJECT"
kubectl get rsmb "$MOBJECT" -o jsonpath="{.data}" | base64 --decode >> "$COMBINED"
kubectl get rsmb "$MOBJECT" -o jsonpath="{.spec.data}" | base64 --decode >> "$COMBINED"
done
```

Распакуйте объединенный tar-файл для получения ресурсов резервной копии
```shell
mkdir -p "./backup"
tar -xf "$COMBINED" -C "./backup --strip-components=2
tar -xf "$COMBINED" -C ./backup --strip-components=2
```
Проверьте содержимое резервной копии
```shell
Expand Down
Loading
Loading