Storage: Add support for storage bucket backup (from Incus)

[boltmark]

This PR adds support for storage bucket backups. It includes cherry-picks from lxc/incus#365, and the below description comes in part from the associated incus PR.

Description

API

The following endpoints are added, along with the storage_bucket_backup extension:

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups

POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

DELETE /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>/export

Export

CLI

$ lxc storage bucket export
Description:
  Export storage buckets as tarball.

Usage:
  lxc storage bucket export [<remote>:]<pool> <bucket> [<path>] [flags]

Examples:
  lxc storage bucket default b1
      Download a backup tarball of the b1 storage bucket.

Flags:
      --compression   Define a compression algorithm: for backup or none
      --target        Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ lxc storage bucket export default bucket0
Backup exported successfully!                      

Archive

The export command creates a tarball from the defined bucket, which is structured as follows:

bucket0.tar.gz
├── backup
│   └── index.yaml
│   └── bucket
    │   └── file1
    │   └── file2

• The bucket directory contains the actual data
• backup.yml contains the bucket metadata and keys:

name: bucket0
backend: dir
pool: default
config:
  bucket:
    config: {}
    description: ""
    name: bucket0
    s3_url: ""
    location: none
  bucket_keys:
  - description: Admin user
    role: admin
    access-key: OO9O3OXLJWMED0W8P2I5
    secret-key: Ny4W9glDRFks7DzxlPQibREOAUSc1De9Yv0mOkuY
    name: admin
  - description: ""
    role: read-only
    access-key: JIY5PI6RPQWP7CAKJN57
    secret-key: APC95jzb9u7XUKw7J7AWAdVnaNVbjqj5dOLddkhT
    name: reader

Import

CLI

$ lxc storage bucket import
Description:
  Import backups of storage buckets.

Usage:
  lxc storage bucket import [<remote>:]<pool> <backup file> [<bucket>] [flags]

Examples:
  lxc storage bucket import default backup0.tar.gz
                Create a new storage bucket using backup0.tar.gz as the source.

Flags:
      --target   Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ lxc storage bucket import default backup.tar.gz bucket0
Backup exported successfully!                      

Design decisions of note

• One of the existing bucket keys is used by default, as everyone can at least read the bucket.
• A TransferManager struct is implemented, which utilizes MinIO to handle downloading files from a bucket to create the backup, and uploading files to a bucket when creating a bucket from a backup.

Overall, what's been changed

• The command storage bucket export and storage bucket import has been added to the command line interface (CLI).
• New APIs added
• Added new functionality for performing storage bucket backups and cleaning up expired backups
• Added new functionality for importing storage bucket backups
• Updated API and CLI documentation

[simondeziel]

I agree with CodeQL that it's risky. It should ideally be done only if requested by the user.

[simondeziel]

@boltmark Could you please make that InsecureSkipVerify configurable and defaulting to false? Thanks!

[github-actions]

Heads up @mionaalex - the "Documentation" label was applied to this issue.

[simondeziel]

Suggested change

	shift 3
	shift 4

[simondeziel]

@boltmark Could you please make that InsecureSkipVerify configurable and defaulting to false? Thanks!

[tomponline]

Hrm, ive not reviewed this yet in detail, but @simondeziel comment about InsecureSkipVerify caught me eye, but not because of the security implications (clearly they are undesirable) but more that it indicates that we have the LXD process apparently (in the case of minio based buckets) looping back to itself (@boltmark this anti-pattern will be familiar to you from your project force deletion PR too) in order to download all the files in an s3 bucket.

Now, I'm guessing the reason its been done like this is in order to support Ceph RADOS buckets (https://documentation.ubuntu.com/lxd/en/latest/reference/storage_cephobject/) where the b.GetBucketURL will be an external service (not LXD).

However this then got me thinking, why haven't we come across this issue before, and the answer appears to be, we have, and it was solved by:

lxd/lxd/storage/drivers/driver_cephobject_buckets.go

Lines 30 to 75 in b784423

	// s3Client returns a configured minio S3 client.
	func (d *cephobject) s3Client(creds S3Credentials) (*minio.Client, error) {
	u, err := url.ParseRequestURI(d.config["cephobject.radosgw.endpoint"])
	if err != nil {
	return nil, fmt.Errorf("Failed parsing cephobject.radosgw.endpoint: %w", err)
	}
	var transport http.RoundTripper
	certFilePath := d.config["cephobject.radosgw.endpoint_cert_file"]
	if u.Scheme == "https" && certFilePath != "" {
	certFilePath = shared.HostPath(certFilePath)
	// Read in the cert file.
	certs, err := os.ReadFile(certFilePath)
	if err != nil {
	return nil, fmt.Errorf("Failed reading %q: %w", certFilePath, err)
	}
	rootCAs := x509.NewCertPool()
	ok := rootCAs.AppendCertsFromPEM(certs)
	if !ok {
	return nil, fmt.Errorf("Failed adding S3 client certificates")
	}
	// Trust the cert pool in our client.
	config := &tls.Config{
	RootCAs: rootCAs,
	}
	transport = &http.Transport{TLSClientConfig: config}
	}
	minioClient, err := minio.New(path.Join(u.Host, u.Path), &minio.Options{
	Creds: credentials.NewStaticV4(creds.AccessKey, creds.SecretKey, ""),
	Secure: u.Scheme == "https",
	Transport: transport,
	})
	if err != nil {
	return nil, err
	}
	return minioClient, nil
	}

and for minio backends (which uses http rather than https as LXD's minio is only bound to local-loopback):

lxd/lxd/storage/s3/miniod/miniod.go

Lines 75 to 85 in b784423

	func (p *Process) S3Client() (*minio.Client, error) {
	s3Client, err := minio.New(p.url.Host, &minio.Options{
	Creds: credentials.NewStaticV4(p.username, p.password, ""),
	Secure: false,
	})
	if err != nil {
	return nil, err
	}
	return s3Client, nil
	}

So we need to check if using these existing client functions allow LXD to connect directly to the s3 service (rather than looping through LXD in the case of minio) and avoid the skipping of TLS checks when connecting to an external ceph rados gw service when TLS is enabled?

Also, suggest looking for follow up commits in Incus to see if this has already been addressed.

Also, if simpler, we could add to the storage drivers interface a GetS3Client() function which each storage driver could supply, to make the logic simpler in backendLXD.

[tomponline]

Need to resolve the loop-back and TLS checks skipping issue.