From 1b340cd582874e60ee99124dba2797e84bc53fb7 Mon Sep 17 00:00:00 2001 From: Ben Bardin Date: Fri, 12 Jan 2024 16:49:25 -0500 Subject: [PATCH] Azure blob storage: Correct scheme prefix to azure-blob:// (#17939) --- .../cockroachcloud/cdc/external-urls.md | 2 +- .../_includes/releases/v22.2/v22.2.15.md | 1 - .../_includes/v22.2/cdc/external-urls.md | 12 ++++++------ .../_includes/v23.1/cdc/external-urls.md | 2 +- src/current/_includes/v23.1/misc/azure-blob.md | 3 +++ .../_includes/v23.2/cdc/external-urls.md | 2 +- .../v23.2/cdc/list-cloud-changefeed-uris.md | 2 +- src/current/_includes/v23.2/misc/azure-blob.md | 3 +++ src/current/v22.2/cancel-job.md | 10 +++++----- src/current/v22.2/pause-job.md | 6 +++--- src/current/v22.2/resume-job.md | 6 +++--- src/current/v22.2/show-jobs.md | 16 ++++++++-------- src/current/v23.1/alter-backup.md | 4 ++-- src/current/v23.1/cancel-job.md | 10 +++++----- src/current/v23.1/changefeed-sinks.md | 2 +- .../v23.1/cloud-storage-authentication.md | 10 +++++++--- src/current/v23.1/create-changefeed.md | 2 +- .../v23.1/create-external-connection.md | 1 + src/current/v23.1/pause-job.md | 6 +++--- src/current/v23.1/resume-job.md | 6 +++--- src/current/v23.1/show-jobs.md | 18 +++++++++--------- .../take-and-restore-encrypted-backups.md | 8 ++++---- .../take-and-restore-locality-aware-backups.md | 2 +- src/current/v23.1/use-cloud-storage.md | 6 +++--- src/current/v23.2/alter-backup.md | 16 ++++++++-------- src/current/v23.2/cancel-job.md | 10 +++++----- src/current/v23.2/changefeed-sinks.md | 2 +- .../v23.2/cloud-storage-authentication.md | 10 +++++++--- src/current/v23.2/pause-job.md | 6 +++--- src/current/v23.2/resume-job.md | 6 +++--- src/current/v23.2/show-jobs.md | 18 +++++++++--------- .../take-and-restore-encrypted-backups.md | 8 ++++---- .../take-and-restore-locality-aware-backups.md | 2 +- src/current/v23.2/use-cloud-storage.md | 6 +++--- 34 files changed, 119 insertions(+), 105 deletions(-) create mode 100644 src/current/_includes/v23.1/misc/azure-blob.md create mode 100644 src/current/_includes/v23.2/misc/azure-blob.md diff --git a/src/current/_includes/cockroachcloud/cdc/external-urls.md b/src/current/_includes/cockroachcloud/cdc/external-urls.md index 4fe1f5d6c1a..7a72714bf6d 100644 --- a/src/current/_includes/cockroachcloud/cdc/external-urls.md +++ b/src/current/_includes/cockroachcloud/cdc/external-urls.md @@ -42,7 +42,7 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` +Azure | `azure-blob://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` Google Cloud | `gs://acme-co` HTTP | `http://localhost:8080/employees` NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees` [5](#considerations) diff --git a/src/current/_includes/releases/v22.2/v22.2.15.md b/src/current/_includes/releases/v22.2/v22.2.15.md index 6f152b1469a..da20054254d 100644 --- a/src/current/_includes/releases/v22.2/v22.2.15.md +++ b/src/current/_includes/releases/v22.2/v22.2.15.md @@ -35,7 +35,6 @@ Release Date: October 23, 2023 - When upgrading from previous releases, it was possible that ambiguous references to [sequences](https://www.cockroachlabs.com/docs/v22.2/create-sequence) would fail to be resolved by ID if they were in the same schema. Additionally, the logging is improved for ambiguous cases where the reference cannot be resolved. [#110721][#110721] - Fixed a bug where [`RESET `](https://www.cockroachlabs.com/docs/v22.2/row-level-ttl#reset-a-storage-parameter-to-its-default-value)[(`ttl_expire_after`)](https://www.cockroachlabs.com/docs/v22.2/row-level-ttl#using-ttl_expire_after) could incorrectly remove [`ttl_expiration_expression`](https://www.cockroachlabs.com/docs/v22.2/row-level-ttl#using-ttl_expiration_expression). [#110750][#110750] - Fixed a bug where [`format_type`](https://www.cockroachlabs.com/docs/v22.2/functions-and-operators#compatibility-functions) builtin did not honor `typemod` information for [array](https://www.cockroachlabs.com/docs/v22.2/array) types, leading to incorrect output. [#110939][#110939] -- External connection URLs now accept the scheme `azure-blob` for connections to [Azure](https://www.cockroachlabs.com/docs/v22.2/use-cloud-storage#url-format) Blob Storage. For backward compatibility, schemes `azure` and `azure-storage` schemes continue to work for connections to Azure Blob Storage. [#111286][#111286] - Fixed a bug where `indoption` inside [`pg_index`](https://www.cockroachlabs.com/docs/v22.2/pg-catalog#data-exposed-by-pg_catalog) was not properly encoded, causing it to be unable to be decoded as `int2vector`. [#111956][#111956] - Fixed a bug where the [optimizer](https://www.cockroachlabs.com/docs/v22.2/cost-based-optimizer) failed to honor the `statement_timeout` session setting when generating constrained index scans for queries with large `IN` lists or `= ANY` predicates on multiple index key columns, which may lead to an [out-of-memory crash (OOM)](https://www.cockroachlabs.com/docs/v22.2/cluster-setup-troubleshooting#out-of-memory-oom-crash) on the node. [#112075][#112075] - Fixed a bug that caused internal errors during query [optimization](https://www.cockroachlabs.com/docs/v22.2/cost-based-optimizer) in rare cases. The bug has been present since version 2.1.11, but it is more likely to occur in version 21.2.0 and later, though it is still rare. The bug only presents when a query contains `min` and `max` [aggregate functions](https://www.cockroachlabs.com/docs/v22.2/functions-and-operators#aggregate-functions). [#112299][#112299] diff --git a/src/current/_includes/v22.2/cdc/external-urls.md b/src/current/_includes/v22.2/cdc/external-urls.md index 2b2f88762e1..7db4d6406a8 100644 --- a/src/current/_includes/v22.2/cdc/external-urls.md +++ b/src/current/_includes/v22.2/cdc/external-urls.md @@ -4,7 +4,7 @@ Location | Scheme | Host | Parameters | |-------------------------------------------------------------+-------------+--------------------------------------------------+---------------------------------------------------------------------------- -Amazon | `s3` | Bucket name | `AUTH` [1](#considerations) (optional; can be `implicit` or `specified`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` +Amazon | `s3` | Bucket name | `AUTH` [1](#considerations) (optional; can be `implicit` or `specified`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` Azure | `azure` | N/A (see [Example file URLs](#example-file-urls) | `AZURE_ACCOUNT_KEY`, `AZURE_ACCOUNT_NAME` Google Cloud [2](#considerations) | `gs` | Bucket name | `AUTH` (optional; can be `default`, `implicit`, or `specified`), `CREDENTIALS` HTTP [3](#considerations) | `http` | Remote host | N/A @@ -39,10 +39,10 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti #### Example file URLs -Location | Example +Location | Example -------------+---------------------------------------------------------------------------------- -Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` -Google Cloud | `gs://acme-co` -HTTP | `http://localhost:8080/employees` +Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` +Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` +Google Cloud | `gs://acme-co` +HTTP | `http://localhost:8080/employees` NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees` [5](#considerations) diff --git a/src/current/_includes/v23.1/cdc/external-urls.md b/src/current/_includes/v23.1/cdc/external-urls.md index 0517fb48c68..d87cb7538d1 100644 --- a/src/current/_includes/v23.1/cdc/external-urls.md +++ b/src/current/_includes/v23.1/cdc/external-urls.md @@ -42,7 +42,7 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` +Azure | `azure-blob://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` Google Cloud | `gs://acme-co` HTTP | `http://localhost:8080/employees` NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees` [5](#considerations) diff --git a/src/current/_includes/v23.1/misc/azure-blob.md b/src/current/_includes/v23.1/misc/azure-blob.md new file mode 100644 index 00000000000..d8b947c3a5a --- /dev/null +++ b/src/current/_includes/v23.1/misc/azure-blob.md @@ -0,0 +1,3 @@ +{{site.data.alerts.callout_info}} +For backward compatibility, schemes `azure://` and `azure-storage://` are also supported, though `azure-blob://` is recommended. +{{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v23.2/cdc/external-urls.md b/src/current/_includes/v23.2/cdc/external-urls.md index 0517fb48c68..d87cb7538d1 100644 --- a/src/current/_includes/v23.2/cdc/external-urls.md +++ b/src/current/_includes/v23.2/cdc/external-urls.md @@ -42,7 +42,7 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` +Azure | `azure-blob://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co` Google Cloud | `gs://acme-co` HTTP | `http://localhost:8080/employees` NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees` [5](#considerations) diff --git a/src/current/_includes/v23.2/cdc/list-cloud-changefeed-uris.md b/src/current/_includes/v23.2/cdc/list-cloud-changefeed-uris.md index 6c9849b111e..798c6f58fb9 100644 --- a/src/current/_includes/v23.2/cdc/list-cloud-changefeed-uris.md +++ b/src/current/_includes/v23.2/cdc/list-cloud-changefeed-uris.md @@ -1,6 +1,6 @@ Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}'` -Azure Blob Storage | `'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'` +Azure Blob Storage | `'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'` Google Cloud | `'gs://{BUCKET NAME}/{PATH}?AUTH=specified&CREDENTIALS={ENCODED KEY}'` HTTP | `'file-http(s)://localhost:8080/{PATH}'` or `'http(s)://localhost:8080/{PATH}'`

**Note:** Using `http(s)` without the `file-` prefix is deprecated as a [changefeed sink]({% link {{ page.version.version }}/changefeed-sinks.md %}) scheme. There is continued support for `http(s)`, but it will be removed in a future release. We recommend implementing the `file-http(s)` scheme for changefeed messages. \ No newline at end of file diff --git a/src/current/_includes/v23.2/misc/azure-blob.md b/src/current/_includes/v23.2/misc/azure-blob.md new file mode 100644 index 00000000000..d8b947c3a5a --- /dev/null +++ b/src/current/_includes/v23.2/misc/azure-blob.md @@ -0,0 +1,3 @@ +{{site.data.alerts.callout_info}} +For backward compatibility, schemes `azure://` and `azure-storage://` are also supported, though `azure-blob://` is recommended. +{{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/v22.2/cancel-job.md b/src/current/v22.2/cancel-job.md index 7e476490ac3..b0f318eaee8 100644 --- a/src/current/v22.2/cancel-job.md +++ b/src/current/v22.2/cancel-job.md @@ -46,11 +46,11 @@ Parameter | Description > SHOW JOBS; ~~~ ~~~ -+----------------+---------+-------------------------------------------+... -| id | type | description |... -+----------------+---------+-------------------------------------------+... -| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -+----------------+---------+-------------------------------------------+... ++----------------+---------+------------------------------------------------+... +| id | type | description |... ++----------------+---------+------------------------------------------------+... +| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... ++----------------+---------+------------------------------------------------+... ~~~ ~~~ sql > CANCEL JOB 27536791415282; diff --git a/src/current/v22.2/pause-job.md b/src/current/v22.2/pause-job.md index 0eeeb18bfc1..138e1dde4cd 100644 --- a/src/current/v22.2/pause-job.md +++ b/src/current/v22.2/pause-job.md @@ -50,9 +50,9 @@ Parameter | Description ~~~ ~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... +-----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... ~~~ {% include_cached copy-clipboard.html %} diff --git a/src/current/v22.2/resume-job.md b/src/current/v22.2/resume-job.md index f5cff78f663..ca2ad18024d 100644 --- a/src/current/v22.2/resume-job.md +++ b/src/current/v22.2/resume-job.md @@ -43,9 +43,9 @@ Parameter | Description ~~~ ~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... +-----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... ~~~ {% include_cached copy-clipboard.html %} diff --git a/src/current/v22.2/show-jobs.md b/src/current/v22.2/show-jobs.md index 45083c9bbc7..552f5e827be 100644 --- a/src/current/v22.2/show-jobs.md +++ b/src/current/v22.2/show-jobs.md @@ -107,9 +107,9 @@ Status | Description ~~~ ~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++---------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... ~~~ ### Filter jobs @@ -122,9 +122,9 @@ You can filter jobs by using `SHOW JOBS` as the data source for a [`SELECT`](sel ~~~ ~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++---------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... ~~~ @@ -235,8 +235,8 @@ To block `SHOW JOB` until the provided job ID reaches a terminal state, use `SHO > SHOW JOB WHEN COMPLETE 27536791415282; ~~~ ~~~ - job_id | job_type | description |... -+----------------+-----------+-------------------------------------------+... + job_id | job_type | description |... ++----------------+-----------+------------------------------------------------+... 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... ~~~ diff --git a/src/current/v23.1/alter-backup.md b/src/current/v23.1/alter-backup.md index a6dd2d6978f..6cb5e463234 100644 --- a/src/current/v23.1/alter-backup.md +++ b/src/current/v23.1/alter-backup.md @@ -79,7 +79,7 @@ To add a new KMS key to the most recent backup: {% include_cached copy-clipboard.html %} ~~~ sql -ALTER BACKUP LATEST IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +ALTER BACKUP LATEST IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' ADD NEW_KMS = 'azure-kms:///{new key}/{new key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' WITH OLD_KMS = 'azure-kms:///{old key}/{old key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; ~~~ @@ -88,7 +88,7 @@ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement tha {% include_cached copy-clipboard.html %} ~~~ sql -ALTER BACKUP '2023/03/14-203808.29' IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +ALTER BACKUP '2023/03/14-203808.29' IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' ADD NEW_KMS = 'azure-kms:///{new key}/{new key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' WITH OLD_KMS = 'azure-kms:///{old key}/{old key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; ~~~ diff --git a/src/current/v23.1/cancel-job.md b/src/current/v23.1/cancel-job.md index f399b7ea18a..d3fb5574f94 100644 --- a/src/current/v23.1/cancel-job.md +++ b/src/current/v23.1/cancel-job.md @@ -48,11 +48,11 @@ Parameter | Description > SHOW JOBS; ~~~ ~~~ -+----------------+---------+-------------------------------------------+... -| id | type | description |... -+----------------+---------+-------------------------------------------+... -| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -+----------------+---------+-------------------------------------------+... ++----------------+---------+------------------------------------------------+... +| id | type | description |... ++----------------+---------+------------------------------------------------+... +| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ++----------------+---------+------------------------------------------------+... ~~~ ~~~ sql > CANCEL JOB 27536791415282; diff --git a/src/current/v23.1/changefeed-sinks.md b/src/current/v23.1/changefeed-sinks.md index b1f3a67bcd9..362574d3d50 100644 --- a/src/current/v23.1/changefeed-sinks.md +++ b/src/current/v23.1/changefeed-sinks.md @@ -299,7 +299,7 @@ Examples of supported cloud storage sink URIs: ### Azure Blob Storage ~~~ -'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}' +'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}' ~~~ ### Google Cloud Storage diff --git a/src/current/v23.1/cloud-storage-authentication.md b/src/current/v23.1/cloud-storage-authentication.md index 74a8716bdb7..5d98fcdf060 100644 --- a/src/current/v23.1/cloud-storage-authentication.md +++ b/src/current/v23.1/cloud-storage-authentication.md @@ -686,7 +686,7 @@ You can authenticate to Azure with explicit credentials in the following ways: - `AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration. ~~~ - azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID} + azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID} ~~~ You can authenticate to Azure Storage and Azure Key Vault with this URI format. @@ -699,9 +699,11 @@ You can authenticate to Azure with explicit credentials in the following ways: It is necessary to [url encode](https://wikipedia.org/wiki/Percent-encoding) the account key since it is base64-encoded and may contain `+`, `/`, `=` characters. ~~~ - azure://{container name}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}&AZURE_ENVIRONMENT=AZUREUSGOVERNMENTCLOUD + azure-blob://{container name}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}&AZURE_ENVIRONMENT=AZUREUSGOVERNMENTCLOUD ~~~ + {% include {{ page.version.version }}/misc/azure-blob.md %} + ## Azure Blob Storage implicit authentication {{site.data.alerts.callout_info}} @@ -782,9 +784,11 @@ To set up `implicit` authentication to Azure Blob Storage (or a KMS resource), y {% include_cached copy-clipboard.html %} ~~~sql - BACKUP DATABASE {database} INTO 'azure://{container name}?AUTH=implicit&AZURE_ACCOUNT_NAME={account name}'; + BACKUP DATABASE {database} INTO 'azure-blob://{container name}?AUTH=implicit&AZURE_ACCOUNT_NAME={account name}'; ~~~ + {% include {{ page.version.version }}/misc/azure-blob.md %} +
diff --git a/src/current/v23.1/create-changefeed.md b/src/current/v23.1/create-changefeed.md index a53ed85bae3..6c448600449 100644 --- a/src/current/v23.1/create-changefeed.md +++ b/src/current/v23.1/create-changefeed.md @@ -107,7 +107,7 @@ The following are example file URLs for each of the cloud storage schemes: Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}'` -Azure Blob Storage | `'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'` +Azure Blob Storage | `'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'` Google Cloud | `'gs://{BUCKET NAME}/{PATH}?AUTH=specified&CREDENTIALS={ENCODED KEY'` HTTP | `'http://localhost:8080/{PATH}'` diff --git a/src/current/v23.1/create-external-connection.md b/src/current/v23.1/create-external-connection.md index 7048e0d2c8b..192584ce2aa 100644 --- a/src/current/v23.1/create-external-connection.md +++ b/src/current/v23.1/create-external-connection.md @@ -56,6 +56,7 @@ Storage or sink | Operation support [Amazon S3]({% link {{ page.version.version }}/use-cloud-storage.md %}) | Backups, restores, imports, exports, changefeeds [Amazon S3 KMS]({% link {{ page.version.version }}/take-and-restore-encrypted-backups.md %}#aws-kms-uri-format) | Encrypted backups [Azure Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) | Backups, restores, imports, exports, changefeeds +[Azure Key Vault]({% link {{ page.version.version }}/take-and-restore-encrypted-backups.md %}#azure-key-vault-uri-format) | Encrypted backups [Confluent Schema Registry]({% link {{ page.version.version }}/create-changefeed.md %}#confluent-registry) | Changefeeds [Google Cloud Pub/Sub]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub) | Changefeeds [Google Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) | Backups, restores, imports, exports, changefeeds diff --git a/src/current/v23.1/pause-job.md b/src/current/v23.1/pause-job.md index b2bb2c08fa5..87bb74d5521 100644 --- a/src/current/v23.1/pause-job.md +++ b/src/current/v23.1/pause-job.md @@ -74,9 +74,9 @@ See the following pages for details on metrics: ~~~ ~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... +-----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ {% include_cached copy-clipboard.html %} diff --git a/src/current/v23.1/resume-job.md b/src/current/v23.1/resume-job.md index 8c2b37494cc..7569ef93439 100644 --- a/src/current/v23.1/resume-job.md +++ b/src/current/v23.1/resume-job.md @@ -45,9 +45,9 @@ Parameter | Description ~~~ ~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... +-----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ {% include_cached copy-clipboard.html %} diff --git a/src/current/v23.1/show-jobs.md b/src/current/v23.1/show-jobs.md index 659e6f50ab9..cfdbf77bc89 100644 --- a/src/current/v23.1/show-jobs.md +++ b/src/current/v23.1/show-jobs.md @@ -116,9 +116,9 @@ We recommend monitoring paused jobs to protect historical data from [garbage col ~~~ ~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++---------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ ### Filter jobs @@ -131,9 +131,9 @@ You can filter jobs by using `SHOW JOBS` as the data source for a [`SELECT`]({% ~~~ ~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++---------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ @@ -235,9 +235,9 @@ To block `SHOW JOB` until the provided job ID reaches a terminal state, use `SHO > SHOW JOB WHEN COMPLETE 27536791415282; ~~~ ~~~ - job_id | job_type | description |... -+----------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ ### Show jobs for a schedule diff --git a/src/current/v23.1/take-and-restore-encrypted-backups.md b/src/current/v23.1/take-and-restore-encrypted-backups.md index 64223743225..1da266441ef 100644 --- a/src/current/v23.1/take-and-restore-encrypted-backups.md +++ b/src/current/v23.1/take-and-restore-encrypted-backups.md @@ -224,7 +224,7 @@ RESTORE FROM LATEST IN 's3://{BUCKET NAME}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET {% include_cached copy-clipboard.html %} ~~~ sql -BACKUP INTO 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +BACKUP INTO 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH kms = 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; ~~~ @@ -234,7 +234,7 @@ BACKUP INTO 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account {% include_cached copy-clipboard.html %} ~~~ sql -BACKUP INTO 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +BACKUP INTO 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH KMS=( 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}', 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' @@ -247,7 +247,7 @@ To view a [kms encrypted locality-aware backup](#use-key-management-service), us {% include_cached copy-clipboard.html %} ~~~ sql -SHOW BACKUP FROM '2023/07/14-211406.03' IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +SHOW BACKUP FROM '2023/07/14-211406.03' IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH KMS=( 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}', 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' @@ -286,7 +286,7 @@ database_name | parent_schema_name | object_name | object_type | back {% include_cached copy-clipboard.html %} ~~~ sql -RESTORE FROM LATEST IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +RESTORE FROM LATEST IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH kms = 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; ~~~ diff --git a/src/current/v23.1/take-and-restore-locality-aware-backups.md b/src/current/v23.1/take-and-restore-locality-aware-backups.md index 44e2e680470..b65bd119ba5 100644 --- a/src/current/v23.1/take-and-restore-locality-aware-backups.md +++ b/src/current/v23.1/take-and-restore-locality-aware-backups.md @@ -208,7 +208,7 @@ The `system.zones` table stores your cluster's [zone configurations]({% link {{ {% include_cached copy-clipboard.html %} ~~~ sql RESTORE TABLE system.zones FROM '2021/03/23-213101.37' IN - 'azure://acme-co-backup?AZURE_ACCOUNT_KEY=hash&AZURE_ACCOUNT_NAME=acme-co' + 'azure-blob://acme-co-backup?AZURE_ACCOUNT_KEY=hash&AZURE_ACCOUNT_NAME=acme-co' WITH into_db = 'newdb'; ~~~ diff --git a/src/current/v23.1/use-cloud-storage.md b/src/current/v23.1/use-cloud-storage.md index f96c83c02f3..bb78325c6db 100644 --- a/src/current/v23.1/use-cloud-storage.md +++ b/src/current/v23.1/use-cloud-storage.md @@ -35,7 +35,7 @@ The following table provides a list of the parameters supported by each storage Location | Scheme | Host | Parameters ------------------------------------------------------------+-------------+--------------------------------------------------+---------------------------------------------------------------------------- Amazon S3 | `s3` | Bucket name | [`AUTH`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#amazon-s3-specified): `implicit` or `specified` (default: `specified`). When using `specified` pass user's `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

[`ASSUME_ROLE`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) (optional): Pass the [ARN](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) of the role to assume. Use in combination with `AUTH=implicit` or `specified`.

[`AWS_SESSION_TOKEN`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}) (optional): For more information, see Amazon's guide on [temporary credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).

[`S3_STORAGE_CLASS`](#amazon-s3-storage-classes) (optional): Specify the Amazon S3 storage class for created objects. Note that Glacier Flexible Retrieval and Glacier Deep Archive are not compatible with incremental backups. **Default**: `STANDARD`. -Azure Blob Storage | `azure` | Storage container | `AZURE_ACCOUNT_NAME`: The name of your Azure account.

`AZURE_ACCOUNT_KEY`: Your Azure account key. You must [url encode](https://wikipedia.org/wiki/Percent-encoding) your Azure account key before authenticating to Azure Storage. For more information, see [Authentication - Azure Storage]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#azure-blob-storage-specified-authentication).

`AZURE_ENVIRONMENT`: (optional) {% include {{ page.version.version }}/misc/azure-env-param.md %}

`AZURE_CLIENT_ID`: Application (client) ID for your [App Registration](https://learn.microsoft.com/azure/active-directory/develop/quickstart-register-app#register-an-application).

`AZURE_CLIENT_SECRET`: Client credentials secret generated for your App Registration.

`AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration. +Azure Blob Storage | `azure-blob` | Storage container | `AZURE_ACCOUNT_NAME`: The name of your Azure account.

`AZURE_ACCOUNT_KEY`: Your Azure account key. You must [url encode](https://wikipedia.org/wiki/Percent-encoding) your Azure account key before authenticating to Azure Storage. For more information, see [Authentication - Azure Storage]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#azure-blob-storage-specified-authentication).

`AZURE_ENVIRONMENT`: (optional) {% include {{ page.version.version }}/misc/azure-env-param.md %}

`AZURE_CLIENT_ID`: Application (client) ID for your [App Registration](https://learn.microsoft.com/azure/active-directory/develop/quickstart-register-app#register-an-application).

`AZURE_CLIENT_SECRET`: Client credentials secret generated for your App Registration.

`AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration.

**Note:** For backward compatibility, schemes `azure://` and `azure-storage://` are also supported, though `azure-blob://` is recommended. Google Cloud Storage | `gs` | Bucket name | `AUTH`: `implicit`, or `specified` (default: `specified`); `CREDENTIALS`

[`ASSUME_ROLE`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-google-cloud-storage-assume-role) (optional): Pass the [service account name](https://cloud.google.com/iam/docs/understanding-service-accounts) of the service account to assume.

For more information, see [Authentication - Google Cloud Storage]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#google-cloud-storage-specified). HTTP | `http` | Remote host | N/A

For more information, see [Authentication - HTTP]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#http-authentication). NFS/Local [1](#considerations) | `nodelocal` | `nodeID` [2](#considerations) (see [Example file URLs](#example-file-urls)) | N/A @@ -64,7 +64,7 @@ Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [` Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure Blob Storage | `azure://acme-co/employees?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` +Azure Blob Storage | `azure-blob://acme-co/employees?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` Google Cloud Storage | `gs://acme-co/employees?AUTH=specified&CREDENTIALS=encoded-123` NFS/Local | `nodelocal://1/path/employees` @@ -75,7 +75,7 @@ Example URLs for [`IMPORT`]({% link {{ page.version.version }}/import.md %}) giv Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure Blob Storage | `azure://acme-co/employees.sql?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` +Azure Blob Storage | `azure-blob://acme-co/employees.sql?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` Google Cloud Storage | `gs://acme-co/employees.sql?AUTH=specified&CREDENTIALS=encoded-123` HTTP | `http://localhost:8080/employees.sql` NFS/Local | `nodelocal://1/path/employees` diff --git a/src/current/v23.2/alter-backup.md b/src/current/v23.2/alter-backup.md index 590c5b64819..2effdb19274 100644 --- a/src/current/v23.2/alter-backup.md +++ b/src/current/v23.2/alter-backup.md @@ -58,7 +58,7 @@ To add a new KMS key to the most recent backup: ALTER BACKUP LATEST IN 's3://{BUCKET NAME}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' ADD NEW_KMS = 'aws:///{new-key}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}®ION={location}' WITH OLD_KMS = 'aws:///{old-key}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}®ION={location}'; -~~~ +~~~ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement that points to the full backup: @@ -67,7 +67,7 @@ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement tha ALTER BACKUP '2022/03/23-213101.37' IN 's3://{BUCKET NAME}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' ADD NEW_KMS = 'aws:///{new-key}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}®ION={location}' WITH OLD_KMS = 'aws:///{old-key}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}®ION={location}'; -~~~ +~~~ To list backup directories at a collection's URI, see [`SHOW BACKUP`]({% link {{ page.version.version }}/show-backup.md %}). @@ -79,19 +79,19 @@ To add a new KMS key to the most recent backup: {% include_cached copy-clipboard.html %} ~~~ sql -ALTER BACKUP LATEST IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +ALTER BACKUP LATEST IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' ADD NEW_KMS = 'azure-kms:///{new key}/{new key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' WITH OLD_KMS = 'azure-kms:///{old key}/{old key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; -~~~ +~~~ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement that points to the full backup: {% include_cached copy-clipboard.html %} ~~~ sql -ALTER BACKUP '2023/03/14-203808.29' IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +ALTER BACKUP '2023/03/14-203808.29' IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' ADD NEW_KMS = 'azure-kms:///{new key}/{new key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' WITH OLD_KMS = 'azure-kms:///{old key}/{old key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; -~~~ +~~~ To list backup directories at a collection's URI, see [`SHOW BACKUP`]({% link {{ page.version.version }}/show-backup.md %}). @@ -104,7 +104,7 @@ To add a new KMS key to the most recent backup: ALTER BACKUP LATEST IN 'gs://{BUCKET NAME}?AUTH=specified&CREDENTIALS={ENCODED KEY}' ADD NEW_KMS = 'gs:///projects/{project name}/locations/{location}/keyRings/{key ring name}/cryptoKeys/{new key}?AUTH=specified&CREDENTIALS={encoded key}' WITH OLD_KMS = 'gs:///projects/{project name}/locations/{location}/keyRings/{key ring name}/cryptoKeys/{old key}?AUTH=specified&CREDENTIALS={encoded key}'; -~~~ +~~~ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement that points to the full backup: @@ -113,7 +113,7 @@ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement tha ALTER BACKUP '2022/03/23-213101.37' IN 'gs://{BUCKET NAME}?AUTH=specified&CREDENTIALS={ENCODED KEY}' ADD NEW_KMS = 'gs:///projects/{project name}/locations/{location}/keyRings/{key ring name}/cryptoKeys/{new key}?AUTH=specified&CREDENTIALS={encoded key}' WITH OLD_KMS = 'gs:///projects/{project name}/locations/{location}/keyRings/{key ring name}/cryptoKeys/{old key}?AUTH=specified&CREDENTIALS={encoded key}'; -~~~ +~~~ To list backup directories at a collection's URI, see [`SHOW BACKUP`]({% link {{ page.version.version }}/show-backup.md %}). diff --git a/src/current/v23.2/cancel-job.md b/src/current/v23.2/cancel-job.md index f399b7ea18a..1b90666d6dc 100644 --- a/src/current/v23.2/cancel-job.md +++ b/src/current/v23.2/cancel-job.md @@ -48,11 +48,11 @@ Parameter | Description > SHOW JOBS; ~~~ ~~~ -+----------------+---------+-------------------------------------------+... -| id | type | description |... -+----------------+---------+-------------------------------------------+... -| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -+----------------+---------+-------------------------------------------+... ++----------------+---------+------------------------------------------------+... +| id | type | description |... ++----------------+---------+------------------------------------------------+... +| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ++----------------+---------+------------------------------------------------+... ~~~ ~~~ sql > CANCEL JOB 27536791415282; diff --git a/src/current/v23.2/changefeed-sinks.md b/src/current/v23.2/changefeed-sinks.md index 5f15fea4900..044d3dc11ce 100644 --- a/src/current/v23.2/changefeed-sinks.md +++ b/src/current/v23.2/changefeed-sinks.md @@ -327,7 +327,7 @@ Examples of supported cloud storage sink URIs: ### Azure Blob Storage ~~~ -'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}' +'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}' ~~~ ### Google Cloud Storage diff --git a/src/current/v23.2/cloud-storage-authentication.md b/src/current/v23.2/cloud-storage-authentication.md index 077d7516b80..50beea2b97d 100644 --- a/src/current/v23.2/cloud-storage-authentication.md +++ b/src/current/v23.2/cloud-storage-authentication.md @@ -702,7 +702,7 @@ You can authenticate to Azure with explicit credentials in the following ways: - `AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration. ~~~ - azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID} + azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID} ~~~ You can authenticate to Azure Storage and Azure Key Vault with this URI format. @@ -715,9 +715,11 @@ You can authenticate to Azure with explicit credentials in the following ways: It is necessary to [URL encode](https://wikipedia.org/wiki/Percent-encoding) the account key since it is base64-encoded and may contain `+`, `/`, `=` characters. ~~~ - azure://{container name}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}&AZURE_ENVIRONMENT=AZUREUSGOVERNMENTCLOUD + azure-blob://{container name}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}&AZURE_ENVIRONMENT=AZUREUSGOVERNMENTCLOUD ~~~ + {% include {{ page.version.version }}/misc/azure-blob.md %} + ## Azure Blob Storage implicit authentication {{site.data.alerts.callout_info}} @@ -798,9 +800,11 @@ To set up `implicit` authentication to Azure Blob Storage (or a KMS resource), y {% include_cached copy-clipboard.html %} ~~~sql - BACKUP DATABASE {database} INTO 'azure://{container name}?AUTH=implicit&AZURE_ACCOUNT_NAME={account name}'; + BACKUP DATABASE {database} INTO 'azure-blob://{container name}?AUTH=implicit&AZURE_ACCOUNT_NAME={account name}'; ~~~ + {% include {{ page.version.version }}/misc/azure-blob.md %} +
diff --git a/src/current/v23.2/pause-job.md b/src/current/v23.2/pause-job.md index 275f21402d9..b361ecefca6 100644 --- a/src/current/v23.2/pause-job.md +++ b/src/current/v23.2/pause-job.md @@ -74,9 +74,9 @@ See the following pages for details on metrics: ~~~ ~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... +-----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ {% include_cached copy-clipboard.html %} diff --git a/src/current/v23.2/resume-job.md b/src/current/v23.2/resume-job.md index 8c2b37494cc..fc71722ede9 100644 --- a/src/current/v23.2/resume-job.md +++ b/src/current/v23.2/resume-job.md @@ -45,9 +45,9 @@ Parameter | Description ~~~ ~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... +-----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ {% include_cached copy-clipboard.html %} diff --git a/src/current/v23.2/show-jobs.md b/src/current/v23.2/show-jobs.md index 88c6ad82ace..12b22f239e6 100644 --- a/src/current/v23.2/show-jobs.md +++ b/src/current/v23.2/show-jobs.md @@ -116,9 +116,9 @@ We recommend monitoring paused jobs to protect historical data from [garbage col ~~~ ~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++---------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ ### Filter jobs @@ -131,9 +131,9 @@ You can filter jobs by using `SHOW JOBS` as the data source for a [`SELECT`]({% ~~~ ~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++---------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ @@ -235,9 +235,9 @@ To block `SHOW JOB` until the provided job ID reaches a terminal state, use `SHO > SHOW JOB WHEN COMPLETE 27536791415282; ~~~ ~~~ - job_id | job_type | description |... -+----------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... + job_id | job_type | description |... ++----------------+-----------+------------------------------------------------+... + 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |... ~~~ ### Show jobs for a schedule diff --git a/src/current/v23.2/take-and-restore-encrypted-backups.md b/src/current/v23.2/take-and-restore-encrypted-backups.md index dabc602dfc0..ebda739f1e5 100644 --- a/src/current/v23.2/take-and-restore-encrypted-backups.md +++ b/src/current/v23.2/take-and-restore-encrypted-backups.md @@ -224,7 +224,7 @@ To take an encrypted backup with Azure KMS, use the `kms` [option]({% link {{ pa {% include_cached copy-clipboard.html %} ~~~ sql -BACKUP INTO 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +BACKUP INTO 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH kms = 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; ~~~ @@ -234,7 +234,7 @@ To take a [locality-aware backup with kms encryption](#locality-aware-backup-wit {% include_cached copy-clipboard.html %} ~~~ sql -BACKUP INTO 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +BACKUP INTO 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH KMS=( 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}', 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' @@ -247,7 +247,7 @@ To view a [kms encrypted locality-aware backup](#use-key-management-service), us {% include_cached copy-clipboard.html %} ~~~ sql -SHOW BACKUP FROM '2023/07/14-211406.03' IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +SHOW BACKUP FROM '2023/07/14-211406.03' IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH KMS=( 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}', 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}' @@ -286,7 +286,7 @@ To decrypt an [encrypted backup](#take-an-encrypted-azure-blob-storage-backup), {% include_cached copy-clipboard.html %} ~~~ sql -RESTORE FROM LATEST IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' +RESTORE FROM LATEST IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}' WITH kms = 'azure-kms:///{key}/{key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'; ~~~ diff --git a/src/current/v23.2/take-and-restore-locality-aware-backups.md b/src/current/v23.2/take-and-restore-locality-aware-backups.md index 44e2e680470..b65bd119ba5 100644 --- a/src/current/v23.2/take-and-restore-locality-aware-backups.md +++ b/src/current/v23.2/take-and-restore-locality-aware-backups.md @@ -208,7 +208,7 @@ The `system.zones` table stores your cluster's [zone configurations]({% link {{ {% include_cached copy-clipboard.html %} ~~~ sql RESTORE TABLE system.zones FROM '2021/03/23-213101.37' IN - 'azure://acme-co-backup?AZURE_ACCOUNT_KEY=hash&AZURE_ACCOUNT_NAME=acme-co' + 'azure-blob://acme-co-backup?AZURE_ACCOUNT_KEY=hash&AZURE_ACCOUNT_NAME=acme-co' WITH into_db = 'newdb'; ~~~ diff --git a/src/current/v23.2/use-cloud-storage.md b/src/current/v23.2/use-cloud-storage.md index 34191b9ab78..e1cad97f69a 100644 --- a/src/current/v23.2/use-cloud-storage.md +++ b/src/current/v23.2/use-cloud-storage.md @@ -35,7 +35,7 @@ The following table provides a list of the parameters supported by each storage Location | Scheme | Host | Parameters ------------------------------------------------------------+-------------+--------------------------------------------------+---------------------------------------------------------------------------- Amazon S3 | `s3` | Bucket name | [`AUTH`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#amazon-s3-specified): `implicit` or `specified` (default: `specified`). When using `specified` pass user's `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

[`ASSUME_ROLE`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) (optional): Pass the [ARN](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) of the role to assume. Use in combination with `AUTH=implicit` or `specified`.

[`AWS_SESSION_TOKEN`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}) (optional): For more information, see Amazon's guide on [temporary credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).

[`S3_STORAGE_CLASS`](#amazon-s3-storage-classes) (optional): Specify the Amazon S3 storage class for created objects. Note that Glacier Flexible Retrieval and Glacier Deep Archive are not compatible with incremental backups. **Default**: `STANDARD`. -Azure Blob Storage | `azure` | Storage container | `AZURE_ACCOUNT_NAME`: The name of your Azure account.

`AZURE_ACCOUNT_KEY`: Your Azure account key. You must [url encode](https://wikipedia.org/wiki/Percent-encoding) your Azure account key before authenticating to Azure Storage. For more information, see [Authentication - Azure Storage]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#azure-blob-storage-specified-authentication).

`AZURE_ENVIRONMENT`: (optional) {% include {{ page.version.version }}/misc/azure-env-param.md %}

`AZURE_CLIENT_ID`: Application (client) ID for your [App Registration](https://learn.microsoft.com/azure/active-directory/develop/quickstart-register-app#register-an-application).

`AZURE_CLIENT_SECRET`: Client credentials secret generated for your App Registration.

`AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration. +Azure Blob Storage | `azure-blob` | Storage container | `AZURE_ACCOUNT_NAME`: The name of your Azure account.

`AZURE_ACCOUNT_KEY`: Your Azure account key. You must [url encode](https://wikipedia.org/wiki/Percent-encoding) your Azure account key before authenticating to Azure Storage. For more information, see [Authentication - Azure Storage]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#azure-blob-storage-specified-authentication).

`AZURE_ENVIRONMENT`: (optional) {% include {{ page.version.version }}/misc/azure-env-param.md %}

`AZURE_CLIENT_ID`: Application (client) ID for your [App Registration](https://learn.microsoft.com/azure/active-directory/develop/quickstart-register-app#register-an-application).

`AZURE_CLIENT_SECRET`: Client credentials secret generated for your App Registration.

`AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration.

**Note:** For backward compatibility, schemes `azure://` and `azure-storage://` are also supported, though `azure-blob://` is recommended. Google Cloud Storage | `gs` | Bucket name | `AUTH`: `implicit`, or `specified` (default: `specified`); `CREDENTIALS`

[`ASSUME_ROLE`]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-google-cloud-storage-assume-role) (optional): Pass the [service account name](https://cloud.google.com/iam/docs/understanding-service-accounts) of the service account to assume.

For more information, see [Authentication - Google Cloud Storage]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#google-cloud-storage-specified). HTTP | `file-http(s)` / `http(s)` | Remote host | N/A

**Note:** Using `http(s)` without the `file-` prefix is deprecated as a [changefeed sink]({% link {{ page.version.version }}/changefeed-sinks.md %}) scheme. There is continued support for `http(s)`, but it will be removed in a future release. We recommend implementing the `file-http(s)` scheme for changefeed messages.

For more information, refer to [Authentication - HTTP]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#http-authentication). NFS/Local [1](#considerations) | `nodelocal` | `nodeID` [2](#considerations) (see [Example file URLs](#example-file-urls)) | N/A @@ -64,7 +64,7 @@ Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [` Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure Blob Storage | `azure://acme-co/employees?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` +Azure Blob Storage | `azure-blob://acme-co/employees?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` Google Cloud Storage | `gs://acme-co/employees?AUTH=specified&CREDENTIALS=encoded-123` NFS/Local | `nodelocal://1/path/employees` @@ -75,7 +75,7 @@ Example URLs for [`IMPORT`]({% link {{ page.version.version }}/import.md %}) and Location | Example -------------+---------------------------------------------------------------------------------- Amazon S3 | `s3://acme-co/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure Blob Storage | `azure://acme-co/employees.sql?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` +Azure Blob Storage | `azure-blob://acme-co/employees.sql?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}` Google Cloud Storage | `gs://acme-co/employees.sql?AUTH=specified&CREDENTIALS=encoded-123` HTTP | `http://localhost:8080/employees.sql` NFS/Local | `nodelocal://1/path/employees`