From 8994a870c7d33808bf42b5dad25779eb5b892828 Mon Sep 17 00:00:00 2001 From: Bryce Groff Date: Wed, 15 May 2024 16:49:40 -0700 Subject: [PATCH] Update the documentation to reflect the new STS Assume Role auth. --- docs/integrations/destinations/s3.md | 160 +++++++++++++++++++++------ 1 file changed, 125 insertions(+), 35 deletions(-) diff --git a/docs/integrations/destinations/s3.md b/docs/integrations/destinations/s3.md index 89fa171541c8..e726d3f25298 100644 --- a/docs/integrations/destinations/s3.md +++ b/docs/integrations/destinations/s3.md @@ -6,15 +6,27 @@ This page guides you through the process of setting up the S3 destination connec List of required fields: -- **Access Key ID** -- **Secret Access Key** - **S3 Bucket Name** - **S3 Bucket Path** - **S3 Bucket Region** +If you are using STS Assume Role, you must provide the following: + +- **Role ARN** + +Otherwise, if you are using AWS credentials you must provide the following: + +- **Access Key ID** +- **Secret Access Key** + +If you are using an Instance Profile, you may omit the Access Key ID and Secret Access Key, +as well as, the Role ARN. + +Additionally the following prerequisites are required: + 1. Allow connections from Airbyte server to your AWS S3/ Minio S3 cluster \(if they exist in separate VPCs\). -2. An S3 bucket with credentials or an instance profile with read/write permissions configured for +2. An S3 bucket with credentials, a Role ARN, or an instance profile with read/write permissions configured for the host (ec2, eks). 3. [Enforce encryption of data in transit](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html#transit) @@ -22,8 +34,7 @@ List of required fields: ### Step 1: Set up S3 -[Sign in](https://console.aws.amazon.com/iam/) to your AWS account. Use an existing or create new -[Access Key ID and Secret Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#:~:text=IAM%20User%20Guide.-,Programmatic%20access,-You%20must%20provide). +[Sign in](https://console.aws.amazon.com/iam/) to your AWS account. Prepare S3 bucket that will be used as destination, see [this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create @@ -34,6 +45,109 @@ to an unencrypted connection. Airbyte recommends all connections be configured t support for AWS's [shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/) +#### Create bucket a Policy +1. Open the [IAM console](https://console.aws.amazon.com/iam/home#home). +2. In the IAM dashboard, select **Policies**, then click **Create Policy**. +3. Select the **JSON** tab, then paste the following JSON into the Policy editor (be sure to substitute in your bucket name): +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:PutObjectAcl", + "s3:ListBucket", + "s3:ListBucketMultipartUploads", + "s3:AbortMultipartUpload", + "s3:GetBucketLocation" + ], + "Resource": [ + "arn:aws:s3:::YOUR_BUCKET_NAME/*", + "arn:aws:s3:::YOUR_BUCKET_NAME" + ] + } + ] +} +``` + +:::note +At this time, object-level permissions alone are not sufficient to successfully authenticate the connection. Please ensure you include the **bucket-level** permissions as provided in the example above. +::: + +4. Give your policy a descriptive name, then click **Create policy**. + +#### Authentication Option 1: Using an IAM Role (Most secure) + + +:::note +This authentication method is currently in the testing phase. To enable it for your workspace, please contact our Support Team. +::: + + +1. In the IAM dashboard, click **Roles**, then **Create role**. +2. Choose the appropriate trust entity and attach the policy you created. +3. Set up a trust relationship for the role. For example for **AWS account** trusted entity use default AWS account on your instance (it will be used to assume role). To use **External ID** set it to environment variables as `export AWS_ASSUME_ROLE_EXTERNAL_ID="{your-external-id}"`. Edit the trust relationship policy to reflect this: +``` +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::{your-aws-account-id}:user/{your-username}" + }, + "Action": "sts:AssumeRole", + "Condition": { + "StringEquals": { + "sts:ExternalId": "{your-external-id}" + } + } + } + ] +} +``` + + +2. Choose the **AWS account** trusted entity type. +3. Set up a trust relationship for the role. This allows the Airbyte instance's AWS account to assume this role. You will also need to specify an external ID, which is a secret key that the trusting service (Airbyte) and the trusted role (the role you're creating) both know. This ID is used to prevent the "confused deputy" problem. The External ID should be your Airbyte workspace ID, which can be found in the URL of your workspace page. Edit the trust relationship policy to include the external ID: +``` +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::094410056844:user/delegated_access_user" + }, + "Action": "sts:AssumeRole", + "Condition": { + "StringEquals": { + "sts:ExternalId": "{your-airbyte-workspace-id}" + } + } + } + ] +} +``` + +4. Complete the role creation and note the Role ARN. +5. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**. + + +##### Authentication Option 2: Using an IAM User + +Use an existing or create new +[Access Key ID and Secret Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#:~:text=IAM%20User%20Guide.-,Programmatic%20access,-You%20must%20provide). + +1. In the IAM dashboard, click **Users**. Select an existing IAM user or create a new one by clicking **Add users**. +2. If you are using an _existing_ IAM user, click the **Add permissions** dropdown menu and select **Add permissions**. If you are creating a _new_ user, you will be taken to the Permissions screen after selecting a name. +3. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**. +4. After successfully creating your user, select the **Security credentials** tab and click **Create access key**. You will be prompted to select a use case and add optional tags to your access key. Click **Create access key** to generate the keys. + ### Step 2: Set up the S3 destination connector in Airbyte @@ -55,6 +169,10 @@ support for AWS's to objects in the bucket. - **Secret Access Key** - Corresponding key to the above key id. + - **Role ARN** + - See + [this](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) + on how to create a role. - **S3 Bucket Name** - See [this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create an S3 bucket. @@ -95,7 +213,8 @@ support for AWS's will require [read and write permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html) to objects in the staging bucket. _ If the Access Key and Secret Access Key are not provided, the - authentication will rely on the instanceprofile. _ **Secret Access Key** _ Corresponding key to + authentication will rely either on the Role ARN using STS Assume Role or on the instanceprofile. +5. _ **Secret Access Key** _ Corresponding key to the above key id. _ Make sure your S3 bucket is accessible from the machine running Airbyte. _ This depends on your networking setup. _ You can check AWS S3 documentation with a tutorial on how to properly configure your S3's access @@ -120,35 +239,6 @@ support for AWS's 5. Click `Set up destination`. -In order for everything to work correctly, it is also necessary that the user whose "S3 Key Id" and -"S3 Access Key" are used have access to both the bucket and its contents. Minimum required Policies -to use: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:DeleteObject", - "s3:PutObjectAcl", - "s3:ListBucket", - "s3:ListBucketMultipartUploads", - "s3:AbortMultipartUpload", - "s3:GetBucketLocation" - ], - "Resource": [ - "arn:aws:s3:::YOUR_BUCKET_NAME/*", - "arn:aws:s3:::YOUR_BUCKET_NAME" - ] - } - ] -} -``` - The full path of the output data with the default S3 Path Format `${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_` is: