Skip to content

Commit

Permalink
Update the documentation to reflect the new STS Assume Role auth.
Browse files Browse the repository at this point in the history
  • Loading branch information
@bgroff
bgroff committed May 15, 2024
1 parent 0870cd1 commit 8994a87
Showing 1 changed file with 125 additions and 35 deletions.
160 changes: 125 additions & 35 deletions docs/integrations/destinations/s3.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,35 @@ This page guides you through the process of setting up the S3 destination connec

List of required fields:

- **Access Key ID**
- **Secret Access Key**
- **S3 Bucket Name**
- **S3 Bucket Path**
- **S3 Bucket Region**

If you are using STS Assume Role, you must provide the following:

- **Role ARN**

Otherwise, if you are using AWS credentials you must provide the following:

- **Access Key ID**
- **Secret Access Key**

If you are using an Instance Profile, you may omit the Access Key ID and Secret Access Key,
as well as, the Role ARN.

Additionally the following prerequisites are required:

1. Allow connections from Airbyte server to your AWS S3/ Minio S3 cluster \(if they exist in
separate VPCs\).
2. An S3 bucket with credentials or an instance profile with read/write permissions configured for
2. An S3 bucket with credentials, a Role ARN, or an instance profile with read/write permissions configured for
the host (ec2, eks).
3. [Enforce encryption of data in transit](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html#transit)

## Setup guide

### Step 1: Set up S3

[Sign in](https://console.aws.amazon.com/iam/) to your AWS account. Use an existing or create new
[Access Key ID and Secret Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#:~:text=IAM%20User%20Guide.-,Programmatic%20access,-You%20must%20provide).
[Sign in](https://console.aws.amazon.com/iam/) to your AWS account.

Prepare S3 bucket that will be used as destination, see
[this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create
Expand All @@ -34,6 +45,109 @@ to an unencrypted connection. Airbyte recommends all connections be configured t
support for AWS's
[shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/)

#### Create bucket a Policy
1. Open the [IAM console](https://console.aws.amazon.com/iam/home#home).
2. In the IAM dashboard, select **Policies**, then click **Create Policy**.
3. Select the **JSON** tab, then paste the following JSON into the Policy editor (be sure to substitute in your bucket name):
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObjectAcl",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
}
]
}
```

:::note
At this time, object-level permissions alone are not sufficient to successfully authenticate the connection. Please ensure you include the **bucket-level** permissions as provided in the example above.
:::

4. Give your policy a descriptive name, then click **Create policy**.

#### Authentication Option 1: Using an IAM Role (Most secure)

<!-- env:cloud -->
:::note
This authentication method is currently in the testing phase. To enable it for your workspace, please contact our Support Team.
:::
<!-- /env:cloud -->

1. In the IAM dashboard, click **Roles**, then **Create role**. <!-- env:oss -->
2. Choose the appropriate trust entity and attach the policy you created.
3. Set up a trust relationship for the role. For example for **AWS account** trusted entity use default AWS account on your instance (it will be used to assume role). To use **External ID** set it to environment variables as `export AWS_ASSUME_ROLE_EXTERNAL_ID="{your-external-id}"`. Edit the trust relationship policy to reflect this:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{your-aws-account-id}:user/{your-username}"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "{your-external-id}"
}
}
}
]
}
```
<!-- /env:oss -->
<!-- env:cloud -->
2. Choose the **AWS account** trusted entity type.
3. Set up a trust relationship for the role. This allows the Airbyte instance's AWS account to assume this role. You will also need to specify an external ID, which is a secret key that the trusting service (Airbyte) and the trusted role (the role you're creating) both know. This ID is used to prevent the "confused deputy" problem. The External ID should be your Airbyte workspace ID, which can be found in the URL of your workspace page. Edit the trust relationship policy to include the external ID:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::094410056844:user/delegated_access_user"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "{your-airbyte-workspace-id}"
}
}
}
]
}
```
<!-- /env:cloud -->
4. Complete the role creation and note the Role ARN.
5. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**.


##### Authentication Option 2: Using an IAM User

Use an existing or create new
[Access Key ID and Secret Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#:~:text=IAM%20User%20Guide.-,Programmatic%20access,-You%20must%20provide).

1. In the IAM dashboard, click **Users**. Select an existing IAM user or create a new one by clicking **Add users**.
2. If you are using an _existing_ IAM user, click the **Add permissions** dropdown menu and select **Add permissions**. If you are creating a _new_ user, you will be taken to the Permissions screen after selecting a name.
3. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**.
4. After successfully creating your user, select the **Security credentials** tab and click **Create access key**. You will be prompted to select a use case and add optional tags to your access key. Click **Create access key** to generate the keys.

### Step 2: Set up the S3 destination connector in Airbyte

<!-- env:cloud -->
Expand All @@ -55,6 +169,10 @@ support for AWS's
to objects in the bucket.
- **Secret Access Key**
- Corresponding key to the above key id.
- **Role ARN**
- See
[this](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html)
on how to create a role.
- **S3 Bucket Name**
- See [this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)
to create an S3 bucket.
Expand Down Expand Up @@ -95,7 +213,8 @@ support for AWS's
will require
[read and write permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html)
to objects in the staging bucket. _ If the Access Key and Secret Access Key are not provided, the
authentication will rely on the instanceprofile. _ **Secret Access Key** _ Corresponding key to
authentication will rely either on the Role ARN using STS Assume Role or on the instanceprofile.
5. _ **Secret Access Key** _ Corresponding key to
the above key id. _ Make sure your S3 bucket is accessible from the machine running Airbyte. _
This depends on your networking setup. _ You can check AWS S3 documentation with a tutorial on
how to properly configure your S3's access
Expand All @@ -120,35 +239,6 @@ support for AWS's

5. Click `Set up destination`.

In order for everything to work correctly, it is also necessary that the user whose "S3 Key Id" and
"S3 Access Key" are used have access to both the bucket and its contents. Minimum required Policies
to use:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObjectAcl",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
}
]
}
```

The full path of the output data with the default S3 Path Format
`${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_` is:

Expand Down

0 comments on commit 8994a87

Please sign in to comment.