How to maintaine the Bucket when using new Endpoint spec

[hantsy]

In the newest AWS SDK Kotlin, it prefers the new Endpoint spec which includes bucket in the endpoint url.

In AWS SDk Kotlin examples, you can set a bucket param to check the existences, or create a bucket before get/list/put/delete operations.

In the new Endpoint, it bound bucket in the url, that the bucket should be existed firstly or it will create a bucket automatically?

[lucix-aws]

Hi @hantsy--

Writing to hopefully provide some clarification w.r.t. endpoint resolution and how the SDK resolves endpoints.

When working with custom endpoints, S3 can be a bit confusing due to its feature-richness and the way those features affect endpoint resolution. When performing operations against live S3, the user will generally not be setting their own endpoint, thus the considerations here w.r.t. resolution are generally not at play for them.

That said, with respect to the following statement:

In the newest AWS SDK Kotlin, it prefers the new Endpoint spec which includes bucket in the endpoint url.

I'd like to clarify that this is not the case--when setting a custom endpoint in client config (config.endpointUrl), you do not need (and should not) include the bucket in the url. The default endpoint provider will receive your custom url from client config, and the bucket name from the request input, and determine how to represent the bucket name in the final endpoint. Note that if you do this, you should leave the endpointProvider setting on the client config as the default.

For S3 specifically, the bucket will end up as part of the endpoint URL in one of two primary ways:

1. virtualhost-style e.g. bucket-name.s3-api.com
2. path-style e.g. s3-api.com/bucket-name

The virtualhost form is somewhat newer and we can only guarantee that it (or any feature) will work as expected against live S3. We only recently gained virtualhost support in the Kotlin SDK. I suspect that this relates to the core issue you were experiencing in #842 - I can't confirm, but my educated guess is that with your custom URL, we were resolving to a virtualhost-style endpoint, which wasn't working with your localstack deployment, whereas forcing path-style resolution was.

For that reason, going forward I do recommend you set forcePathStyle to true when pointing to a non-live S3 implementation.

Beyond that, the S3 API behaves as it always has - you must explicitly create a bucket before performing any storage operations on it.

[lucix-aws]

With regards to this topic, we recently published a new developer guide entry for endpoint configuration: https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/config-endpoint.html.