Reducing deploy times for new products

[automatthew]

Deploys of an API backend to production currently include directives in sky.yaml to set up caching using CloudFront.

Settng up a custom CloudFront distribution adds about 25 minutes to sky publish <env>. This makes it onerous to troubleshoot when a deploy fails during or after the CloudFront caching setup.

The impact of this is limited though, as the only CloudFormation Resource in our stack (at this time) that depends on the CloudFront distribution is Route53, for DNS. (Of course we may add other dependent resources in the future)

Thus for existing Labs products, the risk is limited if we are not changing the DNS names or beginning to make use of new AWS cloud services.

For first production deploys of new products, however, or when we add a new CloudFormation Resource to the stack (Cognito, SQS, etc.), setting up a custom CloudFront cache can make the process more frustrating.

But do products in the nest, or even fledglings, need custom caching? Do their APIs even need to be served on our custom subdomains? We do need the APIs served over HTTPS, but this (as well as caching) is already handled for us by API Gateway defaults:

https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-custom-domains.html

After deploying your API, you (and your customers) can invoke the API using the default root URL of the following format:
https://api-id.execute-api.region.amazonaws.com/stage

For every API you create, API Gateway sets up an Amazon CloudFront distribution for the API. Requests with the default API URL are routed through the corresponding CloudFront distribution. Similarly, for every custom domain name, API Gateway sets up a CloudFront distribution. An API request with the custom domain name is routed through the custom domain name's CloudFront distribution.

The former case, where we just use the default domain, takes only 3-4 minutes to deploy. We haven't tried the latter yet (custom domain with API Gateway directly), so we don't know how it will behave. Perhaps the 30 minute time for new custom CloudFront distributions will also happen with custom API Gateway domains.

My recommendation is that we use the default API Gateway URLs for every product, in every environment, until we have a need to make public the URL for a particular HTTP API.

At the time we do want to use a custom domain name for an API, we should use the API Gateway feature for this directly, instead of CloudFront.

Future

We should also determine if there's a way to use our own, existing CloudFront distro to serve production API without tearing everything down and building it again. Or when this is and isn't desirable.