Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[operation]Make the S3 buckets shentu-data-snapshot-full and shentu-data-snapshot-light public accessible #733

Open
haozhan9 opened this issue Jul 21, 2023 · 1 comment
Assignees
Labels
P2 Regular; Update monthly

Comments

@haozhan9
Copy link
Contributor

Since the fixing of AWS findings, these two buckets are not simply public available any more. This issue is created to trace the work of making them public accessible through the cloudfront. The recommendation from platform guy is pasted below:

You shouldn't be connecting to the s3 endpoint directly even for public content. For public content you should set up:
Cloudfront -> [Resource policy to restrict Cloudfront via OAC] S3 bucket
There are a lot of advantages to fronting your public content via Cloudfront instead of going directly to the bucket, I'll just list off a few:
This proves that you made a conscious decision to make your content public. It is hard for the security team to keep up with what buckets should be public and what should not be (just in Certik we have 36 AWS accounts) so AWS recommends blocking all public access to s3 and enforcing everyone to do this extra step proving they want their content to be public
Cloudfront is a CDN, this will cache your objects on edge nodes making end user experience better by access objects faster
In the future, security team might use web application firewall (WAF) to do things like prevent access from known abuse IPs (ie comprised machines). It is best practice for all content exposed to the internet to be filtered by WAF. We cannot put a WAF in front of the S3 public endpoint but can do it in front of Cloudfront
Informational: Client -> Cloudfront -> S3 connection path is free so there is no cost concern
S3 does not allow custom domain but cloudfront allows you to add a custom domain, which you should be using. In a Organization, we should never use domains in production that we don't own (ie .s3.region.amazonaws.com or .cloudfront.net ). Say you want to go multi-region, move how your data is served, or need to move your data to a different AWS account; you don't want to be tied down the a domain you cannot move because then you need to tell everyone, since this is public content, to change the URL they are using. You should use a custom domain, which by the way every account has a domain it can use, to just repoint to the new location so your end user is not affected.

@haozhan9 haozhan9 added the P2 Regular; Update monthly label Jul 21, 2023
@skyargos skyargos self-assigned this Aug 16, 2023
@skyargos
Copy link
Contributor

Here is the operation process given by AWS.

Url for snapshot-light
https://d35kzm6d1vecp9.cloudfront.net/shentu.tar.gz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 Regular; Update monthly
Projects
None yet
Development

No branches or pull requests

2 participants