Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow lambda security group access to cache security group in CDK stack #109

Merged
merged 2 commits into from
Jan 25, 2021
Merged

Allow lambda security group access to cache security group in CDK stack #109

merged 2 commits into from
Jan 25, 2021

Conversation

leothomas
Copy link
Contributor

@leothomas leothomas commented Jan 25, 2021 •
edited
Loading

Closes #108

What I did:

Granted the lambda's security group access to the cache's security group through CDK.

How I did it:

I created a custom security group in the lambda function's VPC, and directly attached this security group to the (using the security_groups:[] property of the aws_lambda.Function class - this skips the automatic creation a default security group in the VPC.

I was then able to grant the lambda 's custom security group access to the cache security group (using the ec2.SecurityGroup.add_ingress_rule() property) so that the lambda function has access to the cache.

How you can test it:

Running this code snippet for the first time:

for e in ["", "global", "du", "be", "tk","la", "sf", "ny", "gl"]:
    start = time.time()
    r = re.get(f"https://08dvkxus0a.execute-api.us-east-1.amazonaws.com/v1/datasets/{e}") # staging url 
    end = time.time()
    print(f"Requested: /{e}, response time: {str(end-start)}, Response headers: {r.headers}")

produces the following output:

Requested: / , response time: 4.346861839294434, Response headers: {'Date': 'Mon, 25 Jan 2021 15:43:59 GMT', 'Content-Type': 'application/json', 'Content-Length': '4450', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkR5igsIAMEVFQ='}

Requested: /global, response time: 0.15545296669006348, Response headers: {'Date': 'Mon, 25 Jan 2021 15:43:59 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkR8jvLoAMEWDw='}

Requested: /du, response time: 0.7655160427093506, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:00 GMT', 'Content-Type': 'application/json', 'Content-Length': '2264', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkR9j87IAMEVGQ='}

Requested: /be, response time: 0.7159359455108643, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '2501', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSFjwEIAMEWDw='}

Requested: /tk, response time: 0.5140988826751709, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '3190', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSMjyqIAMEVcA='}

Requested: /la, response time: 0.6045031547546387, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:02 GMT', 'Content-Type': 'application/json', 'Content-Length': '2728', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSRgmfIAMEVQg='}

Requested: /sf, response time: 0.6608572006225586, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:02 GMT', 'Content-Type': 'application/json', 'Content-Length': '3148', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSYifeIAMEVaQ='}

Requested: /ny, response time: 0.6407091617584229, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:03 GMT', 'Content-Type': 'application/json', 'Content-Length': '3047', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSejJQIAMEVTQ='}

Requested: /gl, response time: 0.39092206954956055, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:03 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSkgQroAMEVew='}

Notice that:

  • Each spotlight endpoint responds in ~0.5-0.6 seconds
  • The full /datasets request takes ~4 seconds

Running the code snippet a second time produces:

Requested: / , response time: 0.19382977485656738, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '4450', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTajsqIAMEVEg='}

Requested: /global, response time: 0.11775016784667969, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTchDioAMEVXw='}

Requested: /du, response time: 0.1087641716003418, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '2264', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTdgszoAMEVQg='}

Requested: /be, response time: 0.11829781532287598, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '2501', 'Connection': 'keep-alive', 'vary': 'Accept-Encoding', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'Apigw-Requestid': 'ZtkTej5gIAMEVcA='}

Requested: /tk, response time: 0.12758183479309082, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '3190', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTfgSvoAMEVLw='}

Requested: /la, response time: 0.11828804016113281, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '2728', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTgixRIAMEVSQ='}

Requested: /sf, response time: 0.1327040195465088, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '3148', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkThicDoAMEVMQ='}

Requested: /ny, response time: 0.10931611061096191, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '3047', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTjj8LIAMEV2A='}

Requested: /gl, response time: 0.12949299812316895, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:10 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTkhMYIAMEVXg='}

Notice that:

  • Full /datasets request takes ~0.2 seconds
  • Each spotlight takes ~0.1 seconds
  • Each response headers object includes 'X-Cache': 'HIT'

**Note: **

  • I separated the vpc_access_policy_statement statement into vpc_log_access and vpc_ec2_access for clarity
  • I renamed iam_policy_statement to s3_full_access_to_data_bucket for clarity

@leothomas leothomas added the bug Something isn't working label Jan 25, 2021
@leothomas leothomas self-assigned this Jan 25, 2021
drewbo
drewbo suggested changes Jan 25, 2021
View reviewed changes
Copy link
Contributor

@drewbo drewbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caching implementation look exactly right. Made one small comment on the naming improvement

stack/app.py Outdated Show resolved Hide resolved
@leothomas leothomas requested a review from drewbo January 25, 2021 17:31
@drewbo drewbo merged commit 53b309b into develop Jan 25, 2021
@drewbo drewbo deleted the bugfix/grant-lambda-access-to-cache-security-group branch January 25, 2021 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drewbo drewbo drewbo approved these changes

@olafveerman olafveerman Awaiting requested review from olafveerman

Assignees

@leothomas leothomas

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Include all cache configuration in CDK
2 participants
@leothomas @drewbo