Add ApiGatewayHttpApiV2ProxyRequestTranslator and ApiGatewayProxyRequestTranslator #1901

gcbeattyAWS · 2024-12-09T15:12:39Z

DOTNET-7838

Description of changes:

Add functions to translate httpcontext into APIGatewayHttpApiV2ProxyRequest and APIGatewayProxyRequest
Add unit tests for above classes

Testing details

Wrote unit tests
Manually deployed a lambda into my aws account, and wrote a console app that uses my request translator and invokes the function. validated that the request fields seem to be sent correctly

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/ApiGatewayHttpApiV2ProxyRequestTranslator.cs

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/ApiGatewayProxyRequestTranslator.cs

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/ApiGatewayRequestTranslatorFactory.cs

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/IRouteConfigurationParser.cs

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/HttpContextExtensions.cs

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/HttpContextExtensions.cs

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/ExceptionExtensions.cs

normj

I don't see any tests for special characters in headers, resource paths or query strings. Have we done a comparison yet with API Gateway? We should have tests for those cases. That includes the RawPath property as well.

gcbeattyAWS · 2024-12-11T01:18:24Z

I don't see any tests for special characters in headers, resource paths or query strings. Have we done a comparison yet with API Gateway? We should have tests for those cases. That includes the RawPath property as well.

so i did some tests of checking the existing functionality of api gateway in production, with special characters in the header by

deploying an api gateway
deploying a lambda
attaching the lambda to the api gatway
invoking the api gateway via command line (curl), and sending a non url encoded header value of special+++
logging the event (apigatewayrequest) object in the lambda itself, and it prints out special+++ (not encoded), so i assumed no encoding is required for that - but let me double check on this and get back to you

normj · 2024-12-11T01:20:24Z

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Utilities/HttpRequestUtility.cs

+        if (string.IsNullOrEmpty(contentType))
+            return false;
+
+        return contentType.StartsWith("image/", StringComparison.OrdinalIgnoreCase) ||


We have this list of content types we use for response in the ASP.NET Core bridge library. https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.AspNetCoreServer/AbstractAspNetCoreFunction.cs#L59

We should also think about how this can be configurable. Probably via environment variables that we can have users configure in Aspire.

normj · 2024-12-11T01:21:53Z

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/HttpContextExtensions.cs

+            Headers = headers,
+            QueryStringParameters = queryStringParameters,
+            PathParameters = pathParameters ?? new Dictionary<string, string>(),
+            Body = HttpRequestUtility.ReadRequestBody(request),


Can you take note that when we get these ToAPIGatewayXXX() methods hooked up to the main code path that is asking for the APIGatewayHttpApiV2ProxyRequest. When we convert the APIGatewayHttpApiV2ProxyRequest to the MemoryStream to Lambda we should check the size of the stream is no more then 6MB. If so throw back the same error that users would get from API Gateway.

yeah i will connect with phil on where he wants to do this logic at

normj · 2024-12-11T01:29:56Z

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/HttpContextExtensions.cs

+            MultiValueHeaders = multiValueHeaders,
+            QueryStringParameters = queryStringParameters,
+            MultiValueQueryStringParameters = multiValueQueryStringParameters,
+            PathParameters = pathParameters ?? new Dictionary<string, string>(),


Why are we initializing this to an empty collection if we don't have any. Is this what API Gateway does?

good catch. i just did a test
Invoke-WebRequest -Uri https://myapi-gateway-url.amazonaws.com/testfunction -Method GET
where it echos the api gateway request object and i see the value is null when there are no path parameters.

pathParameters: null, stageVariables: null, body: null, isBase64Encoded: false }

i will update the code

ill also double check the query string and header default values

ive updated the logic to handle this now

normj · 2024-12-11T01:33:06Z

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Utilities/HttpRequestUtility.cs

+
+        foreach (var header in headers)
+        {
+            singleValueHeaders[header.Key] = header.Value.Last() ?? "";


My memory for API Gateway is that in HTTP V2 where there is just a single headers collection the headers are comma delimited. But we need to verify that. I'm sure it isn't just get the last value.

yeah you are right on this https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html

Format 2.0 doesn't have multiValueHeaders or multiValueQueryStringParameters fields. Duplicate headers are combined with commas and included in the headers field. Duplicate query strings are combined with commas and included in the queryStringParameters field.

i think i got confused when reading the other docs for v1 https://aws.amazon.com/blogs/compute/support-for-multi-value-parameters-in-amazon-api-gateway/

Before this change, API Gateway used to retain only the last value and drop everything else for a multi-valued parameter. You can see the original behavior in the queryStringParameters parameter in the above input, where only the “fish” value is retained.

I will update accordingly

ive updated the logic to handle this now. this logic will still just return the dicts as-is but then i have the caller perform the comma separated merging.

normj · 2024-12-11T01:34:39Z

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Utilities/HttpRequestUtility.cs

+
+        foreach (var param in query)
+        {
+            singleValueParams[param.Key] = param.Value.Last() ?? "";


Same comment as above for ExtractHeaders

ive updated the logic to handle this now

normj · 2024-12-11T01:57:33Z

4. invoking the api gateway via command line (curl), and sending a non url encoded header value of special+++

@gcbeattyAWS Also check more characters then + like spaces and other white space and Unicode characters. resource path and query strings can also be special. Especially when you through /, & and `?' in the values for resource path segments and query string values. In the ASP.NET Core Lambda bridge library where I have to do all of this in reverse I had to do some encoding handling.

aws-lambda-dotnet/Libraries/src/Amazon.Lambda.AspNetCoreServer/Internal/Utilities.cs

Line 148 in 3596a58

    
           internal static string CreateQueryStringParameters(IDictionary<string, string> singleValues, IDictionary<string, IList<string>> multiValues, bool urlEncodeValue)

gcbeattyAWS · 2024-12-12T17:08:22Z

invoking the api gateway via command line (curl), and sending a non url encoded header value of special+++

@gcbeattyAWS Also check more characters then + like spaces and other white space and Unicode characters. resource path and query strings can also be special. Especially when you through /, & and `?' in the values for resource path segments and query string values. In the ASP.NET Core Lambda bridge library where I have to do all of this in reverse I had to do some encoding handling.

aws-lambda-dotnet/Libraries/src/Amazon.Lambda.AspNetCoreServer/Internal/Utilities.cs

Line 148 in 3596a58

internal static string CreateQueryStringParameters(IDictionary<string, string> singleValues, IDictionary<string, IList<string>> multiValues, bool urlEncodeValue)

so i have updated the logic based on my findings:1

For query strings. QueryStringParameters should be the decoded values (this already happens automatically when accessing Request.QueryString. RawQueryString should be the encoded query string without the ?.
Headers do not seem to have any additional logic of decoding that i have seen. they are just a pass-through to lambda.
The path and path parameters should be decoded (this is also happening automatically now by accessing via request.Path)
The api gatway cannot handle %2f in the urls ("/") character

i have added test cases to check all of these scenarios

gcbeattyAWS · 2024-12-12T18:09:15Z

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/HttpContextExtensions.cs

+
+        if (request.Cookies.Any())
+        {
+            httpApiV2ProxyRequest.Cookies = request.Cookies.Select(c => $"{c.Key}={c.Value}").ToArray();


headers and cookies are just sent as-is (from my testing)

gcbeattyAWS commented Dec 9, 2024

View reviewed changes

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/ApiGatewayHttpApiV2ProxyRequestTranslator.cs Outdated Show resolved Hide resolved

gcbeattyAWS requested a review from philasmar December 9, 2024 15:17

gcbeattyAWS marked this pull request as ready for review December 9, 2024 15:17

gcbeattyAWS requested a review from normj December 9, 2024 15:45

philasmar requested changes Dec 9, 2024

View reviewed changes

gcbeattyAWS force-pushed the gcbeatty/apiparse branch from 471264d to 2e94f8b Compare December 9, 2024 17:26

gcbeattyAWS commented Dec 9, 2024

View reviewed changes

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/IRouteConfigurationParser.cs Outdated Show resolved Hide resolved

gcbeattyAWS added the Release Not Needed Add this label if a PR does not need to be released. label Dec 10, 2024

gcbeattyAWS commented Dec 10, 2024

View reviewed changes

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/HttpContextExtensions.cs Outdated Show resolved Hide resolved

gcbeattyAWS marked this pull request as draft December 10, 2024 19:23

Add request parsing into api gateway

77f7679

gcbeattyAWS force-pushed the gcbeatty/apiparse branch from f996e5b to 77f7679 Compare December 10, 2024 19:31

gcbeattyAWS changed the base branch from feature/lambdatesttool-v2 to asmarp/api-gateway-emulator-skeleton December 10, 2024 19:31

fix merge

4a86383

gcbeattyAWS marked this pull request as ready for review December 10, 2024 19:37

gcbeattyAWS commented Dec 10, 2024

View reviewed changes

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/HttpContextExtensions.cs Show resolved Hide resolved

gcbeattyAWS added 2 commits December 10, 2024 14:43

add docs

8bb6fa9

fix namespace

467f18d

gcbeattyAWS commented Dec 10, 2024

View reviewed changes

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Extensions/ExceptionExtensions.cs Show resolved Hide resolved

gcbeattyAWS requested a review from philasmar December 10, 2024 20:40

normj reviewed Dec 11, 2024

View reviewed changes

normj requested changes Dec 11, 2024

View reviewed changes

gcbeattyAWS added 3 commits December 11, 2024 13:29

use null as default value and fix v2 querystring/header logic

fecf91d

url path decoding

ce29cc8

Update comments

24e2c34

update content types for binary

25d613a

gcbeattyAWS requested a review from normj December 12, 2024 17:15

gcbeattyAWS added 2 commits December 12, 2024 12:38

add tests for binary types

dcacd0b

add tests for headers special characters

1080c5c

gcbeattyAWS commented Dec 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ApiGatewayHttpApiV2ProxyRequestTranslator and ApiGatewayProxyRequestTranslator #1901

Add ApiGatewayHttpApiV2ProxyRequestTranslator and ApiGatewayProxyRequestTranslator #1901

gcbeattyAWS commented Dec 9, 2024 •

edited

Loading

normj left a comment •

edited

Loading

gcbeattyAWS commented Dec 11, 2024 •

edited

Loading

normj Dec 11, 2024

normj Dec 11, 2024

normj Dec 11, 2024

gcbeattyAWS Dec 12, 2024

normj Dec 11, 2024

gcbeattyAWS Dec 11, 2024

gcbeattyAWS Dec 11, 2024

gcbeattyAWS Dec 12, 2024 •

edited

Loading

normj Dec 11, 2024

gcbeattyAWS Dec 11, 2024

gcbeattyAWS Dec 12, 2024 •

edited

Loading

normj Dec 11, 2024

gcbeattyAWS Dec 12, 2024 •

edited

Loading

normj commented Dec 11, 2024

gcbeattyAWS commented Dec 12, 2024 •

edited

Loading

gcbeattyAWS Dec 12, 2024

Add ApiGatewayHttpApiV2ProxyRequestTranslator and ApiGatewayProxyRequestTranslator #1901

Are you sure you want to change the base?

Add ApiGatewayHttpApiV2ProxyRequestTranslator and ApiGatewayProxyRequestTranslator #1901

Conversation

gcbeattyAWS commented Dec 9, 2024 • edited Loading

normj left a comment • edited Loading

Choose a reason for hiding this comment

gcbeattyAWS commented Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcbeattyAWS Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcbeattyAWS Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcbeattyAWS Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

normj commented Dec 11, 2024

gcbeattyAWS commented Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

gcbeattyAWS commented Dec 9, 2024 •

edited

Loading

normj left a comment •

edited

Loading

gcbeattyAWS commented Dec 11, 2024 •

edited

Loading

gcbeattyAWS Dec 12, 2024 •

edited

Loading

gcbeattyAWS Dec 12, 2024 •

edited

Loading

gcbeattyAWS Dec 12, 2024 •

edited

Loading

gcbeattyAWS commented Dec 12, 2024 •

edited

Loading