[BUG] AwsSigV4HttpConnection generates the wrong request signature to sign low-level client requests.

[zordark]

What is the bug?

Good day.
We have long used an old version of the OpenSearch service with ElasticSearch under the hood. We decided to migrate to the newer version (OpenSearch 2.13). During our infrastructure rollout (with AWS CDK), we have a lambda function (written in .NET 8) called from the custom resource provider that sets OpenSearch cluster roles mapping. Lambda uses the same role as we use for the master user for the OpenSearch cluster. This setup works for us with the old version. Since we moved to the latest OpenSearch version, we decided to migrate to the latest OpenSearch.Net client as well.
After we did it, we have a problem: that lambda call to cluster fails with the next error:

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been.....

The way we instantiate client and connection is:

var pool = new SingleNodeConnectionPool(new Uri($"https://{config.EsDomainUrl}"));
IConnection connection = new AwsSigV4HttpConnection();

var settings = new ConnectionSettings(pool, connection).ServerCertificateValidationCallback((sender, cert, chain, errors) => true);

if (config.LogSeverity == LogSeverity.Debug)
{
    settings
        .EnableDebugMode()
        .MemoryStreamFactory(new MemoryStreamFactory())
        .PrettyJson()
        .DisableDirectStreaming();
}

return new OpenSearchClient(settings);

and then we have a handler to execute HTTP requests to OS cluster which uses the client created above

var lowLevelClient = client.LowLevel;

var method = Enum.Parse<HttpMethod>(request.Method!);
var response = await lowLevelClient.DoRequestAsync<DynamicResponse>(method, request.Path,
                ct, PostData.String(request.Body.ToString()));

Where request is an object that comes from the AWS CDK custom resource provider framework.

The complete response is below (I just formatted it a bit to make it more readable)

Unsuccessful (403) low level call on PUT: /_plugins/_security/api/rolesmapping/all_access?pretty=true&error_trace=true
# Audit trail of this API call:
 - [1] BadResponse: Node: https://vpc-object-finder-search-XXXXXXX.us-east-1.es.amazonaws.com/ Took: 00:00:01.9802597
# OriginalException: OpenSearch.Net.OpenSearchClientException: Request failed to execute. Call: Status code 403 from: PUT /_plugins/_security/api/rolesmapping/all_access?pretty=true&error_trace=true
# Request:
{
  "backend_roles": [
    "arn:aws:iam::XXXXXXXXXXX:role/RootInfraStackDev-OsAdminUserRoleC8B73E44-IcpYjgPjd0GW",
    "arn:aws:iam::XXXXXXXXXXX:role/RootInfraStackDev-LambdaServiceRoleUsEast1FF936886-9V9BMstTxwY8"
  ],
  "hosts": [],
  "users": []
}
# Response:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been
			'PUT
			/_plugins/_security/api/rolesmapping/all_access
			error_trace=true&pretty=true
			accept:application/json
			content-type:application/json
			host:vpc-object-finder-search-vep4wj3hpvuor5fi7wj2msc7pm.us-east-1.es.amazonaws.com
			opensearch-client-meta:opensearch=1.8.0,a=1,net=8.0.8,so=8.0.8
			x-amz-content-sha256:3d92ca02549ddb3a91f1bf7ae8a945069ecbfed972e66da31d13d0910b42b8ba
			x-amz-date:20241108T140507Z
			x-amz-security-token:IQoJb3JpZ2luX2VjENf//////////wEaCXVzLWVhc3QtMSJGMEQCICP5AHtgvjZk2UN1lUhu7GOZsFdikhMvSgvjuplKB5HiAiBDb0mX0UZb5tIeiUQVRb5iqqxn6s/R55pwczYa3hiPrSrTAwhfEAAaDDE0Nzk5NzE1MjI4NiIMkHfRAL0LmwoSx3DOKrADeaL+l+y59dqdmBkZHqw7sTm3lkZByQ7EQPRoN4Q1NXNhHV4Yz54Qjicbqkr85cX72lfHKnI6NGyHDJ9ibI5kAatntziQV5m+myiPvW4x3C7s1iGg+xA+XaqHCj+yEcdnOqgwwI61bdJtjVwQ8/Dwe6i0iOR2vvc07VM70uApDmuVLIQjWhd0su9SEJkedTm6aFj8HyzyW9bS490I2LN+JkLJkTL8wModidd59/leBhfHjFoHRrhNqSx/rIaxBsxMCgh8mw0PPZuVG/CHEHIdZd1FqcRnh/FshpgCNuYzeaaFXVx2KJcP/bnHseI0u4kGi5kCxC3U9cstr2JxKjyaQttKqrZYIY+KBb7Gn4fvYWOXr3L2emyso+/snDJXYnZ9mit0ZiFme7s0Cb4BOyDCP9/HpVl2mpmWQGQTojNEB3rNMHCP5i0M5G4PHXva4N0a1WyWbCEpA6dK5Z+9mQeMQG8YQlF5Rh45gqX2ykvaU4EAdl+ychgLvQvZP1o4NtM7FP8qlcVlq/zAMHrhILAsr8ddYrFpblUqcxeJH5Ii7dmRlo/rOPcsD0E3dHGZcO2MMI61uLkGOp8B6VBK5FNjx0ImFtiR0jZT7J5dmTGB/3euODiTRU70mG5bJgZ4O5WUu24nljkG5gGPzne2Q2Xmm5VitEBIQcwkRQv/GSPRj9Xtdu21qPqAGhK8koMRKg/H7EkjnKlqHKJiN2e+mdJB98tj0Jf8ALMhUFER5o/NmBJuHtql3ltJg0W4TJC2PAOZOFsmBzw1Uzpj3kd8tbHpIL7ursIUtt3x
			x-amzn-trace-id:Self=1-672e1a94-3b195b8c1f57da8150f57fa3;Root=1-672e1a8c-5114c5f91004d4683f59115d;Sampled=0

			accept;content-type;host;opensearch-client-meta;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amzn-trace-id
			3d92ca02549ddb3a91f1bf7ae8a945069ecbfed972e66da31d13d0910b42b8ba'

			The String-to-Sign should have been
			'AWS4-HMAC-SHA256
			20241108T140507Z
			20241108/us-east-1/es/aws4_request
			940caf65890de067dc0d2cca3d71a1be8d487728076958006ab906e24477011a			
			'\n"}
# TCP states:
  Established: 4

# ThreadPool statistics:
  Worker: 
    Busy: 1
    Free: 32766
    Min: 2
    Max: 32767
  IOCP: 
    Busy: 0
    Free: 1000
    Min: 1
    Max: 1000

What can be the reason for this problem?

[dblock]

I would start debugging this by double checking that you can make simpler requests with the .NET client (e.g. GETs before PUT's which will narrow down the problem). Maybe try standalone command-line code from the same host that's known to work (https://github.com/dblock/opensearch-dotnet-client-demo) before doing it from within your application, and then we can compare?

[Xtansia]

@zordark Are you passing query parameters as part of request.Path? If so that may be the issue, as it's assumed to be only the path and the query parameters would normally be passed as part of IRequestParameters. The methods under client.Http. provide easier ways of passing query parameters but we don't have a generic method that accepts the HttpMethod as a parameter. Would gladly welcome the addition of such a method if you were interested in contributing

[zordark]

@zordark Are you passing query parameters as part of request.Path? If so that may be the issue, as it's assumed to be only the path and the query parameters would normally be passed as part of IRequestParameters. The methods under client.Http. provide easier ways of passing query parameters but we don't have a generic method that accepts the HttpMethod as a parameter. Would gladly welcome the addition of such a method if you were interested in contributing

I don't add any parameters explicitly, and I think they are added automatically based on .Net client settings.
I've reused an idea from the samples in this repo to set up security in the OS cluster during CDK deployment, and this is how custom provider payload looks like:

    new CustomResource(scope, `${osDomainName}OsSignedRequestsResource${regionSuffix}`, {
        serviceToken: osRequestProvider.serviceToken,
        properties: {
            requests: [
                {
                    "method": "PUT",
                    "path": "_plugins/_security/api/rolesmapping/all_access",
                    "body": {
                        "backend_roles": [
                            props.osAdminUserRole.roleArn,
                            props.lambdaServiceRole.roleArn
                        ],
                        "hosts": [],
                        "users": []
                    }
                }
            ]
        }
    });

[zordark]

Another problem related to this is that I can see real responses only because the client logs raw responses. But when it comes to deserialization I see that error:

2024-11-08T14:05:11.265Z	fe1a9671-2596-473a-b29d-a77d7c57c829	fail	Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'OriginalException' with type 'OpenSearch.Net.OpenSearchClientException'. Path 'Data.Response'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value)
   at SimpleConsoleLogger.DefaultLogger.Log(LogItem item)
   at SimpleConsoleLogger.ILoggerExtensions.Error(ILogger logger, String message, Object data)
   at GlobalSearch.Tools.Services.CustomResourceService.Process(HttpRequestCollection input, CancellationToken ct) in /home/vsts/work/1/s/src/GlobalSearch.Tools/Services/CustomResourceService.cs:line 47
   at GlobalSearch.Tools.Functions.ExecuteSignedHttpRequest(CustomResourceProviderRequest`1 input, ILambdaContext context) in /home/vsts/work/1/s/src/GlobalSearch.Tools/Functions.cs:line 79
   at lambda_method1(Closure, Stream, ILambdaContext, Stream)
   at Amazon.Lambda.RuntimeSupport.HandlerWrapper.<>c__DisplayClass8_0.<GetHandlerWrapper>b__0(InvocationRequest invocation) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/HandlerWrapper.cs:line 54
   at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.InvokeOnceAsync(CancellationToken cancellationToken) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/LambdaBootstrap.cs:line 220

[zordark]

Good day. Eventually, we found the issue. Someone in our team created the wrapper with the same name.
But I'm still curious why it can affect the sign process. The wrapper was created to trace HTTP requests into XRay.

public class AwsSigV4HttpConnection : OpenSearch.Net.Auth.AwsSigV4.AwsSigV4HttpConnection
{
    protected override HttpMessageHandler CreateHttpClientHandler(RequestData requestData)
    {
        var baseHandler = base.CreateHttpClientHandler(requestData);
        return bool.Parse(Environment.GetEnvironmentVariable("XRayDisabled") ?? "false") // For local run.
               || AWSXRayRecorder.Instance.IsTracingDisabled()
            ? baseHandler
            : new HttpClientXRayTracingHandler(baseHandler);
    }
}