Bug: sam local invoke throws exception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:

[vamsikrishna507]

Description:

when i was trying to invoke sam local invoke for my lambda function, exception occurred related to the certificate though my java has the certificate

my lambda function calls external api for some information
Exception details:

I/O error on GET request for "https://api-metoffice.apiconnect.ibmcloud.com/v0/forecasts/point/hourly": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target: org.springframework.web.client.ResourceAccessException
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://api-metoffice.apiconnect.ibmcloud.com/v0/forecasts/point/hourly": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:905)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:885)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)
at uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.QueryTemperature(DailyLambdaHandler.java:58)
at uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.handleRequest(DailyLambdaHandler.java:34)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
at org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:79)
at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:879)
... 8 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
at java.base/sun.security.validator.Validator.validate(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
... 27 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
... 32 more

END RequestId: be97ec6b-f8a4-42d4-802c-df3add59a8c7
REPORT RequestId: be97ec6b-f8a4-42d4-802c-df3add59a8c7 Init Duration: 0.92 ms Duration: 20870.55 ms Billed Duration: 20871 ms Memory Size: 512 MB Max Memory Used: 512 MB
{"errorMessage": "I/O error on GET request for "https://api-metoffice.apiconnect.ibmcloud.com/v0/forecasts/point/hourly\": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", "errorType": "org.springframework.web.client.ResourceAccessException", "stackTrace": ["org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:905)", "org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:885)", "org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)", "org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.QueryTemperature(DailyLambdaHandler.java:58)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.handleRequest(DailyLambdaHandler.java:34)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)", "java.base/java.lang.reflect.Method.invoke(Unknown Source)"], "cause": {"errorMessage": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", "errorType": "javax.net.ssl.SSLHandshakeException", "stackTrace": ["java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)", "java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)", "java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)", "java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)", "java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)", "java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)", "java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)", "java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)", "java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)", "org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:79)", "org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)", "org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)", "org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:879)", "org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)", "org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.QueryTemperature(DailyLambdaHandler.java:58)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.handleRequest(DailyLambdaHandler.java:34)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)", "java.base/java.lang.reflect.Method.invoke(Unknown Source)"], "cause": {"errorMessage": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", "errorType": "sun.security.validator.ValidatorException", "stackTrace": ["java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)", "java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)", "java.base/sun.security.validator.Validator.validate(Unknown Source)", "java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)", "java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)", "java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)", "java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)", "java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)", "java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)", "java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)", "org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:79)", "org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)", "org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)", "org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:879)", "org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)", "org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.QueryTemperature(DailyLambdaHandler.java:58)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.handleRequest(DailyLambdaHandler.java:34)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)", "java.base/java.lang.reflect.Method.invoke(Unknown Source)"], "cause": {"errorMessage": "unable to find valid certification path to requested target", "errorType": "sun.security.provider.certpath.SunCertPathBuilderException", "stackTrace": ["java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)", "java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)", "java.base/java.security.cert.CertPathBuilder.build(Unknown Source)", "java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)", "java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)", "java.base/sun.security.validator.Validator.validate(Unknown Source)", "java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)", "java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)", "java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)", "java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)", "java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)", "java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)", "java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)", "java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)", "java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)", "java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)", "org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:79)", "org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)", "org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)", "org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:879)", "org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)", "org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.QueryTemperature(DailyLambdaHandler.java:58)", "uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler.handleRequest(DailyLambdaHandler.java:34)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)", "java.base/java.lang.reflect.Method.invoke(Unknown Source)"]}}}}

Steps to reproduce:

Observed result:

Expected result:

There shouldn't be exception thrown

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macbook pro, 14.3 version
2. sam --version: SAM CLI, version 1.109.0
3. AWS region:us-west-2

# Paste the output of `sam --info` here

Add --debug flag to command you are running

[vamsikrishna507]

➜ test-project sam --info
{
"version": "1.109.0",
"system": {
"python": "3.12.2",
"os": "macOS-14.3-arm64-arm-64bit"
},
"additional_dependencies": {
"docker_engine": "20.10.21",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}

[vamsikrishna507]

Template.yaml:

AppFunction:
Type: AWS::Serverless::Function
Properties:
Runtime: java17
Handler: uk.gov.dwp.coldweatherpay.weatherdatacapture.lambda.DailyLambdaHandler::handleRequest
Timeout: 60
MemorySize: 512
CodeUri: ./target/weather-data-capture-1.0.0.jar

[mildaniel]

Hey @vamsikrishna507, does this network call succeed if you run your code outside of SAM CLI (and outside of a container)?

[vamsikrishna507]

Hello mildaniel,

Yes, It works outside of the container and even it works when i deploy the code in to aws environment and testing there

[vamsikrishna507]

This is blocking testing of lambda locally and please prioritise with workaround to continue

[lucashuy]

Hi, is there a custom certificate that is suppose to be used to complete the API calls? If there is a custom certificate that is being used to call the API, then those aren't passed into the invoke container on it's own, and you may need to create a custom invoke image to use locally.

A workaround for this is to test on the cloud, using sam sync, you can synchronize any code changes to the cloud, and use sam remote invoke to invoke that function.

[vamsikrishna507]

Hello,

can you share a snippet of docker file to generate image locally using docker file and run it locally which can copy the certificate?

[lucashuy]

If you have a certificate, you can create a Dockerfile based off the the existing Java 17 Lambda runtime image (public.ecr.aws/lambda/java:17) to copy the certificate into the image's key store so that it can be used. Depending on what certificate you have, you can use keytool to import the certificate. The Dockerfile would look something like this:

FROM public.ecr.aws/lambda/java:17

ADD your_certificate.crt /some/path/in/image
RUN keytool -importcert -file <the added cert> -storepass <password> -keystore <keystore name>

You'll have to upload the build Docker image somewhere (I used AWS ECR), and then use it with sam local invoke --invoke-image <url to uploaded image>.

Something worth noting is that you mentioned the network call works when it was deployed to AWS. Did you end up using or uploading your certificate somewhere in AWS?

[vamsikrishna507]

Hello,

Thanks for the details.
I didn't upload any certificate in AWS for running the lambda function but it worked

[lucashuy]

Thanks for the response. Do other projects work when invoking in a container? You can use sam init to create a Java based hello world project that will perform a network call to get the machine's public facing IP address. When you have the hello world project, you can sam build and sam local invoke to see if the network call works.

[phamductri]

Another workaround is your create a custom JKS truststore, import your certificate into that custom trust store, then include that custom trust store in your src/main/resources/ folder, then pass into the AWS lambda JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=mytruststore.jks -Djavax.net.ssl.trustStorePassword=changeit

[hnnasit]

Hi @vamsikrishna507, just following up on @lucashuy's comment if you got a chance to test it as this might not be due to a SAM CLI issue.

[mildaniel]

Closing as this seems to be a network configuration issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.