Bug: sam local invoke - ConnectionError

[isaacasancheza]

Description:

sam local invoke fails to locally test a lambda function using a lambda layer.

Steps to reproduce:

1. git clone https://github.com/isaacsancheza/bug-sam-local-invoke
2. cd bug-sam-local-invoke
3. pipenv install --dev
4. pipenv run cdk synth
5. sam local invoke -t ./cdk.out/Stack.template.json Function

Observed result:

$ sam local invoke -t ./cdk.out/Stack.template.json Function

2023-10-17 22:38:09,513 | No config file found in this directory.                                                                     
2023-10-17 22:38:09,520 | OSError occurred while reading TOML file: [Errno 2] No such file or directory:                              
'/tmp/bug-sam-local-invoke/cdk.out/samconfig.toml'                                                                                    
2023-10-17 22:38:09,523 | Config file location: /tmp/bug-sam-local-invoke/cdk.out/samconfig.toml                                      
2023-10-17 22:38:09,525 | Config file '/tmp/bug-sam-local-invoke/cdk.out/samconfig.toml' does not exist                               
2023-10-17 22:38:09,576 | OSError occurred while reading TOML file: [Errno 2] No such file or directory:                              
'/tmp/bug-sam-local-invoke/cdk.out/samconfig.toml'                                                                                    
2023-10-17 22:38:09,578 | Using config file: samconfig.toml, config environment: default                                              
2023-10-17 22:38:09,579 | Expand command line arguments to:                                                                           
2023-10-17 22:38:09,581 | --template_file=/tmp/bug-sam-local-invoke/cdk.out/Stack.template.json --function_logical_id=Function        
--no_event --layer_cache_basedir=/home/isaac/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1      
2023-10-17 22:38:09,583 | local invoke command is called                                                                              
2023-10-17 22:38:09,585 | Collected default values for parameters: {'BootstrapVersion': '/cdk-bootstrap/hnb659fds/version'}           
2023-10-17 22:38:09,634 | CDK Path for resource LayerB20D2F06 is ['Stack', 'Layer', 'Resource']                                       
2023-10-17 22:38:09,636 | CDK Path for resource FunctionServiceRole675BB04A is ['Stack', 'Function', 'ServiceRole', 'Resource']       
2023-10-17 22:38:09,638 | CDK Path for resource Function76856677 is ['Stack', 'Function', 'Resource']                                 
2023-10-17 22:38:09,639 | CDK Path for resource CDKMetadata is ['Stack', 'CDKMetadata', 'Default']                                    
2023-10-17 22:38:09,642 | 0 stacks found in the template                                                                              
2023-10-17 22:38:09,644 | Collected default values for parameters: {'BootstrapVersion': '/cdk-bootstrap/hnb659fds/version'}           
2023-10-17 22:38:09,685 | CDK Path for resource LayerB20D2F06 is ['Stack', 'Layer', 'Resource']                                       
2023-10-17 22:38:09,687 | CDK Path for resource FunctionServiceRole675BB04A is ['Stack', 'Function', 'ServiceRole', 'Resource']       
2023-10-17 22:38:09,689 | CDK Path for resource Function76856677 is ['Stack', 'Function', 'Resource']                                 
2023-10-17 22:38:09,690 | CDK Path for resource CDKMetadata is ['Stack', 'CDKMetadata', 'Default']                                    
2023-10-17 22:38:09,693 | 4 resources found in the stack                                                                              
2023-10-17 22:38:09,695 | --base-dir is not presented, adjusting uri                                                                  
asset.fc472b326c13965c5989b2222206ed4aa71988221f5fa91fd50e39016ca2b9dc relative to                                                    
/tmp/bug-sam-local-invoke/cdk.out/Stack.template.json                                                                                 
2023-10-17 22:38:09,699 | Found Lambda function with name='Function76856677' and                                                      
CodeUri='asset.e14bec811d85afa19daa35d2c8e7b4c205e533cf0a2b3c36bbd98ea606604449'                                                      
2023-10-17 22:38:09,702 | --base-dir is not presented, adjusting uri                                                                  
asset.e14bec811d85afa19daa35d2c8e7b4c205e533cf0a2b3c36bbd98ea606604449 relative to                                                    
/tmp/bug-sam-local-invoke/cdk.out/Stack.template.json                                                                                 
2023-10-17 22:38:09,705 | Skip building pre-built function: Function                                                                  
2023-10-17 22:38:09,716 | Found one Lambda function with name 'Function'                                                              
2023-10-17 22:38:09,718 | Invoking lambda.handler (python3.10)                                                                        
2023-10-17 22:38:09,719 | No environment variables found for function 'Function76856677'                                              
2023-10-17 22:38:09,721 | Loading AWS credentials from session with profile 'None'                                                    
2023-10-17 22:38:10,422 | Resolving code path. Cwd=/tmp/bug-sam-local-invoke/cdk.out,                                                 
CodeUri=/tmp/bug-sam-local-invoke/cdk.out/asset.e14bec811d85afa19daa35d2c8e7b4c205e533cf0a2b3c36bbd98ea606604449                      
2023-10-17 22:38:10,426 | Resolved absolute path to code is                                                                           
/tmp/bug-sam-local-invoke/cdk.out/asset.e14bec811d85afa19daa35d2c8e7b4c205e533cf0a2b3c36bbd98ea606604449                              
2023-10-17 22:38:10,428 | Code                                                                                                        
/tmp/bug-sam-local-invoke/cdk.out/asset.e14bec811d85afa19daa35d2c8e7b4c205e533cf0a2b3c36bbd98ea606604449 is not a zip/jar file        
2023-10-17 22:38:10,430 | Code                                                                                                        
/tmp/bug-sam-local-invoke/cdk.out/asset.fc472b326c13965c5989b2222206ed4aa71988221f5fa91fd50e39016ca2b9dc is not a zip/jar file        
2023-10-17 22:38:10,433 | LayerB20D2F06 is a local Layer in the template                                                              
2023-10-17 22:38:10,435 | Resolving code path. Cwd=/tmp/bug-sam-local-invoke/cdk.out,                                                 
CodeUri=/tmp/bug-sam-local-invoke/cdk.out/asset.fc472b326c13965c5989b2222206ed4aa71988221f5fa91fd50e39016ca2b9dc                      
2023-10-17 22:38:10,442 | Local image was not found.                                                                                  
Building image...2023-10-17 22:38:11,338 | Cleaning all decompressed code dirs                                                                         
2023-10-17 22:38:11,341 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics  
2023-10-17 22:38:11,363 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics  
2023-10-17 22:38:11,366 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'd2d99296-052e-4e06-830d-7e7b84430b53',        
'installationId': '9a646c51-0813-45dc-94fa-0b905fd7ddc4', 'sessionId': '5da3d444-858d-47c5-a00d-11406558559b', 'executionEnvironment':
'CLI', 'ci': False, 'pyversion': '3.11.3', 'samcliVersion': '1.98.0', 'awsProfileProvided': False, 'debugFlagProvided': True,         
'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CDK', 'gitOrigin': None, 'projectName': 
'abf1449499536411d1de1399e249678ec982799e753d91918c6185809f473e8f', 'initialCommit': None}, 'duration': 1763, 'exitReason':           
'ConnectionError', 'exitCode': 255}}]}                                                                                                
2023-10-17 22:38:11,366 | Unable to find Click Context for getting session_id.                                                        
2023-10-17 22:38:11,380 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a1555a41-739b-4dd3-8049-60bd722e6361',            
'installationId': '9a646c51-0813-45dc-94fa-0b905fd7ddc4', 'sessionId': '5da3d444-858d-47c5-a00d-11406558559b', 'executionEnvironment':
'CLI', 'ci': False, 'pyversion': '3.11.3', 'samcliVersion': '1.98.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes':  
{'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'ef41f47691cf4a96b96b95dae1b3dbf3',         
'time_stamp': '2023-10-18 04:38:09.523', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml',    
'thread_id': '1759910ca01f4282af814c1e38b92d32', 'time_stamp': '2023-10-18 04:38:09.578', 'exception_name': None}, {'event_name':     
'UsedFeature', 'event_value': 'CDK', 'thread_id': '906938b4b49f463bbc66599a3c8f96e5', 'time_stamp': '2023-10-18 04:38:11.344',        
'exception_name': None}]}}}]}                                                                                                         
2023-10-17 22:38:11,954 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed    
out. (read timeout=0.1)                                                                                                               
2023-10-17 22:38:11,958 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed    
out. (read timeout=0.1)                                                                                                               

Error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Traceback:
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "samcli/cli/cli_config_file.py", line 297, in wrapper
  File "click/decorators.py", line 92, in new_func
  File "click/core.py", line 783, in invoke
  File "samcli/lib/telemetry/metric.py", line 184, in wrapped
  File "samcli/lib/telemetry/metric.py", line 149, in wrapped
  File "samcli/lib/utils/version_checker.py", line 42, in wrapped
  File "samcli/cli/main.py", line 95, in wrapper
  File "samcli/commands/local/invoke/cli.py", line 103, in cli
  File "samcli/commands/local/invoke/cli.py", line 201, in do_cli
  File "samcli/commands/local/lib/local_lambda.py", line 144, in invoke
  File "samcli/lib/telemetry/metric.py", line 324, in wrapped_func
  File "samcli/local/lambdafn/runtime.py", line 189, in invoke
  File "samcli/local/lambdafn/runtime.py", line 85, in create
  File "samcli/local/docker/lambda_container.py", line 94, in __init__
  File "samcli/local/docker/lambda_container.py", line 237, in _get_image
  File "samcli/local/docker/lambda_image.py", line 238, in build
  File "samcli/local/docker/lambda_image.py", line 337, in _build_image
  File "docker/api/build.py", line 264, in build
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 233, in _post
  File "requests/sessions.py", line 637, in post
  File "requests/sessions.py", line 589, in request
  File "requests/sessions.py", line 703, in send
  File "requests/adapters.py", line 501, in send

An unexpected error was encountered while executing "sam local invoke".
Search for an existing issue:
https://github.com/aws/aws-sam-cli/issues?q=is%3Aissue+is%3Aopen+Bug%3A%20sam%20local%20invoke%20-%20ConnectionError
Or create a bug report:
https://github.com/aws/aws-sam-cli/issues/new?template=Bug_report.md&title=Bug%3A%20sam%20local%20invoke%20-%20ConnectionError

Expected result:

The function is executed.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Debian GNU/Linux 12 (bookworm)
2. sam --version: SAM CLI, version 1.98.0
3. AWS region: us-east-1

$ sam --info
{
  "version": "1.98.0",
  "system": {
    "python": "3.11.3",
    "os": "Linux-6.1.0-13-amd64-x86_64-with-glibc2.36"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.6",
    "aws_cdk": "2.101.1 (build 16ddad1)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[mndeveci]

Hi there,

I wasn't able to re-produce this issue on my machine but I was using an earlier version of Docker engine. After I've tried to update 24.0.6, I've got issues like the one above. But I was also not able to fully complete the update (it was still asking me to restart docker, which it couldn't restart itself). In the end I've removed docker installation and did a clean install and then I am now able to invoke my function.

Can you try to re-install Docker on your machine as well to see if it is going to solve this problem?

Thanks

[isaacasancheza]

Thanks for pointing out about docker. As suggested, I removed docker and did a clean install but the error was still there. Even installed an early version (23.0) but the error was still there. I was using docker engine and ended installing docker desktop. After installing docker desktop, I got a message about enabling virtualization. After enabling it at BIOS, I was able to run docker desktop. After then I was able to execute the sam local invoke without problem.

I was curious if docker desktop solved the problem or enabling virtualization so I removed docker desktop and installed docker engine again but with virtualization enabled at BIOS. Surprisingly the error is gone and now I'm able to test my functions.

Thanks.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.