Bug: sam build - FileNotFoundError, file is part of aws-sdk node module

[San-Zer0-786]

Description:

sam build does not work. It gives an error message, file not found. The file not found references a part of node_modules of aws-sdk. I cannot simply edit these inside the .aws-sam/build folder as we know, this is a build folder and hence changes will be overwritten.
I am unable to find other users who have the same issue in terms of the context referencing a file not found in @AWS-SDK node_module which is strange,

Steps to reproduce:

sam build

Observed result:

Please see end of post to prevent duplication. In essence, during build process, a file is not found when copying artifacts which causes the process to terminate and fail.

Expected result:

The sam build process completes normally.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 11
2. sam --version: 1.98.0
3. AWS region: eu-west-2

{
  "version": "1.98.0",
  "system": {
    "python": "3.9.13",
    "os": "Windows-10-10.0.22621-SP0"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

2024-02-04 23:19:32,521 | Copying artifacts from
C:\Users\userName\OneDrive\Desktop\AT\project-name\.aws-sam\build\CognitoClinicUserPoolStack\PreSignUpFunctio
n to
C:\Users\userName\OneDrive\Desktop\AT\project-name\.aws-sam\build\UserApiStack\GetContentFromTestFunct
ion
2024-02-04 23:19:42,030 | Copying artifacts from
C:\Users\userName\OneDrive\Desktop\AT\project-name\.aws-sam\build\CognitoClinicUserPoolStack\PreSignUpFunctio
n to
C:\Users\userName\OneDrive\Desktop\AT\project-name\.aws-sam\build\ManagementFunctionsStack\UploadPaidContentZ
ipFileFunction
2024-02-04 23:19:43,955 | Exception raised during the execution
2024-02-04 23:19:43,961 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-02-04 23:19:43,966 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-02-04 23:19:43,968 | Unable to find Click Context for getting session_id.

Error: [Errno 2] No such file or directory: 'C:\\Users\\userName\\OneDrive\\Desktop\\AT\\project-name\\.aws-sam\\build\\ManagementFunctionsStack\\UploadPaidContentZipFileFunction\\node_modules\\@aws-sdk\\client-ses\\node_modules\\@aws-sdk\\credential-provider-process\\dist-types\\ts3.4\\getValidatedProcessCredentials.d.ts'
Traceback:
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\cli\cli_config_file.py", line 297, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\telemetry\metric.py", line 184, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\telemetry\metric.py", line 149, in wrapped
    return_value = func(*args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\version_checker.py", line 42, in wrapped
    actual_result = func(*args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\cli\main.py", line 95, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\commands\build\command.py", line 171, in cli
    do_cli(
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\commands\build\command.py", line 260, in do_cli
    ctx.run()
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\commands\build\build_context.py", line 281, in run
    self._build_result = builder.build()
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\app_builder.py", line 217, in build
    return ApplicationBuildResult(build_graph, build_strategy.build())
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 394, in build
    return super().build()
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 81, in build
    result.update(self._build_functions(self._build_graph))
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 400, in _build_functions
    return self._run_builds_async(
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 416, in _run_builds_async
    async_results = async_context.run_async()
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\async_utils.py", line 131, in run_async
    return run_given_tasks_async(self._async_tasks, event_loop)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\async_utils.py", line 90, in run_given_tasks_async
    return event_loop.run_until_complete(_run_given_tasks_async(tasks, event_loop, executor))
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 647, in run_until_complete
    return future.result()
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\async_utils.py", line 58, in _run_given_tasks_async
    raise result
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 427, in build_single_function_definition
    return self._delegate_build_strategy.build_single_function_definition(build_definition)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 566, in build_single_function_definition
    return self._incremental_build_strategy.build_single_function_definition(build_definition)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 460, in build_single_function_definition
    return self._delegate_build_strategy.build_single_function_definition(build_definition)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\build\build_strategy.py", line 195, in build_single_function_definition
    osutils.copytree(single_build_dir, artifacts_dir)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\osutils.py", line 179, in copytree
    copytree(new_source, new_destination, ignore=ignore)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\osutils.py", line 179, in copytree
    copytree(new_source, new_destination, ignore=ignore)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\osutils.py", line 179, in copytree
    copytree(new_source, new_destination, ignore=ignore)
  [Previous line repeated 5 more times]
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\site-packages\samcli\lib\utils\osutils.py", line 181, in copytree
    shutil.copy2(new_source, new_destination)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 444, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 266, in copyfile
    with open(dst, 'wb') as fdst:

[mndeveci]

Hi there,

Can you provide more details of your lambda function? Especially, if you can share simplified version of package.json file, it will help us a lot to find the root cause.

I was able to build a node20.x lambda function with following package.json (dependencies);

{
  "name": "hello_world",
  "version": "1.0.0",
  "description": "hello world sample for NodeJS",
  "main": "app.js",
  "author": "SAM CLI",
  "license": "MIT",
  "dependencies": {
    "axios": ">=1.6.0",
    "aws-sdk": "2.1551.0"
  },
  "scripts": {
    "test": "mocha tests/unit/"
  },
  "devDependencies": {
    "chai": "^4.3.6",
    "mocha": "^10.2.0"
  }
}

[San-Zer0-786]

Hi there,

Can you provide more details of your lambda function? Especially, if you can share simplified version of package.json file, it will help us a lot to find the root cause.

I was able to build a node20.x lambda function with following package.json (dependencies);

{
  "name": "hello_world",
  "version": "1.0.0",
  "description": "hello world sample for NodeJS",
  "main": "app.js",
  "author": "SAM CLI",
  "license": "MIT",
  "dependencies": {
    "axios": ">=1.6.0",
    "aws-sdk": "2.1551.0"
  },
  "scripts": {
    "test": "mocha tests/unit/"
  },
  "devDependencies": {
    "chai": "^4.3.6",
    "mocha": "^10.2.0"
  }
}

Thanks for replying

All lambda functions in the project use this package.json:

`

{
  "name": "lambda",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "^3.423.0",
    "@aws-sdk/client-ses": "^3.449.0",
    "ajv": "^8.12.0",
    "ajv-formats": "^2.1.1",
    "axios": "^1.5.1",
    "jszip": "^3.10.1",
    "sanitize-html": "^2.11.0"
  },
  "devDependencies": {
    "eslint": "^8.52.0"
  }
}

`

I wondered at the start - if the error is happening due to a file not found part of "@aws-sdk/client-ses": "^3.449.0",

Is it failing because client-ses was upgraded to 3.5 recently, so the ^ would install 3.5 accordingly?
I then forced it to be 3.449.0 but unfortunately, this didn't help fix it.

It is very strange, during Copying artifacts from
C:\Users\userName\OneDrive\Desktop\AT\project-name.aws-sam\build\CognitoClinicUserPoolStack\PreSignUpFunctio
n to
C:\Users\userName\OneDrive\Desktop\AT\project-name.aws-sam\build\UserApiStack\GetContentFromTestFunct
ion

this all works, but then the next one fails, saying it is unable to find a node_module part of client-ses. Not sure what control I have over that instead of npm install, the module relies on other node modules which I cannot edit directly as it will overwrite

[mndeveci]

Thanks for providing the package.json file. I've applied that to a hello world template and my sam build succeeds with no issues.

Error: [Errno 2] No such file or directory: 'C:\\Users\\userName\\OneDrive\\Desktop\\AT\\project-name\\.aws-sam\\build\\ManagementFunctionsStack\\UploadPaidContentZipFileFunction\\node_modules\\@aws-sdk\\client-ses\\node_modules\\@aws-sdk\\credential-provider-process\\dist-types\\ts3.4\\getValidatedProcessCredentials.d.ts'

One problem I see about this one above is that there are multiple node_modules in the path, I would only expect there should be one node_modules and dependencies should live there. For instance, after my build if I list the files under .aws-sam/build/HelloWorldFunction/node_modules/@aws-sdk/client-ses, there is no node_modules folder.

❯ ls -ltra .aws-sam/build/HelloWorldFunction/node_modules/@aws-sdk/client-ses
total 120
drwxr-xr-x  28 user  wheel    896  6 Feb 14:17 ..
-rw-r--r--   1 user  wheel  11390  6 Feb 14:17 LICENSE
drwxr-xr-x  17 user  wheel    544  6 Feb 14:17 dist-cjs
drwxr-xr-x  17 user  wheel    544  6 Feb 14:17 dist-es
-rw-r--r--   1 user  wheel   3670  6 Feb 14:17 package.json
drwxr-xr-x   8 user  wheel    256  6 Feb 14:17 .
-rw-r--r--   1 user  wheel  41373  6 Feb 14:17 README.md
drwxr-xr-x  18 user  wheel    576  6 Feb 14:17 dist-types

Is it possible to provide a simplified example for us to reproduce it?

[San-Zer0-786]

I managed to resolve - it was a Windows error. Apparently there is a length limit in reading a filepath. I did the following to get this resolved:

https://superuser.com/questions/1807770/enable-long-paths-on-win-11-home

Definitely an issue that wasted time, good ol' Windows...

[lucashuy]

Thanks for letting us know that you've resolve the issue (a Windows long path problem), I'm going to go ahead and resolve this. If there are any other issues that you face with SAM CLI, feel free to open a new issue to report it.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.