feat: Migrate S3 Client from AWS SDK v2 (deprecated) to v3

[vahidalizad]

Summary

This pull request migrates the S3 client implementation from AWS SDK v2 to v3.

Changes

1. Initialization: Updated the S3 client initialization to use S3Client from @aws-sdk/client-s3 package.
2. Commands: Replaced v2 commands (getObject, putObject, etc.) with their v3 equivalents (GetObjectCommand, PutObjectCommand, etc.).
3. Response Handling: Modified response handling to accommodate changes in the v3 API, particularly in how streams and buffers are managed.
4. Testing: Added unit tests to cover new functionality and ensure compatibility with the existing codebase. Implemented mocks for offline testing using Jasmine.

Release Dependency

This pull request is intended for after the release of the asynchronous getFileLocation function on the parse-server repository. Please ensure that this dependency is met before merging.

The relevant pull request for adding support of asynchronous getFileLocation in adapters can be found here: parse-server PR #9271.

References

• AWS SDK for JavaScript v3 Documentation

Please review the changes and provide feedback. Thank you for your time and consideration.

Closes: #197

[parse-github-assistant]

Thanks for opening this pull request!

[mtrezza]

Continuing from #220 (comment).

I notice that this CI does not test against Parse Server, so with or without parse-community/parse-server#9271, the CI passes. That is not good, because w/o integration test we cannot really know whether this all works together. That has been less of an issue in the past, where this adapter just had to comply with the interface. But now that it depends on a specific implementation of the interface (async support for interface methods), it would be good to test. Since none of the parse server adapters does integration tests, this is not an exception, and we could go ahead and merge this, but someone should test this end-to-end and confirm this is working before we merge.

[vahidalizad]

Continuing from #220 (comment).

I notice that this CI does not test against Parse Server, so with or without parse-community/parse-server#9271, the CI passes. That is not good, because w/o integration test we cannot really know whether this all works together. That has been less of an issue in the past, where this adapter just had to comply with the interface. But now that it depends on a specific implementation of the interface (async support for interface methods), it would be good to test. Since none of the parse server adapters does integration tests, this is not an exception, and we could go ahead and merge this, but someone should test this end-to-end and confirm this is working before we merge.

I completely agree that testing integrations is crucial, especially given the dependency on async support for interface methods. That said, I think we could merge this into a separate branch for now and make adding integration tests a separate task. We could raise an issue for that and have others contribute to the testing. Once the tests are complete and we’ve confirmed everything works end-to-end, we can merge it into the master branch.

[mtrezza]

Our CD automation setup currently does not allow maintaining feature branches. Hence we'll have to merge this into master.

I think it would be enough if you could test this out with the latest version of Parse Server that supports the async interface. Maybe @mman also has time to test this out, just to confirm it works. Then we can go ahead and merge this and open a Server e2e test as a separate issue for later on.

[mman]

@mtrezza I would love to test it but currently I am low-priority stuck on two conflicting issues:

const ParsePushAdapter = require('@parse/push-adapter').ParsePushAdapter;

that I used to work around push adapter version issues fails with: Error [ERR_REQUIRE_ESM]: require() of ES Module not supported..

And when I convert my code to ESM, parse server fails to start with (simplified):

Error [ERR_REQUIRE_ESM]: require() of ES Module ./cloud/main.js from ./node_modules/parse-server/lib/ParseServer.js not supported.

not sure what is the solution yet but will have a time to look into it next week...

[mtrezza]

@mman does this help you? parse-community/parse-server#9316 (comment)

[mman]

@mman does this help you? parse-community/parse-server#9316 (comment)

I think it's actually this one: parse-community/parse-server#7559 with workaround mentioned here: https://stackoverflow.com/questions/77638089/how-to-make-parse-server-detect-im-using-esm-now where instead of passing a cloud code path to Parse Server constructor, you actually have to import cloud code in your main .js file and pass the imported code to the constructor...

[vahidalizad]

OK in that case I'm gonna add the new integration tests too and try to sync the dependencies with the parse-server itself lets have a cleaning task as well because all the codes have changed since the last update

[mtrezza]

That would be great. We should add them in a different PR and keep this one open. Just because we now have a presumably working PR that we could be merged in case the integration test PR stales. I have opened a new issue #222 for that.

There are repos that have integration tests which you could use as a guideline on how to set up Parse Server. See for example the Parse JS SDK - it may not have the cleanest implementation because it developed over time, but as a starting point it may be fine.

[mtrezza]

Great work, just a few tiny things that are quickly to resolve. One thing however is that instead of spying in some tests it is using a mock S3 client, which we should avoid if possible. Could you revert that back to spying?

[mtrezza]

Looks good! Thanks for your effort. I think this is good to merge. Just to confirm, there is no breaking change in this PR, correct? In other words there's no action required for the developer when updating the s3 storage adapter - not in code nor in AWS S3 config.

@mman have you found anything unexpected while you've been testing the adapter?

[jimnor0xF]

Tried running this in my env, ran into some issues since I am using the s3overrides config option and passing credentials explicitly at the moment.

Erroring out with: error: Could not load credentials from any providers

import S3Adapter from '@parse/s3-files-adapter';

const awsS3Config = {
  bucket: R2_BUCKET,
  region: 'auto',
  directAccess: true,
  fileAcl: 'none',
  baseUrl: R2_BASE_URL,
  baseUrlDirect: true,
  signatureVersion: 'v4',
  s3overrides: {
    accessKeyId: R2_ACCESS_KEY_ID,
    secretAccessKey: R2_ACCESS_KEY_SECRET,
    endpoint: `https://${CF_ACCOUNT_NUMBER}.r2.cloudflarestorage.com`,
  },
};


export const config = {
...
  filesAdapter: new S3Adapter(awsS3Config),
}

However, I see that in README.md the following is mentioned, so this should be working still?

If for some reason you really need to be able to set the key and secret explicitly, you can still do it using s3overrides as described below and setting accessKeyId and secretAccessKey in the s3Overrides object.

[jimnor0xF]

However, I see that in README.md the following is mentioned, so this should be working still?

If for some reason you really need to be able to set the key and secret explicitly, you can still do it using s3overrides as described below and setting accessKeyId and secretAccessKey in the s3Overrides object.

@vahidalizad
Looks like this is because it was previously enough to merge options.s3overrides into s3Options.

I see that in v3 you now need to put the secret key/id under s3Options.credentials. That case is handled when you have secretKey and accessKey right under options. But not in the case when the secrets are put into s3overrides.

Besides above, I tested with a working configuration and it seems to be working perfectly. Thanks for your efforts! 😄

[mtrezza]

In general we want to avoid breaking changes if possible. If the benefit is worth a breaking change, then we can adopt it and release as a new major version. Otherwise, we'd add the missing test case for how @jimnor0xF sets the credentials in v2 and we'd adapt this PR to make the same test pass with v3.

[vahidalizad]

Thank you for the feedback!

You're correct, the AWS SDK v3 moved all credential-related values from the main parameters into a dedicated credentials key. I believe this change was made to consolidate the growing number of credential-related parameters into a single, structured object for better organization.

In my opinion, it would be better to align with the SDK v3 approach to avoid potential confusion and ensure consistency with their design. We could document this change clearly in the README

That said, if you’d prefer to avoid the breaking change, I can adjust the PR to extend the main parameters for backwards compatibility. Should we limit this extension to just accessKeyId and secretAccessKey, or include all credential-related fields?

Let me know your thoughts

[mtrezza]

In my opinion, it would be better to align with the SDK v3 approach to avoid potential confusion and ensure consistency with their design. We could document this change clearly in the README

I'd agree. Could you add a small migration guide to the README? And I think we'd need an additional test case, because the credential issue hasn't been picked up by the CI. The test would use the v3 credential structure to identify a similar issue in case the credential structure changes again in the future. We'll then publish a release 4.0 with your PR.

[mman]

@vahidalizad Could you please take a look at my issue and suggested fixes at #230. If I am not mistaken in my investigation the AWS S3 JS SDK v3 changed the behavior from of CreateBucketCommand compared to v2 and breaks Digital Ocean Spaces compatibility.