Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Migrate S3 Client from AWS SDK v2 (deprecated) to v3 #221

Open
wants to merge 60 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
98cf010
migrate initialization and createBucket function
vahidalizad Aug 5, 2024
6f49645
install aws s3 v3 packages
vahidalizad Aug 5, 2024
889ca3e
add credentials to set-up tests
vahidalizad Aug 5, 2024
a9a0d46
migrate endpoint test to v3 pattern
vahidalizad Aug 5, 2024
ed0fd9e
migrate aws functions to v3
vahidalizad Aug 7, 2024
dfe0fb2
fix testing of presignedUrl
vahidalizad Aug 7, 2024
06f8483
fix mock s3 client for the create file tests
vahidalizad Aug 7, 2024
771b533
add some tests inline to debug
vahidalizad Aug 7, 2024
f493800
fix output of getFileData to be a full Buffer
vahidalizad Aug 7, 2024
e5dcf37
add fileStream tests
vahidalizad Aug 7, 2024
0897b90
change handleFileStream response handling
vahidalizad Aug 7, 2024
3ff1c77
setup fake adapter for testing to work offline as well
vahidalizad Aug 7, 2024
92d4c10
Update index.js
mtrezza Aug 7, 2024
bf36a95
remove dotenv package
vahidalizad Aug 7, 2024
da2c3e1
reduce the wating time on resolving the function getSignedUrl
vahidalizad Aug 7, 2024
2eef4a1
remove unnecessary initialization comments
vahidalizad Aug 7, 2024
05b2035
fix handleFileStream error handling
vahidalizad Aug 10, 2024
0d3e766
fix test getFileStream tests to run offline
vahidalizad Aug 10, 2024
d4062cb
remove deasync and use async getFileLocation
vahidalizad Aug 10, 2024
a69cb5b
sync dependencies with parse-server and add prettier config
vahidalizad Oct 10, 2024
a5928f3
remove unused dependencies
vahidalizad Oct 10, 2024
5ab5614
add integration tests
vahidalizad Oct 10, 2024
bc08dfb
prettier format all files
vahidalizad Oct 10, 2024
1704457
update packages
vahidalizad Oct 10, 2024
1f1257d
fix coverage directory
vahidalizad Oct 10, 2024
d738c85
add ci command
vahidalizad Oct 10, 2024
e50978d
add integration test command
vahidalizad Oct 10, 2024
3fbe88a
revert @semantic dependencies
vahidalizad Oct 10, 2024
a71372a
Merge branch 'master' into aws-sdk-v3-async
vahidalizad Oct 22, 2024
82bef30
fix merge conflicts of prettier and file names
vahidalizad Oct 22, 2024
b3c2dad
fix lint
vahidalizad Oct 22, 2024
3024336
Update README.md
mtrezza Oct 22, 2024
917f3c4
remove parse server 6 compat
mtrezza Oct 22, 2024
b52723b
run integration tests only for parse server >=7
mtrezza Oct 22, 2024
5d39edd
refactor mock
mtrezza Oct 22, 2024
bf7bfc2
fix mock
mtrezza Oct 22, 2024
daee5e6
remove Parse import
mtrezza Oct 22, 2024
1d5964f
add support for other endpoints
vahidalizad Dec 9, 2024
22071e9
test: add command instance check for s3 send calls
vahidalizad Dec 9, 2024
fe83a76
add test for credentials options
vahidalizad Dec 10, 2024
f4de1fb
add the test for handling stream errors
vahidalizad Dec 10, 2024
ed79c7c
Update index.js
mtrezza Dec 11, 2024
4089ed2
Update index.js
mtrezza Dec 11, 2024
b80db7d
Update spec/test.spec.js
mtrezza Dec 11, 2024
7161e70
Update index.js
mtrezza Dec 11, 2024
8a1163a
Update index.js
mtrezza Dec 11, 2024
fa13535
Update index.js
mtrezza Dec 11, 2024
81f4486
Update spec/test.spec.js
mtrezza Dec 11, 2024
843fef4
Update spec/test.spec.js
mtrezza Dec 11, 2024
1910b59
Update spec/test.spec.js
mtrezza Dec 11, 2024
f4fcda6
hotfix error interpreting index.js
vahidalizad Dec 11, 2024
4a58f6b
remove some mock s3 clients
vahidalizad Dec 11, 2024
74232a4
fix async expectations in unit tests
vahidalizad Dec 11, 2024
f264067
add badges to README
mtrezza Dec 15, 2024
5871b42
Update README.md
mtrezza Dec 15, 2024
77244c6
fix issue #230
vahidalizad Dec 24, 2024
498b043
fix unit tests of bucket operations
vahidalizad Dec 24, 2024
6503652
add credentials tests
vahidalizad Dec 24, 2024
bb83ccb
update readme to add breaking changes
vahidalizad Dec 24, 2024
becb402
fix readme links
vahidalizad Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@
"globals": {
"Parse": true
}
}
}
3 changes: 0 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,6 @@ jobs:
- name: Parse Server 7, Node.js 22
NODE_VERSION: 22.4.1
PARSE_SERVER_VERSION: 7
- name: Parse Server 6
NODE_VERSION: 18
PARSE_SERVER_VERSION: 6
fail-fast: false
name: ${{ matrix.name }}
timeout-minutes: 15
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,4 +177,4 @@ ___



\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,21 +40,24 @@ The official AWS S3 file storage adapter for Parse Server. See [Parse Server S3

### Parse Server

| Version | End-of-Life | Compatible |
|---------|---------------|------------|
| <=5 | December 2023 | ✅ Yes |
| 6 | December 2024 | ✅ Yes |
| 7 | December 2025 | ✅ Yes |
Parse Server S3 Adapter is compatible with the following versions of Parse Server.

| Parse Server Version | End-of-Life | Compatible |
|----------------------|---------------|------------|
| <=5 | December 2023 | ❌ No |
| 6 | December 2024 | ❌ No |
| <7.3.0 | December 2025 | ❌ No |
| >=7.3.0 | December 2025 | ✅ Yes |

### Node.js

This product is continuously tested with the most recent releases of Node.js to ensure compatibility. We follow the [Node.js Long Term Support plan](https://github.com/nodejs/Release) and only test against versions that are officially supported and have not reached their end-of-life date.
Parse Server S3 Adapter is continuously tested with the most recent releases of Node.js to ensure compatibility. We follow the [Node.js Long Term Support plan](https://github.com/nodejs/Release) and only test against versions that are officially supported and have not reached their end-of-life date.

| Version | Latest Version | End-of-Life | Compatible |
|------------|----------------|-------------|------------|
| Node.js 18 | 18.20.4 | April 2025 | ✅ Yes |
| Node.js 20 | 20.15.1 | April 2026 | ✅ Yes |
| Node.js 22 | 22.4.1 | April 2027 | ✅ Yes |
| Node.js Version | End-of-Life | Compatible |
|-----------------|-------------|------------|
| 18 | April 2025 | ✅ Yes |
| 20 | April 2026 | ✅ Yes |
| 22 | April 2027 | ✅ Yes |

## AWS Credentials

Expand Down Expand Up @@ -277,7 +280,6 @@ var S3Adapter = require("@parse/s3-files-adapter");
var AWS = require("aws-sdk");

//Configure Digital Ocean Spaces EndPoint
const spacesEndpoint = new AWS.Endpoint(process.env.SPACES_ENDPOINT);
var s3Options = {
bucket: process.env.SPACES_BUCKET_NAME,
baseUrl: process.env.SPACES_BASE_URL,
Expand All @@ -290,7 +292,7 @@ var s3Options = {
s3overrides: {
accessKeyId: process.env.SPACES_ACCESS_KEY,
secretAccessKey: process.env.SPACES_SECRET_KEY,
endpoint: spacesEndpoint
endpoint: process.env.SPACES_ENDPOINT
}
};

Expand All @@ -306,4 +308,4 @@ var api = new ParseServer({
allowClientClassCreation: false,
filesAdapter: s3Adapter
});
```
```
186 changes: 106 additions & 80 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,27 @@
//
// Stores Parse files in AWS S3.

const AWS = require('aws-sdk');
const {
S3Client,
CreateBucketCommand,
PutObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
} = require('@aws-sdk/client-s3');
const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');
const optionsFromArguments = require('./lib/optionsFromArguments');

const awsCredentialsDeprecationNotice = function awsCredentialsDeprecationNotice() {
// eslint-disable-next-line no-console
console.warn('Passing AWS credentials to this adapter is now DEPRECATED and will be removed in a future version',
'See: https://github.com/parse-server-modules/parse-server-s3-adapter#aws-credentials for details');
console.warn(
'Passing AWS credentials to this adapter is now DEPRECATED and will be removed in a future version',
'See: https://github.com/parse-server-modules/parse-server-s3-adapter#aws-credentials for details'
);
};

const serialize = (obj) => {
const serialize = obj => {
const str = [];
Object.keys(obj).forEach((key) => {
Object.keys(obj).forEach(key => {
if (obj[key]) {
str.push(`${encodeURIComponent(key)}=${encodeURIComponent(obj[key])}`);
}
Expand All @@ -36,6 +45,15 @@ function buildDirectAccessUrl(baseUrl, baseUrlFileKey, presignedUrl, config, fil
return directAccessUrl;
}

function responseToBuffer(response) {
return new Promise((resolve, reject) => {
const chunks = [];
response.Body.on('data', chunk => chunks.push(chunk));
response.Body.on('end', () => resolve(Buffer.concat(chunks)));
response.Body.on('error', reject);
});
}

class S3Adapter {
// Creates an S3 session.
// Providing AWS access, secret keys and bucket are mandatory
Expand All @@ -55,6 +73,7 @@ class S3Adapter {
this._presignedUrlExpires = parseInt(options.presignedUrlExpires, 10);
this._encryption = options.ServerSideEncryption;
this._generateKey = options.generateKey;
this._endpoint = options.s3overrides?.endpoint;
mtrezza marked this conversation as resolved.
Show resolved Hide resolved
// Optional FilesAdaptor method
this.validateFilename = options.validateFilename;

Expand All @@ -65,6 +84,16 @@ class S3Adapter {
globalCacheControl: this._globalCacheControl,
};

if (options.accessKey && options.secretKey) {
awsCredentialsDeprecationNotice();
s3Options.credentials = {
accessKeyId: options.accessKey,
secretAccessKey: options.secretKey,
};
} else if (options.credentials) {
s3Options.credentials = options.credentials;
vahidalizad marked this conversation as resolved.
Show resolved Hide resolved
}

if (options.accessKey && options.secretKey) {
awsCredentialsDeprecationNotice();
s3Options.accessKeyId = options.accessKey;
Expand All @@ -73,29 +102,27 @@ class S3Adapter {

Object.assign(s3Options, options.s3overrides);

this._s3Client = new AWS.S3(s3Options);
this._s3Client = new S3Client(s3Options);
this._hasBucket = false;
}

createBucket() {
let promise;
if (this._hasBucket) {
promise = Promise.resolve();
} else {
promise = new Promise((resolve) => {
this._s3Client.createBucket(() => {
this._hasBucket = true;
resolve();
});
});
async createBucket() {
if (this._hasBucket) { return; }
mtrezza marked this conversation as resolved.
Show resolved Hide resolved

try {
await this._s3Client.send(new CreateBucketCommand({ Bucket: this._bucket }));
this._hasBucket = true;
} catch (error) {
if (error.name === 'BucketAlreadyOwnedByYou') { this._hasBucket = true; }
else { throw error; }
mtrezza marked this conversation as resolved.
Show resolved Hide resolved
}
return promise;
}

// For a given config object, filename, and data, store a file in S3
// Returns a promise containing the S3 object creation response
createFile(filename, data, contentType, options = {}) {
async createFile(filename, data, contentType, options = {}) {
const params = {
Bucket: this._bucket,
Key: this._bucketPrefix + filename,
Body: data,
};
Expand Down Expand Up @@ -128,52 +155,51 @@ class S3Adapter {
const serializedTags = serialize(options.tags);
params.Tagging = serializedTags;
}
return this.createBucket().then(() => new Promise((resolve, reject) => {
this._s3Client.upload(params, (err, response) => {
if (err !== null) {
return reject(err);
}
return resolve(response);
});
}));
await this.createBucket();
const command = new PutObjectCommand(params);
const response = await this._s3Client.send(command);
const endpoint = this._endpoint || `https://${this._bucket}.s3.${this._region}.amazonaws.com`;
const location = `${endpoint}/${params.Key}`;

return Object.assign(response || {}, { Location: location });
}

deleteFile(filename) {
return this.createBucket().then(() => new Promise((resolve, reject) => {
const params = {
Key: this._bucketPrefix + filename,
};
this._s3Client.deleteObject(params, (err, data) => {
if (err !== null) {
return reject(err);
}
return resolve(data);
});
}));
async deleteFile(filename) {
const params = {
Bucket: this._bucket,
Key: this._bucketPrefix + filename,
};
await this.createBucket();
const command = new DeleteObjectCommand(params);
const response = await this._s3Client.send(command);
return response;
}

// Search for and return a file if found by filename
// Returns a promise that succeeds with the buffer result from S3
getFileData(filename) {
const params = { Key: this._bucketPrefix + filename };
return this.createBucket().then(() => new Promise((resolve, reject) => {
this._s3Client.getObject(params, (err, data) => {
if (err !== null) {
return reject(err);
}
// Something happened here...
if (data && !data.Body) {
return reject(data);
}
return resolve(data.Body);
});
}));
async getFileData(filename) {
const params = {
Bucket: this._bucket,
Key: this._bucketPrefix + filename,
};
await this.createBucket();
const command = new GetObjectCommand(params);
const response = await this._s3Client.send(command);
if (response && !response.Body) { throw new Error(response); }
mtrezza marked this conversation as resolved.
Show resolved Hide resolved

const buffer = await responseToBuffer(response);
return buffer;
}

// Exposed only for testing purposes
getFileSignedUrl(client, command, options) {
return getSignedUrl(client, command, options);
}

// Generates and returns the location of a file stored in S3 for the given request and filename
// The location is the direct S3 link if the option is set,
// otherwise we serve the file through parse-server
getFileLocation(config, filename) {
async getFileLocation(config, filename) {
const fileName = filename.split('/').map(encodeURIComponent).join('/');
if (!this._directAccess) {
return `${config.mount}/files/${config.applicationId}/${fileName}`;
Expand All @@ -184,12 +210,11 @@ class S3Adapter {
let presignedUrl = '';
if (this._presignedUrl) {
const params = { Bucket: this._bucket, Key: fileKey };
if (this._presignedUrlExpires) {
params.Expires = this._presignedUrlExpires;
}
// Always use the "getObject" operation, and we recommend that you protect the URL
// appropriately: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
presignedUrl = this._s3Client.getSignedUrl('getObject', params);
const options = this._presignedUrlExpires ? { expiresIn: this._presignedUrlExpires } : {};

const command = new GetObjectCommand(params);
presignedUrl = await this.getFileSignedUrl(this._s3Client, command, options);

if (!this._baseUrl) {
return presignedUrl;
}
Expand All @@ -203,30 +228,31 @@ class S3Adapter {
return buildDirectAccessUrl(this._baseUrl, baseUrlFileKey, presignedUrl, config, filename);
}

handleFileStream(filename, req, res) {
async handleFileStream(filename, req, res) {
const params = {
Bucket: this._bucket,
Key: this._bucketPrefix + filename,
Range: req.get('Range'),
};
return this.createBucket().then(() => new Promise((resolve, reject) => {
this._s3Client.getObject(params, (error, data) => {
if (error !== null) {
return reject(error);
}
if (data && !data.Body) {
return reject(data);
}
res.writeHead(206, {
'Accept-Ranges': data.AcceptRanges,
'Content-Length': data.ContentLength,
'Content-Range': data.ContentRange,
'Content-Type': data.ContentType,
});
res.write(data.Body);
res.end();
return resolve(data.Body);
});
}));

await this.createBucket();
const command = new GetObjectCommand(params);
const data = await this._s3Client.send(command);
if (data && !data.Body) { throw new Error('S3 object body is missing.'); }
mtrezza marked this conversation as resolved.
Show resolved Hide resolved

res.writeHead(206, {
'Accept-Ranges': data.AcceptRanges,
'Content-Length': data.ContentLength,
'Content-Range': data.ContentRange,
'Content-Type': data.ContentType,
});
data.Body.on('data', chunk => res.write(chunk));
data.Body.on('end', () => res.end());
data.Body.on('error', e => {
res.status(404);
res.send(e.message);
vahidalizad marked this conversation as resolved.
Show resolved Hide resolved
});
return responseToBuffer(data);
}
}

Expand Down
7 changes: 4 additions & 3 deletions lib/optionsFromArguments.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,12 @@
} else if (args.length === 2) {
options.bucket = stringOrOptions;
if (typeof args[1] !== 'object') {
throw new Error('Failed to configure S3Adapter. Arguments don\'t make sense');
throw new Error("Failed to configure S3Adapter. Arguments don't make sense");

Check warning on line 45 in lib/optionsFromArguments.js

View check run for this annotation

Codecov / codecov/patch

lib/optionsFromArguments.js#L45

Added line #L45 was not covered by tests
}
otherOptions = args[1];
} else if (args.length > 2) {
if (typeof args[1] !== 'string' || typeof args[2] !== 'string') {
throw new Error('Failed to configure S3Adapter. Arguments don\'t make sense');
throw new Error("Failed to configure S3Adapter. Arguments don't make sense");
}
options.accessKey = args[0];
options.secretKey = args[1];
Expand All @@ -57,6 +57,7 @@

if (otherOptions) {
options.bucketPrefix = otherOptions.bucketPrefix;
options.credentials = otherOptions.credentials;
options.directAccess = otherOptions.directAccess;
options.fileAcl = otherOptions.fileAcl;
options.baseUrl = otherOptions.baseUrl;
Expand All @@ -80,7 +81,7 @@
options.bucket = s3overrides.params.Bucket;
}
} else if (args.length > 2) {
throw new Error('Failed to configure S3Adapter. Arguments don\'t make sense');
throw new Error("Failed to configure S3Adapter. Arguments don't make sense");

Check warning on line 84 in lib/optionsFromArguments.js

View check run for this annotation

Codecov / codecov/patch

lib/optionsFromArguments.js#L84

Added line #L84 was not covered by tests
}

options = fromOptionsDictionaryOrDefault(options, 's3overrides', s3overrides);
Expand Down
Loading
Loading