Skip to content

Commit

Permalink
Merge branch 'w/2.8/improvement/ZENKO-4941' into tmp/octopus/w/2.9/im…
Browse files Browse the repository at this point in the history 
…provement/ZENKO-4941
  • Loading branch information
@bert-e
bert-e committed Dec 3, 2024
2 parents ef1e1b9 + 0efe925 commit 83fd0ce
Show file tree
Hide file tree
Showing 5 changed files with 167 additions and 21 deletions.
3 changes: 3 additions & 0 deletions .github/scripts/end2end/configs/zenko.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,9 @@ spec:
azure:
archiveTier: "hot"
restoreTimeout: "15s"
scuba:
logging:
logLevel: debug
ingress:
workloadPlaneClass: 'nginx'
controlPlaneClass: 'nginx-control-plane'
Expand Down
2 changes: 1 addition & 1 deletion solution/deps.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ scuba:
sourceRegistry: ghcr.io/scality
dashboard: scuba/scuba-dashboards
image: scuba
tag: 1.0.8
tag: 1.0.9
envsubst: SCUBA_TAG
sorbet:
sourceRegistry: ghcr.io/scality
Expand Down
115 changes: 115 additions & 0 deletions tests/ctst/features/quotas/Quotas.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,90 @@ Feature: Quota Management for APIs
| 100 | 0 | 200 | IAM_USER |
| 100 | 200 | 200 | IAM_USER |

@2.6.0
@PreMerge
@Quotas
@CronJob
@DataDeletion
@NonVersioned
Scenario Outline: Quotas are affected by deletion operations between count items runs
Given an action "DeleteObject"
And a permission to perform the "PutObject" action
And a STORAGE_MANAGER type
And a bucket quota set to 1000 B
And an account quota set to 1000 B
And an upload size of 1000 B for the object "obj-1"
And a bucket quota set to <bucketQuota> B
And an account quota set to <accountQuota> B
And a <userType> type
And an environment setup for the API
And an "existing" IAM Policy that "applies" with "ALLOW" effect for the current API
When I wait 3 seconds
And I PUT an object with size <uploadSize>
Then the API should "fail" with "QuotaExceeded"
When the "count-items" cronjobs completes without error
# Wait for inflights to be read by SCUBA
When I wait 3 seconds
# At this point if negative inflights are not supported, write should
# not be possible, as the previous inflights are now part of the current
# metrics.
And i delete object "obj-1"
# Wait for inflights to be read by SCUBA
And I wait 3 seconds
And I PUT an object with size <uploadSize>
Then the API should "succeed" with ""

Examples:
| uploadSize | bucketQuota | accountQuota | userType |
| 100 | 200 | 0 | ACCOUNT |
| 100 | 0 | 200 | ACCOUNT |
| 100 | 200 | 200 | ACCOUNT |
| 100 | 200 | 0 | IAM_USER |
| 100 | 0 | 200 | IAM_USER |
| 100 | 200 | 200 | IAM_USER |

@2.6.0
@PreMerge
@Quotas
@CronJob
@DataDeletion
@NonVersioned
Scenario Outline: Negative inflights do not allow to bypass the quota
Given an action "DeleteObject"
And a permission to perform the "PutObject" action
And a STORAGE_MANAGER type
And a bucket quota set to 1000 B
And an account quota set to 1000 B
And an upload size of 1000 B for the object "obj-1"
And a bucket quota set to <bucketQuota> B
And an account quota set to <accountQuota> B
And a <userType> type
And an environment setup for the API
And an "existing" IAM Policy that "applies" with "ALLOW" effect for the current API
When I wait 3 seconds
And I PUT an object with size <uploadSize>
Then the API should "fail" with "QuotaExceeded"
When the "count-items" cronjobs completes without error
# Wait for inflights to be read by SCUBA
When I wait 3 seconds
# At this point if negative inflights are not supported, write should
# not be possible, as the previous inflights are now part of the current
# metrics.
And i delete object "obj-1"
# Wait for inflights to be read by SCUBA
And I wait 3 seconds
And I PUT an object with size 1000
Then the API should "fail" with "QuotaExceeded"

Examples:
| uploadSize | bucketQuota | accountQuota | userType |
| 100 | 200 | 0 | ACCOUNT |
| 100 | 0 | 200 | ACCOUNT |
| 100 | 200 | 200 | ACCOUNT |
| 100 | 200 | 0 | IAM_USER |
| 100 | 0 | 200 | IAM_USER |
| 100 | 200 | 200 | IAM_USER |

@2.6.0
@PreMerge
@Quotas
Expand Down Expand Up @@ -129,3 +213,34 @@ Feature: Quota Management for APIs
| RestoreObject | 100 | 0 | 99 | IAM_USER | fail | QuotaExceeded | 3 |
| RestoreObject | 100 | 99 | 99 | IAM_USER | fail | QuotaExceeded | 3 |
| RestoreObject | 100 | 101 | 101 | IAM_USER | succeed | | 3 |

@2.6.0
@PreMerge
@Quotas
@Restore
@Dmf
@ColdStorage
@Only
Scenario Outline: Restored object expiration updates quotas
Given an action "<action>"
And a STORAGE_MANAGER type
And a transition workflow to "e2e-cold" location
And an upload size of <uploadSize> B for the object "obj-1"
Then object "obj-1" should be "transitioned" and have the storage class "e2e-cold"
Given a bucket quota set to <bucketQuota> B
And an account quota set to <accountQuota> B
And a <userType> type
And an environment setup for the API
And an "existing" IAM Policy that "applies" with "ALLOW" effect for the current API
When i restore object "" for 5 days
Then the API should "succeed" with ""
And object "obj-1" should be "restored" and have the storage class "e2e-cold"
Given a STORAGE_MANAGER type
Then object "obj-1" should expire in 5 days
When i wait for 5 days
Then object "obj-1" should be "cold" and have the storage class "e2e-cold"

Examples:
| action | uploadSize | bucketQuota | accountQuota | userType |
| RestoreObject | 100 | 0 | 0 | ACCOUNT |
| RestoreObject | 100 | 101 | 101 | ACCOUNT |
14 changes: 14 additions & 0 deletions tests/ctst/steps/quotas/quotas.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { Scality, Command, Utils, AWSCredentials, Constants, Identity, IdentityE
import { createJobAndWaitForCompletion } from '../utils/kubernetes';
import { createBucketWithConfiguration, putObject } from '../utils/utils';
import { hashStringAndKeepFirst20Characters } from 'common/utils';
import assert from 'assert';

export async function prepareQuotaScenarios(world: Zenko, scenarioConfiguration: ITestCaseHookParameter) {
/**
Expand Down Expand Up @@ -136,6 +137,16 @@ Given('a bucket quota set to {int} B', async function (this: Zenko, quota: numbe
result,
});

// Ensure the quota is set
const resultGet: Command = await Scality.getBucketQuota(
this.parameters,
this.getCommandParameters());
this.logger.debug('GetBucketQuota result', {
resultGet,
});

assert(resultGet.stdout.includes(`${quota}`));

if (result.err) {
throw new Error(result.err);
}
Expand All @@ -158,6 +169,9 @@ Given('an account quota set to {int} B', async function (this: Zenko, quota: num
result,
});

// Ensure the quota is set
assert(JSON.parse(result.stdout).quota === quota);

if (result.err) {
throw new Error(result.err);
}
Expand Down
54 changes: 34 additions & 20 deletions tests/ctst/world/Zenko.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import { AccessKey } from '@aws-sdk/client-iam';
import { Credentials } from '@aws-sdk/client-sts';
import { aws4Interceptor } from 'aws4-axios';
import qs from 'qs';
import fs from 'fs';
import lockFile from 'proper-lockfile';
import Werelogs from 'werelogs';
import {
CacheHelper,
Expand Down Expand Up @@ -633,24 +635,36 @@ export default class Zenko extends World<ZenkoWorldParameters> {

if (!Identity.hasIdentity(IdentityEnum.ACCOUNT, accountName)) {
Identity.useIdentity(IdentityEnum.ADMIN, site.adminIdentityName);

const filePath = `/tmp/account-init-${accountName}.json`;
if (!fs.existsSync(filePath)) {
fs.writeFileSync(filePath, JSON.stringify({
ready: false,
}));
}
let account = null;
CacheHelper.logger.debug('Creating account', {
accountName,
adminIdentityName: site.adminIdentityName,
credentials: Identity.getCurrentCredentials(),
});
// Create the account if already exist will not throw any error
let releaseLock: (() => Promise<void>) | null = null;
try {
await SuperAdmin.createAccount({ accountName });
/* eslint-disable */
} catch (err: any) {
CacheHelper.logger.debug('Error while creating account', {
accountName,
err,
releaseLock = await lockFile.lock(filePath, {
stale: Constants.DEFAULT_TIMEOUT / 2,
retries: {
retries: 5,
factor: 3,
minTimeout: 1000,
maxTimeout: 5000,
}
});
if (!err.EntityAlreadyExists && err.code !== 'EntityAlreadyExists') {
throw err;

try {
await SuperAdmin.createAccount({ accountName });
/* eslint-disable */
} catch (err: any) {
if (!err.EntityAlreadyExists && err.code !== 'EntityAlreadyExists') {
throw err;
}
}
} finally {
if (releaseLock) {
await releaseLock();
}
}
/* eslint-enable */
Expand Down Expand Up @@ -693,7 +707,7 @@ export default class Zenko extends World<ZenkoWorldParameters> {
const accountName = this.sites['source']?.accountName || CacheHelper.parameters.AccountName!;
const accountAccessKeys = Identity.getCredentialsForIdentity(
IdentityEnum.ACCOUNT, this.sites['source']?.accountName
|| CacheHelper.parameters.AccountName!) || {
|| CacheHelper.parameters.AccountName!) || {
accessKeyId: '',
secretAccessKey: '',
};
Expand Down Expand Up @@ -865,7 +879,7 @@ export default class Zenko extends World<ZenkoWorldParameters> {
}

async awsS3Request(method: Method, path: string,
userCredentials: AWSCredentials, headers: object = {}, payload: object = {}) : Promise<Command> {
userCredentials: AWSCredentials, headers: object = {}, payload: object = {}): Promise<Command> {
const interceptor = aws4Interceptor({
options: {
region: 'us-east-1',
Expand All @@ -891,7 +905,7 @@ export default class Zenko extends World<ZenkoWorldParameters> {
statusCode: response.status,
data: response.data as unknown,
};
/* eslint-disable */
/* eslint-disable */
} catch (err: any) {
return {
stdout: '',
Expand Down Expand Up @@ -967,7 +981,7 @@ export default class Zenko extends World<ZenkoWorldParameters> {
}
}

async addWebsiteEndpoint(this: Zenko, endpoint: string) :
async addWebsiteEndpoint(this: Zenko, endpoint: string):
Promise<{ statusCode: number; data: object } | { statusCode: number; err: unknown }> {
return await this.managementAPIRequest('POST',
`/config/${this.parameters.InstanceID}/website/endpoint`,
Expand All @@ -977,7 +991,7 @@ export default class Zenko extends World<ZenkoWorldParameters> {
`"${endpoint}"`);
}

async deleteLocation(this: Zenko, locationName: string) :
async deleteLocation(this: Zenko, locationName: string):
Promise<{ statusCode: number; data: object } | { statusCode: number; err: unknown }> {
return await this.managementAPIRequest('DELETE',
`/config/${this.parameters.InstanceID}/location/${locationName}`);
Expand Down

0 comments on commit 83fd0ce

Please sign in to comment.