Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for scheduling functions #1527

Merged
merged 26 commits into from
Jul 19, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
acec70c
feat: add support for scheduling functions
rtpascual May 16, 2024
a5d6533
update valid natural language and cron
rtpascual May 17, 2024
afbd6f3
Merge branch 'main' into schedule-functions
rtpascual May 17, 2024
f104c9a
Merge branch 'main' into schedule-functions
rtpascual Jun 11, 2024
4346570
pr feedback and add unit tests
rtpascual Jun 11, 2024
d7f26db
add test that throws when schedule will invoke function again before …
rtpascual Jun 14, 2024
562e275
update schedule rate validation
rtpascual Jun 18, 2024
255dfc5
Merge branch 'main' into schedule-functions
rtpascual Jul 9, 2024
46d8d08
Merge branch 'main' into schedule-functions
rtpascual Jul 10, 2024
3e0bccd
add tests
rtpascual Jul 10, 2024
4f797ec
update e2e test
rtpascual Jul 10, 2024
d86ea22
fix e2e test
rtpascual Jul 10, 2024
8e8063f
PR feedback
rtpascual Jul 15, 2024
760888a
Merge branch 'main' into schedule-functions
rtpascual Jul 15, 2024
79dfa9f
fix test
rtpascual Jul 15, 2024
2d11214
add more testing
rtpascual Jul 16, 2024
3c846fb
fix lint and e2e test
rtpascual Jul 16, 2024
f6821ba
try this
rtpascual Jul 16, 2024
f0311eb
try this
rtpascual Jul 16, 2024
0d7e55a
try that
rtpascual Jul 16, 2024
6b89ceb
try this
rtpascual Jul 16, 2024
46a2192
Merge branch 'main' into schedule-functions
rtpascual Jul 16, 2024
2950c72
Update packages/backend-function/src/factory.ts
rtpascual Jul 17, 2024
0fa4a84
Merge branch 'main' into schedule-functions
rtpascual Jul 17, 2024
28a6825
pr feedback
rtpascual Jul 17, 2024
ca2d0bd
pr feedback
rtpascual Jul 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/green-coins-lie.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@aws-amplify/backend-function': minor
---

add support for scheduling functions
1 change: 1 addition & 0 deletions .eslint_dictionary.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"commonjs",
"corepack",
"cors",
"cron",
"ctor",
"darwin",
"datastore",
Expand Down
10 changes: 10 additions & 0 deletions packages/backend-function/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ import { FunctionResources } from '@aws-amplify/plugin-types';
import { ResourceAccessAcceptorFactory } from '@aws-amplify/plugin-types';
import { ResourceProvider } from '@aws-amplify/plugin-types';

// @public (undocumented)
export type Cron = `${string} ${string} ${string} ${string} ${string}` | `${string} ${string} ${string} ${string} ${string} ${string}`;

// @public
export const defineFunction: (props?: FunctionProps) => ConstructFactory<ResourceProvider<FunctionResources> & ResourceAccessAcceptorFactory>;

Expand All @@ -21,11 +24,18 @@ export type FunctionProps = {
memoryMB?: number;
environment?: Record<string, string | BackendSecret>;
runtime?: NodeVersion;
schedule?: TimeInterval | TimeInterval[];
};

// @public (undocumented)
export type NodeVersion = 16 | 18 | 20;

// @public (undocumented)
export type Rate = `every ${number}m` | `every ${number}h` | `every day` | `every week` | `every month` | `every year`;
edwardfoyle marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion for alternative namings:

type TimeInterval = `every xx`;
type CronSchedule = `{string} ...`
type Schedule = TimeInterval | CronSchedule

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was avoiding type Schedule because of the naming clash with the Schedule class from aws-cdk-lib/aws-events. Alternatively we can use FunctionSchedule?


// @public (undocumented)
export type TimeInterval = Rate | Cron;

// (No @packageDocumentation comment for this package)

```
218 changes: 217 additions & 1 deletion packages/backend-function/src/factory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,23 @@ import { FunctionEnvironmentTypeGenerator } from './function_env_type_generator.
import { AttributionMetadataStorage } from '@aws-amplify/backend-output-storage';
import { fileURLToPath } from 'url';
import { AmplifyUserError, TagName } from '@aws-amplify/platform-core';
import { CronOptions, Rule, Schedule } from 'aws-cdk-lib/aws-events';
import * as targets from 'aws-cdk-lib/aws-events-targets';

const functionStackType = 'function-Lambda';

export type Cron =
| `${string} ${string} ${string} ${string} ${string}`
| `${string} ${string} ${string} ${string} ${string} ${string}`;
export type Rate =
| `every ${number}m`
| `every ${number}h`
| `every day`
| `every week`
| `every month`
| `every year`;
export type TimeInterval = Rate | Cron;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Types were done this way because I started getting Expression produces a union type that is too complex to represent Typescript error when I introduced wildcards for cron. Cron type is less strict and we perform our own cron validation to validate what we support (which could change in the future).


/**
* Entry point for defining a function in the Amplify ecosystem
*/
Expand Down Expand Up @@ -87,6 +101,19 @@ export type FunctionProps = {
* Defaults to the oldest NodeJS LTS version. See https://nodejs.org/en/about/previous-releases
*/
runtime?: NodeVersion;

/**
* A time interval string to periodically run the function.
* This can be either a string of `"every <positive whole number><m (minute) or h (hour)>"`, `"every day|week|month|year"` or cron expression.
* Defaults to no scheduling for the function.
* @example
* schedule: "every 5m"
* @example
* schedule: "every week"
* @example
* schedule: "0 9 * * 2" // every Monday at 9am
*/
schedule?: TimeInterval | TimeInterval[];
};

/**
Expand Down Expand Up @@ -131,6 +158,7 @@ class FunctionFactory implements ConstructFactory<AmplifyFunction> {
memoryMB: this.resolveMemory(),
environment: this.props.environment ?? {},
runtime: this.resolveRuntime(),
schedule: this.resolveSchedule(),
};
};

Expand Down Expand Up @@ -220,6 +248,33 @@ class FunctionFactory implements ConstructFactory<AmplifyFunction> {

return this.props.runtime;
};

private resolveSchedule = () => {
edwardfoyle marked this conversation as resolved.
Show resolved Hide resolved
if (!this.props.schedule) {
return [];
}

const schedules = Array.isArray(this.props.schedule)
awsluja marked this conversation as resolved.
Show resolved Hide resolved
? this.props.schedule
: [this.props.schedule];

schedules.forEach((schedule) => {
if (isRate(schedule)) {
const { value } = parseRate(schedule);

if (value && !isPositiveWholeNumber(value)) {
throw new Error(`schedule must be set with a positive whole number`);
}
} else {
if (!isValidCron(schedule)) {
// TODO: Better error messaging here or throw within each part of isValidCron in order to give more concise error messages
throw new Error(`schedule cron expression is not valid`);
}
}
});

return this.props.schedule;
};
}

type HydratedFunctionProps = Required<FunctionProps>;
Expand Down Expand Up @@ -296,7 +351,7 @@ class AmplifyFunction
// This will be overwritten with the typed file at the end of synthesis
functionEnvironmentTypeGenerator.generateProcessEnvShim();

let functionLambda;
let functionLambda: NodejsFunction;
try {
functionLambda = new NodejsFunction(scope, `${id}-lambda`, {
entry: props.entry,
Expand All @@ -323,6 +378,30 @@ class AmplifyFunction
);
}

try {
const timeIntervals = Array.isArray(props.schedule)
? props.schedule
: [props.schedule];

timeIntervals.forEach((interval, index) => {
// Lambda name will be prepended to rule id, so only using index here for uniqueness
const rule = new Rule(this, `lambda-schedule${index}`, {
schedule: Schedule.cron(translateToCronOptions(interval)),
});

rule.addTarget(new targets.LambdaFunction(functionLambda));
});
} catch (error) {
throw new AmplifyUserError(
'NodeJSFunctionScheduleInitializationError',
rtpascual marked this conversation as resolved.
Show resolved Hide resolved
{
message: 'Failed to instantiate schedule for nodejs function',
resolution: 'See the underlying error message for more details.',
},
error as Error
);
}

Tags.of(functionLambda).add(TagName.FRIENDLY_NAME, id);

this.functionEnvironmentTranslator = new FunctionEnvironmentTranslator(
Expand Down Expand Up @@ -396,3 +475,140 @@ const nodeVersionMap: Record<NodeVersion, Runtime> = {
18: Runtime.NODEJS_18_X,
20: Runtime.NODEJS_20_X,
};

const isRate = (timeInterval: TimeInterval): timeInterval is Rate => {
return timeInterval.split(' ')[0] === 'every';
};

const parseRate = (rate: Rate) => {
const interval = rate.split(' ')[1];

const regex = /\d/;
if (interval.match(regex)) {
return {
value: parseInt(interval.substring(0, interval.length - 1)),
unit: interval.charAt(interval.length - 1),
};
}

return {
unit: interval,
};
};

const isPositiveWholeNumber = (test: number) => test > 0 && test % 1 === 0;

const isValidCron = (cron: Cron): boolean => {
const cronParts = cron.split(' ');

if (cronParts.length !== 5 && cronParts.length !== 6) {
return false;
}

const [minute, hour, dayOfMonth, month, dayOfWeek, year] = cronParts;

return (
isValidCronPart(minute, 0, 59) &&
isValidCronPart(hour, 0, 23) &&
(dayOfMonth === '?' || isValidCronPart(dayOfMonth, 1, 31)) &&
isValidCronPart(month, 1, 12) &&
(dayOfWeek === '?' || isValidCronPart(dayOfWeek, 1, 7)) &&
(!year || isValidCronPart(year, 1970, 2199))
);
};

const isValidCronPart = (part: string, min: number, max: number): boolean => {
if (part === '*') {
return true;
}
if (part.includes('/')) {
return isValidStepValue(part, min, max);
}
if (part.includes('-')) {
return isValidRange(part, min, max);
}
if (part.includes(',')) {
return isValidList(part, min, max);
}

return isWholeNumberBetweenInclusive(Number(part), min, max);
};

const isValidStepValue = (value: string, min: number, max: number): boolean => {
const originalBase = value.split('/')[0];
const [base, step] = value.split('/').map(Number);

if (originalBase === '*') {
return !isNaN(step) && step > 0;
}

return (
!isNaN(base) &&
!isNaN(step) &&
isWholeNumberBetweenInclusive(base, min, max) &&
step > 0
);
};

const isValidRange = (value: string, min: number, max: number): boolean => {
const [start, end] = value.split('-').map(Number);
return (
!isNaN(start) &&
!isNaN(end) &&
isWholeNumberBetweenInclusive(start, min, max) &&
isWholeNumberBetweenInclusive(end, min, max) &&
start <= end
);
};

const isValidList = (value: string, min: number, max: number): boolean => {
return value
.split(',')
.every((v) => isWholeNumberBetweenInclusive(Number(v), min, max));
};

const translateToCronOptions = (timeInterval: TimeInterval): CronOptions => {
if (isRate(timeInterval)) {
const { value, unit } = parseRate(timeInterval);
switch (unit) {
case 'm':
return { minute: `*/${value}` };
case 'h':
return { minute: '0', hour: `*/${value}` };
case 'day':
return { minute: '0', hour: '0', day: `*` };
case 'week':
return { minute: '0', hour: '0', weekDay: `1` };
case 'month':
return { minute: '0', hour: '0', day: '1', month: `*` };
case 'year':
return {
minute: '0',
hour: '0',
day: '1',
month: '1',
year: `*`,
};
default:
// This should never happen with strict types
throw new Error('Could not determine the schedule for the function');
}
} else {
const cronArray = timeInterval.split(' ');
const result: Record<string, string> = {
minute: cronArray[0],
hour: cronArray[1],
month: cronArray[3],
year: cronArray.length === 6 ? cronArray[5] : '*',
};

// Branching logic here is because we cannot supply both day and weekDay
if (cronArray[2] === '*') {
result.weekDay = cronArray[4];
} else {
result.day = cronArray[2];
}

return result;
}
};