Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(glue): support AWS Glue 5.0 #32467

Merged
merged 4 commits into from
Dec 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions packages/@aws-cdk/aws-glue-alpha/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ An ETL job processes data in batches using Apache Spark.
declare const bucket: s3.Bucket;
new glue.Job(this, 'ScalaSparkEtlJob', {
executable: glue.JobExecutable.scalaEtl({
glueVersion: glue.GlueVersion.V4_0,
glueVersion: glue.GlueVersion.V5_0,
script: glue.Code.fromBucket(bucket, 'src/com/example/HelloWorld.scala'),
className: 'com.example.HelloWorld',
extraJars: [glue.Code.fromBucket(bucket, 'jars/HelloWorld.jar')],
Expand All @@ -58,7 +58,7 @@ A Streaming job is similar to an ETL job, except that it performs ETL on data st
```ts
new glue.Job(this, 'PythonSparkStreamingJob', {
executable: glue.JobExecutable.pythonStreaming({
glueVersion: glue.GlueVersion.V4_0,
glueVersion: glue.GlueVersion.V5_0,
pythonVersion: glue.PythonVersion.THREE,
script: glue.Code.fromAsset(path.join(__dirname, 'job-script', 'hello_world.py')),
}),
Expand Down Expand Up @@ -94,7 +94,7 @@ These jobs run in a Ray environment managed by AWS Glue.
```ts
new glue.Job(this, 'RayJob', {
executable: glue.JobExecutable.pythonRay({
glueVersion: glue.GlueVersion.V4_0,
glueVersion: glue.GlueVersion.V5_0,
pythonVersion: glue.PythonVersion.THREE_NINE,
runtime: glue.Runtime.RAY_TWO_FOUR,
script: glue.Code.fromAsset(path.join(__dirname, 'job-script', 'hello_world.py')),
Expand Down Expand Up @@ -137,7 +137,7 @@ Enable job run queuing by setting the `jobRunQueuingEnabled` property to `true`.
new glue.Job(this, 'EnableRunQueuing', {
jobName: 'EtlJobWithRunQueuing',
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V4_0,
glueVersion: glue.GlueVersion.V5_0,
pythonVersion: glue.PythonVersion.THREE,
script: glue.Code.fromAsset(path.join(__dirname, 'job-script', 'hello_world.py')),
}),
Expand Down Expand Up @@ -488,7 +488,7 @@ new glue.S3Table(this, 'MyTable', {
declare const myDatabase: glue.Database;
// KMS key is created automatically
new glue.S3Table(this, 'MyTable', {
encryption: glue.TableEncryption.CLIENT_SIDE_KMS,
encryption: glue.TableEncryption.CLIENT_SIDE_KMS,
// ...
database: myDatabase,
columns: [{
Expand Down Expand Up @@ -546,7 +546,7 @@ new glue.S3Table(this, 'MyTable', {
// ...
database: myDatabase,
dataFormat: glue.DataFormat.JSON,
});
});
```

### Primitives
Expand Down
8 changes: 8 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/job-executable.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,19 @@ import { Code } from './code';
export class GlueVersion {
/**
* Glue version using Spark 2.2.1 and Python 2.7
* @deprecated Reached end of support
*/
public static readonly V0_9 = new GlueVersion('0.9');

/**
* Glue version using Spark 2.4.3, Python 2.7 and Python 3.6
* @deprecated Reached end of support
*/
public static readonly V1_0 = new GlueVersion('1.0');

/**
* Glue version using Spark 2.4.3 and Python 3.7
* @deprecated Reached end of support
*/
public static readonly V2_0 = new GlueVersion('2.0');

Expand All @@ -34,6 +37,11 @@ export class GlueVersion {
*/
public static readonly V4_0 = new GlueVersion('4.0');

/**
* Glue version using Spark 3.5.2 and Python 3.11
*/
public static readonly V5_0 = new GlueVersion('5.0');

/**
* Custom Glue version
* @param version custom version
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading