[Amplify gen2] Receive Amplify Data parameters as environment variables in a Amplify Function

[antennix]

Environment information

System:
  OS: macOS 14.5
  CPU: (8) x64 Apple M2
  Memory: 32.73 MB / 16.00 GB
  Shell: Unknown
Binaries:
  Node: 18.18.0 - ~/.nvm/versions/node/v18.18.0/bin/node
  Yarn: undefined - undefined
  npm: 9.8.1 - ~/.nvm/versions/node/v18.18.0/bin/npm
  pnpm: 8.15.4 - ~/.nvm/versions/node/v18.18.0/bin/pnpm
NPM Packages:
  @aws-amplify/backend: 1.0.4
  @aws-amplify/backend-cli: 1.1.0
  aws-amplify: 6.3.8
  aws-cdk: 2.147.3
  aws-cdk-lib: 2.147.3
  typescript: 5.5.3
AWS environment variables:
  AWS_STS_REGIONAL_ENDPOINTS = regional
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1
No CDK environment variables

Description

Is it possible for a Function to receive the Endpoint of Data,
or the name and ARN of Datasources like DynamoDB?

For example, with Storage, you can use the "access" property in the defineStorage function to grant read and write permissions to a Function.
https://docs.amplify.aws/react/build-a-backend/functions/grant-access-to-other-resources/

However, a similar approach cannot be used with defineData as it's not possible to specify properties.

It is possible to directly set values for an already created function in backend.ts using CDK, but
if you're also using defineFunction for resolvers against Data,
I think an error would occur due to circular reference, as the reference order would be resolverFunction → Data → thisFunction.

const l1CloneFunction = backend.cloneFunction.resources.lambda.node
.defaultChild as CfnFunction;

l1CloneFunction.addPropertyOverride(
　"Environment.Variables.TARGET_USER_TABLE",
　backend.data.resources.tables.User.tableName ?? "",
);

Is there a good solution?

[edwardfoyle]

Hi @antennix, you can grant a function access to the Data API by following these docs: https://docs.amplify.aws/react/build-a-backend/data/customize-authz/grant-lambda-function-access-to-api/

We are also actively investigating making the experience of configuring the data client in the context of a lambda function a better experience, so any feedback you have there is welcome!

[antennix]

@edwardfoyle
Thank you. I understand the current approach to Data (AppSync).
I'm hoping that the process of generating GraphQL code will become simpler.

On the other hand, is there an approach for when we want to access the data source (Dynamo) directly?

For example, executing a daily batch via Function
(I believe the scheduling feature will be implemented soon: #1491), and performing specific processing on DynamoDB records.

We could update records through AppSync, but I don't think we can do so without users who are executing Subscriptions noticing. I want to update silently.

[MyNameIsTakenOMG]

Hi, @antennix , I bumped into the same issue a couple of days ago when trying to pass the dynamoDB table name to my lambda hander as an env variable, and I did exact the same thing as you mentioned in the description section, which caused a circular dependencies issue.

Then, I joined in a discussion in another similar open issue #2554, where we believe this could be some sort of bug. But fortunately, other great devs came up with some workarounds, which I already tested in my case and they worked just fine. And I hope they could work for you too. Good luck!

[antennix]

@MyNameIsTakenOMG
Thank you. I think the Lambda custom resource that indirectly retrieves the resource name is a very good idea.

[ykethan]

Marking as feature request to provide Data parameters such as table name and arn as environment variables.

[rpostulart]

I have solved it by adding this to my backend.ts where Event is one of the resources I want access to

// Get the DynamoDB table name
const tableName = backend.data.resources.tables["Event"].tableName;
new cdk.CfnOutput(eventBridgeStack, "EventTableName", {
  value: tableName,
  description: "The imported name of the Event DynamoDB table",
  exportName: "EventTableName",
});

In the stack where I create the lambda function I use:

    // Define Lambda function
    this.lambdaFunction = new lambda.Function(this, "CheckEventsNextWeek", {
      runtime: lambda.Runtime.NODEJS_20_X,
      code: lambda.Code.fromAsset(
        "amplify/custom/services/checkEventsNextWeek/src"
      ),
      handler: "index.handler",
      environment: {
        DB_EVENT_TABLE_NAME: cdk.Fn.importValue("EventTableName"),
      },
    });

Inside the lambda you can the use:
process.env.DB_EVENT_TABLE_NAME

This works like charm

[antennix]

I understand the mechanism of preventing circular references by creating Lambda without using DefineFunction.
Thank you.

On the other hand,
I'd like to create Lambda using defineFunction to make integration with the Amplify system easier.

[bogris]

my workaround is to:

1. add custom keys to outpus:

be.addOutput({
  custom: {
    webhookUrl: functionUrl.url,
    stateMachineArn: notificationsStack.stateMachine.stateMachineArn,
  },
});

2. import the output in lambda and use the value

but now we have a chicken and egg issue because the outputs are artefacts of the build...

3. I pregenerate the outputs before be build:
   amplify.yml

version: 1
backend:
  phases:
    preBuild:
      commands:
        - npx ampx generate outputs --branch $AWS_BRANCH --app-id $AWS_APP_ID
    build:
      commands:
        - npm ci --cache .npm --prefer-offline
        - npx ampx pipeline-deploy --branch $AWS_BRANCH --app-id $AWS_APP_ID

some things to note...

• you need 2 deployments to get updates in the code
• you may need some //@ts-ignore in code where you refference new keys in outputs that are to be deployed, as they will not be there at first deployment..

as we do text refs we won't have any cloudformation dependencies...

idk... hope for an official solution.

I am now thinking of using @rpostulart solution now... :)

hoping for an official stand :)

[pridapablo]

@rpostulart's solution works, but I encountered a significant issue with that approach after pushing to my main branch and attempting to start up my sandbox again:

Error:

Export with name EXPORT is already exported by stack ...

I resolved it by using a different approach: casting the imported function resource in backend.ts as aws_lambda.Function to enable the addEnvironment method, as suggested by Jesse Pangburn in this thread.

Here’s a code snippet:

const customResourceStack = backend.createStack("ProcessingStack");

export const embeddingQueue = new sqs.Queue(
  customResourceStack,
  "EmbeddingQueue"
);

// Add IAM and environment variables to the function
const createEmbeddingsFunction =
  backend.createEmbeddingsForLaws.resources.lambda as aws_lambda.Function; // CAST HERE!
embeddingQueue.grantSendMessages(createEmbeddingsFunction); // IAM
createEmbeddingsFunction.addEnvironment("SQS_QUEUE_URL", embeddingQueue.queueUrl); // ENV

Note: This may cause circular dependency issues, as explained here, but this applies primarily to DynamoDB, not SQS/SNS.

Also, when accessing the environment variable in the Lambda function, do not import the env from the .amplify/generated/env folder, as it won’t work. Instead, use process.env; the variable should be available there. Although it's not type-safe, it works.

[bogris]

@pridapablo I got into a similar error but in my scenario I was trying to deploy 2 branches.

This is caused by the fact that cloudformation exports space si scoped to the account level

For me, the current workaround is to define a secret at the app level and put it in there manually... and before init a local sandbox, ... and after I deploy a new branch.
This gives you the option to have diff values per env.

My mentioned solution with importing outputs is also bad, as it only works with a previous deployment. as in: we can't do a new branch from scratch...

[zxkane]

@rpostulart How did you use the native CDK's Function managed out of defineFunction as a mutation's handler function?

// Define Lambda function
this.lambdaFunction = new lambda.Function(this, "CheckEventsNextWeek", {
  runtime: lambda.Runtime.NODEJS_20_X,
  code: lambda.Code.fromAsset(
    "amplify/custom/services/checkEventsNextWeek/src"
  ),
  handler: "index.handler",
  environment: {
    DB_EVENT_TABLE_NAME: cdk.Fn.importValue("EventTableName"),
  },
});

Inside the lambda you can the use: **process.env.DB_EVENT_TABLE_NAME**

This works like charm

[zxkane]

Without the Lambda resolver's built-in access to the table, the experience becomes much more cumbersome and time-consuming. I now manage the table manually in CDK as follows:

const externalTableStack = backend.createStack('ExternalTableStack');

const leagueTable = new Table(externalTableStack, 'League', {
  partitionKey: {
    name: 'id',
    type: AttributeType.STRING
  },
  billingMode: BillingMode.PAY_PER_REQUEST,
  removalPolicy: RemovalPolicy.DESTROY,
});

backend.data.addDynamoDbDataSource(
  "ExternalLeagueTableDataSource",
  leagueTable
);

leagueTable.grantReadWriteData(backend.leagueHandler.resources.lambda);
(backend.leagueHandler.resources.lambda.node.defaultChild as CfnFunction).addPropertyOverride('LoggingConfig', {
  LogFormat: 'JSON',
  ApplicationLogLevel: process.env.PRODUCTION ? 'WARN' : 'TRACE',
  SystemLogLevel: 'INFO',
});
(backend.leagueHandler.resources.lambda as NodejsFunction).addEnvironment('LEAGUE_TABLE_NAME', leagueTable.tableName);