Backstage pattern timeout on load balancer creation

[shubhamk-cdtn]

Describe the bug
When I try to deploy backstage pattern, the eks-loadbalancer resource creation is timing out with errors like

Received response status [FAILED] from custom resource. Message returned: TimeoutError: {"state":"TIMEOUT","reason":"Waiter has timed out"} at checkExceptions (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/util-waiter/dist-cjs/index.js:59:26) at waitUntilFunctionActiveV2 (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/index.js:5826:49) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async defaultInvokeFunction (/var/task/outbound.js:1:1024) at async invokeUserFunction (/var/task/framework.js:1:2287) at async onEvent (/var/task/framework.js:1:369) at async Runtime.handler (/var/task/cfn-response.js:1:1676) (RequestId: 16ad0de3-1e44-438b-814c-ac208323c802)

On searching for the issue in cloudwatch logs, I get this

2024-07-26T16:49:08.745Z	885eae61-9575-48f9-84d1-c2058a66a94f	INFO	[provider-framework] executing user function arn:aws:lambda:ap-south-1:323155100510:function:backstage-blueprint-awscdkawseksKu-Handler886CB40B-TCMBwnyiTMag with payload {
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:ap-south-1:323155100510:function:backstage-blueprint-awscd-ProviderframeworkonEvent-8xHpBQWOgq7v",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:ap-south-1:323155100510:stack/backstage-blueprint/faf29dd0-4b6c-11ef-83b5-027ab549ec09",
    "RequestId": "16ad0de3-1e44-438b-814c-ac208323c802",
    "LogicalResourceId": "backstageblueprintawsloadbalancercontrollermanifestawsloadbalancercontrollerServiceAccountResource42E6EEE9",
    "ResourceType": "Custom::AWSCDK-EKS-KubernetesResource",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:ap-south-1:323155100510:function:backstage-blueprint-awscd-ProviderframeworkonEvent-8xHpBQWOgq7v",
        "PruneLabel": "aws.cdk.eks/prune-c8e52c43061de408d9284c707a9eaa0c6f9cfaef0f",
        "ClusterName": "backstage-blueprint",
        "Manifest": "[{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\",\"labels\":{\"aws.cdk.eks/prune-c8e52c43061de408d9284c707a9eaa0c6f9cfaef0f\":\"\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"},\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::323155100510:role/backstage-blueprint-backstageblueprintawsloadbalanc-bYL4sAgx0WEu\"}}}]",
        "RoleArn": "arn:aws:iam::323155100510:role/backstage-blueprint-backstageblueprintCreationRoleE-sYYVkymowvO8"
    }
}

2024-07-26T16:52:23.572Z	885eae61-9575-48f9-84d1-c2058a66a94f	INFO	[provider-framework] CREATE failed, responding with a marker physical resource id so that the subsequent DELETE will be ignored

2024-07-26T16:52:23.603Z	885eae61-9575-48f9-84d1-c2058a66a94f	INFO	[provider-framework] submit response to cloudformation https://cloudformation-custom-resource-response-apsouth1.s3.ap-south-1.amazonaws.com//arn%3Aaws%3Acloudformation%3Aap-south-1%3A323155100510%3Astack/backstage-blueprint/faf29dd0-4b6c-11ef-83b5-027ab549ec09%7CbackstageblueprintawsloadbalancercontrollermanifestawsloadbalancercontrollerServiceAccountResource42E6EEE9%7C16ad0de3-1e44-438b-814c-ac208323c802?*** {
    "Status": "FAILED",
    "Reason": "TimeoutError: {\"state\":\"TIMEOUT\",\"reason\":\"Waiter has timed out\"}\n    at checkExceptions (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/util-waiter/dist-cjs/index.js:59:26)\n    at waitUntilFunctionActiveV2 (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/index.js:5826:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async defaultInvokeFunction (/var/task/outbound.js:1:1024)\n    at async invokeUserFunction (/var/task/framework.js:1:2287)\n    at async onEvent (/var/task/framework.js:1:369)\n    at async Runtime.handler (/var/task/cfn-response.js:1:1676)",
    "StackId": "arn:aws:cloudformation:ap-south-1:323155100510:stack/backstage-blueprint/faf29dd0-4b6c-11ef-83b5-027ab549ec09",
    "RequestId": "16ad0de3-1e44-438b-814c-ac208323c802",
    "PhysicalResourceId": "AWSCDK::CustomResourceProviderFramework::CREATE_FAILED",
    "LogicalResourceId": "backstageblueprintawsloadbalancercontrollermanifestawsloadbalancercontrollerServiceAccountResource42E6EEE9"
}

Lambda is timing out I think

To Reproduce

make pattern backstage deploy

cc: @freschri

[freschri]

@shubhamk-cdtn thanks for reaching out. how many times did you try? can you clean all and try again? thanks

[shubhamk-cdtn]

@freschri multiple times actually. Tried on fresh aws account without any success unfortunately

[github-actions]

This issue has been automatically marked as stale because it has been open 60 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

Issue closed due to inactivity.