Deploy on Acceptance stage failing

[williamn]

I tried to follow the Quickstart tutorial but deployment on Acceptance stage is failing, here is the output from mu svc show:

Pipeline URL:	https://console.aws.amazon.com/codesuite/codepipeline/pipelines/mu-56-qs/view?region=ap-southeast-1
+------------+----------+------------------------------------------+--------------------------------------------+---------------------+
|   STAGE    |  ACTION  |                 REVISION                 |                   STATUS                   |     LAST UPDATE     |
+------------+----------+------------------------------------------+--------------------------------------------+---------------------+
| Source     | Source   | d7141868f0293916a8f1bb19422a192bdf52a7cc | Succeeded                                  | 2018-12-21 14:17:38 |
| Build      | Artifact |                                        - | Succeeded                                  | 2018-12-21 14:18:11 |
| Build      | Image    |                                        - | Succeeded                                  | 2018-12-21 14:19:16 |
| Acceptance | Deploy   |                                        - | Failed Build terminated with state: FAILED | 2018-12-21 14:26:01 |
| Acceptance | Test     |                                        - | -                                          |                   - |
| Production | Approve  |                                        - | -                                          |                   - |
| Production | Deploy   |                                        - | -                                          |                   - |
| Production | Test     |                                        - | -                                          |                   - |
+------------+----------+------------------------------------------+--------------------------------------------+---------------------+

Deployments:
+-------------+----------+------------------+---------------------+
| ENVIRONMENT | REVISION |      STATUS      |     LAST UPDATE     |
+-------------+----------+------------------+---------------------+
| production  | d714186  | CREATE_COMPLETE  | 2018-12-21 14:13:55 |
| acceptance  | d714186  | CREATE_COMPLETE  | 2018-12-21 14:13:55 |
+-------------+----------+------------------+---------------------+

Took a look at the CloudWatch logs, here is what I found:

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: ContainerInstances (AWS::AutoScaling::LaunchConfiguration) CREATE_FAILED API: autoscaling:CreateLaunchConfiguration You are not authorized to call EC2 Describe operations. It is required to perform CreateLaunchConfiguration operation.

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: Host2HostRuleEgress (AWS::EC2::SecurityGroupEgress) CREATE_FAILED Resource creation cancelled

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: ContainerInstances (AWS::AutoScaling::LaunchConfiguration) CREATE_FAILED API: autoscaling:CreateLaunchConfiguration You are not authorized to call EC2 Describe operations. It is required to perform CreateLaunchConfiguration operation.

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: Host2HostRuleEgress (AWS::EC2::SecurityGroupEgress) CREATE_FAILED Resource creation cancelled

I ran the command mu svc show using an AdministratorAccess granted IAM user. So I guess permission should not be the issue here.

[waynerobinson]

I'm also getting this on deploy.

Not sure if anything has changed here but it seems that autoscaling:CreateLaunchConfiguration needs some specific extra EC2 permissions to execute that don't seem to be present in the deploy-cluster policy.

[waynerobinson]

I also can't seem to work out a way to override that policy as the custom CloudFormation templates attempt to append the extra policies to the existing array and I can't just modify the CloudFormation template directly as the mu env upsert command will override the mu-iam-common template each time it's run.

[waynerobinson]

By the way, in case it's relevant, I'm using version 1.5.11-develop (because of some other errors that the current stable version didn't seem to be working with).

[waynerobinson]

OK, a couple of updates on this issue.

I have managed to test out some alternative permissions by changing the role manually after mu-iam-common was created (as CloudFormation doesn't automatically check for drift).

I needed to add both ec2:* and iam:CreateServiceLinkedRole to the deploy-cluster policy and it now seems to create correctly. I'm not sure what the exact extra ec2 permissions required are as I just went a bit nuclear on it. But it may just be the "describe" operations as listed in the error above.

[waynerobinson]

When changing ec2:* to ec2:Describe* the deploy step still works. So can probably limit these permissions in this case to just ec2:Describe* and iam:CreateServiceLinkedRole.

[chris-d-edwards]

Hi, I had the same issue but using mu 1.5.10. I've changed the ARN / Resource ID opt in for Fargate/ECS, as highlighted here: issues/414 and this appears to work without the above IAM changes.

[williamn]

@chris-d-edwards I tried to opt in for the new ARN / Resource ID format but still getting the same error. I am using mu version 1.5.10

[chris-d-edwards]

Hi @williamn you need to opt out, so the boxes should be unchecked. Apologies my comment wasn't clear, if you read through the 414 issue, it states that the new format causes the issue.
Did you do this for the root account (I had done this) or for your specific user which should be a the same IAM your using in the command line.

[maxieduncan]

I'm facing the same issue. Changing the ARN / Resource ID format had no impact (I was already opted out and opting in made no difference).

I've tried adding the permissions mentioned above to the role that was being assumed when the error is reported but haven't had any luck.

[maxieduncan]

Actually I was configuring the additional roles in the wrong place, once I added them to the mu-cloudformation-common-us-east-1 Role, the deployment worked as expected.

[drummerjoe]

Just tried the quickstart and had this same issue. Shouldn't the quickstart work out of the box?