Periodic ResourceNotReady: exceeded wait attempts errors

[jroblak]

For some reason, one of our environments seems to take a little longer than average to spin up the ECS Task and associated infra. Because of this, ecs-run-task exits with ResourceNotReady: exceeded wait attempts; however, the task still runs to completion.

Is there any way to either increase the wait time / number of attempts, and/or abort the task if this error arises?

[sherzberg]

I'd also like some way to do a retry. We have been getting an error in the ECS console with message Timeout waiting for network interface provisioning to complete and the panic from ecs-run-task:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x895b85]

goroutine 1 [running]:
github.com/buildkite/ecs-run-task/runner.writeContainerFinishedMessage(0xb226e0, 0xc00009c010, 0xc0002161c0, 0xc0001bd9e0, 0xc0001b82a0, 0x3a, 0x0)
	/Users/lachlan/go/src/github.com/buildkite/ecs-run-task/runner/runner.go:261 +0x155
github.com/buildkite/ecs-run-task/runner.(*Runner).Run(0xc0000ac0e0, 0xb226e0, 0xc00009c010, 0x0, 0x1)
	/Users/lachlan/go/src/github.com/buildkite/ecs-run-task/runner/runner.go:219 +0x12dd
main.main.func1(0xc0000e8580, 0x0, 0x0)
	/Users/lachlan/go/src/github.com/buildkite/ecs-run-task/main.go:115 +0x625
github.com/urfave/cli.HandleAction(0x93db80, 0xa17268, 0xc0000e8580, 0x0, 0x0)
	/Users/lachlan/go/pkg/mod/github.com/urfave/[email protected]/app.go:490 +0xc8
github.com/urfave/cli.(*App).Run(0xc00014cea0, 0xc0000ac000, 0xe, 0xe, 0x0, 0x0)
	/Users/lachlan/go/pkg/mod/github.com/urfave/[email protected]/app.go:264 +0x57c
main.main()
	/Users/lachlan/go/src/github.com/buildkite/ecs-run-task/main.go:125 +0x8c3

We can easily just retry outside of ecs-run-task, but maybe some new cli flags to control retry and wait timeouts would work?

[Eli-Goldberg]

Happens to us as well

[Eli-Goldberg]

Could be a fix:
#26 (comment)

[Eli-Goldberg]

I've opened a pr #35.
It simply retries until the task is done.
Nothing fancy like maximum retries and all, anyone is welcome to modify the solution - but this fixes the panic :)