Support deploying Airbyte on ECS

[sherifnada]

Tell us about the problem you're trying to solve

i would like to deploy Airbyte on ECS. This is known to be a difficult problem because Airbyte requires the ability to dynamically launch new docker containers (connectors are packaged in containers) but ECS does not offer this capability easily. See here for more information.

Nonetheless many companies solely use ECS as their deployment medium and find it to be a deal breaker if Airbyte does not support it.

Describe the solution you’d like

The ability to deploy on ECS

[stadskle]

Hi,

We are one of the companies that would prefer to deploy to ECS and Fargate. Could someone please explain "but ECS does not offer this capability easily" for me?

I do not know the inner workings of Airbyte, so there might be some requirements here I am not getting. However, if the need is to spin up new containers to execute jobs, that should be very possible. ECS support running adhoc tasks, and as long as the container has the appropriate IAM rights, I do not see why this cannot be done from an ECS service?

https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html

When starting an adhoc task you can pass custom commands and env variables.

[Sousaplex]

Just wanted to share that we would also like to deploy to ECS.

[rorymalcolm]

Hi,

We are one of the companies that would prefer to deploy to ECS and Fargate. Could someone please explain "but ECS does not offer this capability easily" for me?

I do not know the inner workings of Airbyte, so there might be some requirements here I am not getting. However, if the need is to spin up new containers to execute jobs, that should be very possible. ECS support running adhoc tasks, and as long as the container has the appropriate IAM rights, I do not see why this cannot be done from an ECS service?

https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html

When starting an adhoc task you can pass custom commands and env variables.

The problem relates to the fact that ECS Fargate does not provide permissions for "Docker in Docker", you can build Docker containers inside of Fargate using Kaniko, but you can't run a Docker container inside of a Fargate container: aws/containers-roadmap#1356

[ndavies-om1]

Hi,
We are one of the companies that would prefer to deploy to ECS and Fargate. Could someone please explain "but ECS does not offer this capability easily" for me?
I do not know the inner workings of Airbyte, so there might be some requirements here I am not getting. However, if the need is to spin up new containers to execute jobs, that should be very possible. ECS support running adhoc tasks, and as long as the container has the appropriate IAM rights, I do not see why this cannot be done from an ECS service?
https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html
When starting an adhoc task you can pass custom commands and env variables.

The problem relates to the fact that ECS Fargate does not provide permissions for "Docker in Docker", you can build Docker containers inside of Fargate using Kaniko, but you can't run a Docker container inside of a Fargate container: aws/containers-roadmap#1356

You would not run a container inside another container, you would have airbyte use the ECS API to run tasks

[stadskle]

Hi,
We are one of the companies that would prefer to deploy to ECS and Fargate. Could someone please explain "but ECS does not offer this capability easily" for me?
I do not know the inner workings of Airbyte, so there might be some requirements here I am not getting. However, if the need is to spin up new containers to execute jobs, that should be very possible. ECS support running adhoc tasks, and as long as the container has the appropriate IAM rights, I do not see why this cannot be done from an ECS service?
https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html
When starting an adhoc task you can pass custom commands and env variables.

The problem relates to the fact that ECS Fargate does not provide permissions for "Docker in Docker", you can build Docker containers inside of Fargate using Kaniko, but you can't run a Docker container inside of a Fargate container: aws/containers-roadmap#1356

You would not run a container inside another container, you would have airbyte use the ECS API to run tasks

yes, exactly my point

[parth114]

I am also considering deploying airbyte on ECS. Any release plans for this ?

[gilbahat]

+1 on ECS deployments. I would wager the easiest way to do this would be to designate SQS queues that trigger the respective jobs. This way, no code to fire up tasks would need to be written. other common alternatives are rabbitmq or redis.

[alexandrecaze]

+1 here, is this issue in the roadmap or is it too complicated ?

[robert-put]

@sherifnada Is there any way to contribute to resolving this issue?

[sherifnada]

will defer to @supertopher since his team now manages infra related concerns such as this one

[espoirMur]

I am thinking about starting some work on this, I will keep on sharing the solution.

[dacreify]

In principal I think Airbyte can run on ECS today, just not ECS Fargate. AFAIK you can volume mount the docker socket when using EC2 instances in your ECS cluster.

[mstapfner]

Are there any updates yet on this topic? Would be interested as well, additionally in running this via GCP CloudRun.

[corrjo]

In principal I think Airbyte can run on ECS today, just not ECS Fargate. AFAIK you can volume mount the docker socket when using EC2 instances in your ECS cluster.

Has anyone tried this?

[geronimo-iia]

I ever done this kind of docker mount with ECS on EC2. It work as soon you use the same container version.
The limitation is that you couldn't scale out the cluster because local container aren't launched and registered into ECS service. It more or less the same as if you using an ec2.

A better approach, could be using aws ecs api to launch dedicated task. This implies to register task definition, cluster, and service before starting airbyte.

[selfscrum]

Hello I would like to have this option too. For us it was a dealbreaker, we are now with AWS Glue and only semi-happy.
Dagster seems to have a good concept how the support of different docker spawns can be built. I envision somethin similar here.
The website is saying "coming soon" for quite a while now. Any new on this?