You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Let's say I have a simple Python app (as a github repo) which loads some data and processes it. I have the data stored in an S3 bucket which I pull through DVC (dvc pull). Everything works fine in a local environment on machine.
Now, I would like to use DVC in production for a dockerized application. How can I make use of DVC within the Dockerfile? I couldn't find any examples in docs so it would be great if someone can point me the right direction.
Locally this setup works fine. I have added the data to an S3 bucket through dvc push. DVC pull works fine too. Now how can I use dvc pull within a dockerfile such that it pulls this data in dockerised container?
I can imagine adding the following to my Dockerfile.
1) Make the AWS security key/secret available as ENV variables
2)pip install dvc[s3]
3) dvc pull
However, dvc needs to have access to its configuration files in a container to connect to the right s3 bucket and get the relevant data. How can I achieve this within a dockerfile? The dockerized application will run in Fargate cluster in AWS.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Let's say I have a simple Python app (as a github repo) which loads some data and processes it. I have the data stored in an S3 bucket which I pull through DVC (dvc pull). Everything works fine in a local environment on machine.
Now, I would like to use DVC in production for a dockerized application. How can I make use of DVC within the Dockerfile? I couldn't find any examples in docs so it would be great if someone can point me the right direction.
Here is my dockerfile.
Here is the repo structure:
Locally this setup works fine. I have added the data to an S3 bucket through dvc push. DVC pull works fine too. Now how can I use dvc pull within a dockerfile such that it pulls this data in dockerised container?
I can imagine adding the following to my Dockerfile.
However, dvc needs to have access to its configuration files in a container to connect to the right s3 bucket and get the relevant data. How can I achieve this within a dockerfile? The dockerized application will run in Fargate cluster in AWS.
Beta Was this translation helpful? Give feedback.
All reactions