Restoring files with Git LFS (#200)

If the user does not have [Git LFS](https://git-lfs.com/) installed when cloning the repo, some of the large files in the repository will not be available. This causes some workflows to break, due to reading one of these missing files. This PR adds Git LFS as one of the requirements for setting up the repo, and update the Makefile to pull files when building the cluster with local files (`make local` and `make local-worker`). --------- Co-authored-by: Renato L. de F. Cunha <[email protected]>
microsoft · Nov 25, 2024 · 98f3f71 · 98f3f71
1 parent ab1f8b2
commit 98f3f71
Show file tree

Hide file tree

Showing 4 changed files with 36 additions and 2 deletions.
diff --git a/Makefile b/Makefile
@@ -43,12 +43,15 @@ help: ## Shows this help message
 	@echo -e This is the farmvibes.ai makefile. Supported targets are:\\n
 	@grep -E -h '\s##\s' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
 
-local: cluster local-rest-api local-cache local-worker local-orchestrator local-data-ops ## Builds all images locally and deploys them into the local farmvibes.ai cluster
+local: cluster restore-git-lfs local-rest-api local-cache local-worker local-orchestrator local-data-ops ## Builds all images locally and deploys them into the local farmvibes.ai cluster
 	[ -z $(WAIT_AT_THE_END) ] || kubectl delete pods -l backend=terravibes && \
 		kubectl wait --for=condition=Available deployment --timeout=300s -l backend=terravibes
 
 revert: cluster revert-rest-api revert-cache revert-worker revert-orchestrator ## Reverts all images to the official version
 
+restore-git-lfs:
+git lfs pull || echo "git lfs was not found. Please see https://git-lfs.com/ to install it." && exit 1
+
 services-base: resources/docker/Dockerfile-services-base
 	@docker manifest inspect `$(subst FILE,$<,$(base_image_name))` || \
 		az acr login -n `$(subst FILE,$<,$(base_image_name)) | cut -d / -f 1 | sed 's|.azurecr.io||g'` || \
@@ -122,7 +125,7 @@ revert-data-ops: cluster repo-$(DATA_OPS_REPO) delete-$(DATA_OPS_DEPLOYMENT) ##
 	DEPLOYMENT=$(DATA_OPS_DEPLOYMENT) IMAGE_FULL_REFERENCE=$(CONTAINER_REGISTRY_BASE)/$(DATA_OPS_REPO):$(FARMVIBES_AI_IMAGE_TAG) $(MAKE) set-registry-image
 	DEPLOYMENT=$(DATA_OPS_DEPLOYMENT) REPLICAS=$(CURRENT_DATA_OPS_REPLICAS) make scale
 
-local-worker: cluster local-worker-repo delete-$(WORKER_DEPLOYMENT) ## Builds and deploys a local WORKER image (enabling debug)
+local-worker: cluster restore-git-lfs local-worker-repo delete-$(WORKER_DEPLOYMENT) ## Builds and deploys a local WORKER image (enabling debug)
 	DEPLOYMENT=$(WORKER_DEPLOYMENT) IMAGE_FULL_REFERENCE=$(WORKER_REPO):$(TAG) $(MAKE) -C . set-registry-image
 	DEPLOYMENT=$(WORKER_DEPLOYMENT) $(MAKE) -C . disable-frozen-modules
 	DEPLOYMENT=$(WORKER_DEPLOYMENT) REPLICAS=$(CURRENT_WORKER_REPLICAS) make scale

diff --git a/docs/source/docfiles/markdown/QUICKSTART.md b/docs/source/docfiles/markdown/QUICKSTART.md
@@ -22,6 +22,9 @@ In order to run FarmVibes.AI cluster, you need the following:
     the repository. If you already have access to the source code, then Git is
     not required.
 
+  * [Git LFS](https://git-lfs.com/) to restore some of the large files in the
+    repository (e.g., model weights).
+
   * [Docker](https://docs.docker.com/engine/install/ubuntu/). Make sure you can
     run the docker client without running `sudo` by adding your user account to
     the `docker` group (which might require a logout/login when adding oneself
@@ -60,6 +63,20 @@ bash ./resources/vm/setup_farmvibes_ai_vm.sh
 
 You might needed to restart your shell session once the script finishes.
 
+## Restore files with Git LFS
+
+In case you did not have Git LFS installed when cloning the repository, you will need to do so
+to restore the large files in the repository. Note that the last step
+["Installing software dependencies](#optional-installing-software-dependencies) already installs
+Git LFS.
+
+To restore the missing files, you can run the following command in the root of the repository:
+
+```shell
+git lfs install
+git lfs pull
+```
+
 ## Install the FarmVibes.AI cluster
 
 With python3.8+ and pip installed on your machine, please install

diff --git a/docs/source/docfiles/markdown/TROUBLESHOOTING.md b/docs/source/docfiles/markdown/TROUBLESHOOTING.md
@@ -122,6 +122,15 @@ that are currently being addressed by the development team.
 
     </details>
 
+    <details>
+    <summary> Updating cluster in the `dev` branch after pulling files with Git LFS </summary>
+
+    If you did not have Git LFS installed when cloning the repository and checking out to `dev`,
+    you will be missing some of the large files in the repository (e.g., ONNX models). Make sure
+    to install and setup Git LFS as described in the [Quickstart guide](QUICKSTART.md#restore-files-with-git-lfs).
+    You will also need to update your cluster with `make local`.
+    </details>
+
 <br>
 
 - **Composing and running workflows:**

diff --git a/resources/vm/setup_farmvibes_ai_vm.sh b/resources/vm/setup_farmvibes_ai_vm.sh
@@ -48,3 +48,8 @@ fi
 
 # Run docker without sudo
 sudo usermod -aG docker $DOCKER_USER
+
+# Run git-lfs install to restore large files
+sudo apt install git-lfs -y
+git lfs install
+git lfs pull