feat: Add docker bake builder (#528)

Introduce Docker Bake for building and pushing images with updated GitHub Actions workflows and Dockerfiles.

  - **GitHub Actions Workflows**:
    - Add `.github/workflows/dev-push-to-hub.yml` and `.github/workflows/main-push-to-hub.yml` for building and pushing images on `dev` and `main` branch merges.
    - Remove `push-to-hub.yml` workflow.
  - **Docker Build System**:
    - Introduce `docker-bake.hcl` for defining Docker build targets and configurations.
    - Update Dockerfiles in `agents-api`, `agents-api.worker`, `agents-api.migration` to use Docker Bake.
  - **Documentation**:
    - Update `CONTRIBUTING.md` with instructions for building Docker images using Docker Bake.
  - **Miscellaneous**:
    - Add submodules for `sdks/node-sdk` and `sdks/python-sdk`.

.github/workflows/dev-push-to-hub.yml

@@ -0,0 +1,46 @@
+name: Build and push images to docker hub on merge to dev
+run-name: ${{ github.actor }} is building and pushing images to docker hub
+
+on:
+  push:
+    branches:
+      - "dev"
+
+jobs:
+  Bake-Push-Images:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set branch name
+        id: variables
+        run: |
+          echo "branch_name=$(echo ${GITHUB_REF#refs/heads/} | tr '/' '-')" >> $GITHUB_OUTPUT
+          echo "git_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: julepai
+          password: "${{ secrets.DOCKER_HUB_PASSWORD }}"
+
+      - name: Build and push agent images
+        uses: docker/bake-action@v3
+        with:
+          files: |
+            ./docker-bake.hcl
+          targets: agents-api
+          push: true
+          set: |
+            *.cache-from=type=gha
+            *.cache-to=type=gha,mode=max
+            *.args.TAG=${{ steps.variables.outputs.branch_name }}
+            *.args.GIT_SHA=${{ steps.variables.outputs.git_sha }}
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/dev' }}

.github/workflows/main-push-to-hub.yml

@@ -0,0 +1,45 @@
+name: Build and push images to docker hub on merge to main
+run-name: ${{ github.actor }} is building and pushing images to docker hub
+
+on:
+  push:
+    branches:
+      - "main"
+
+jobs:
+  Bake-Push-Images:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set git sha
+        id: variables
+        run: |
+          echo "git_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: julepai
+          password: "${{ secrets.DOCKER_HUB_PASSWORD }}"
+
+      - name: Build and push agent images
+        uses: docker/bake-action@v3
+        with:
+          files: |
+            ./docker-bake.hcl
+          targets: agents-api
+          push: true
+          set: |
+            *.cache-from=type=gha
+            *.cache-to=type=gha,mode=max
+            *.args.TAG=latest
+            *.args.GIT_SHA=${{ steps.variables.outputs.git_sha }}
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

CONTRIBUTING.md

@@ -30,4 +30,44 @@ Improvements to documentation are always appreciated! If you see areas that coul
 
 We'd love to hear your feedback and ideas for the project! Feel free to submit an issue or contact the maintainers directly to share your thoughts. Your input is very valuable in shaping the future direction of the project.
 
+## Building Docker Images with Buildx Bake
+
+We use Docker Buildx Bake to build our Docker images. This allows us to build multiple images concurrently and efficiently. Follow these steps to build the Docker images:
+
+1. Ensure you have Docker and Docker Buildx installed on your system.
+
+2. Navigate to the root directory of the project where the `docker-bake.hcl` file is located.
+
+3. To build all services, run:
+   ```
+   docker buildx bake --file docker-bake.hcl
+   ```
+
+4. To build a specific service, use:
+   ```
+   docker buildx bake --file docker-bake.hcl <service-name>
+   ```
+   Replace `<service-name>` with one of the following:
+   - agents-api
+   - agents-api-worker
+   - cozo-migrate
+   - memory-store
+   - integrations
+   - gateway
+   - embedding-service-cpu
+   - embedding-service-gpu
+
+5. To set a custom tag for the images, use:
+   ```
+   docker buildx bake --file docker-bake.hcl --set *.tags=myorg/myimage:v1.0
+   ```
+   Replace `myorg/myimage:v1.0` with your desired image name and tag.
+
+6. By default, the images are built with the "latest" tag. To specify a different tag, you can set the TAG variable:
+   ```
+   docker buildx bake --file docker-bake.hcl --set TAG=v1.2.3
+   ```
+
+Note: The `docker-bake.hcl` file defines the build contexts, Dockerfiles, and tags for each service. If you need to modify the build process for a specific service, update the corresponding target in the `docker-bake.hcl` file.
+
 Thank you for your interest in contributing to this project!

agents-api/Dockerfile

@@ -1,11 +1,14 @@
+# syntax=docker/dockerfile:1
+# check=error=true
+
 FROM python:3.12-slim
 
-ENV PYTHONUNBUFFERED True
+ENV PYTHONUNBUFFERED=1
 ENV POETRY_CACHE_DIR=/tmp/poetry_cache
 
 WORKDIR /app
 
-RUN pip install 'poetry<=1.9.0' \
+RUN pip install 'poetry>=1.8.0,<1.9.0' \
     && poetry config virtualenvs.create false
 
 COPY pyproject.toml poetry.lock ./
@@ -14,6 +17,4 @@ RUN poetry install --no-dev --no-root
 
 COPY . ./
 
-RUN poetry install --no-dev
-
 ENTRYPOINT ["python", "-m", "agents_api.web", "--host", "0.0.0.0", "--port", "8080"]

agents-api/Dockerfile.migration

@@ -1,6 +1,9 @@
+# syntax=docker/dockerfile:1
+# check=error=true
+
 FROM python:3.11-slim
 
-ENV PYTHONUNBUFFERED True
+ENV PYTHONUNBUFFERED=1
 ENV POETRY_CACHE_DIR=/tmp/poetry_cache
 
 WORKDIR /app
@@ -9,8 +12,11 @@ RUN pip install --no-cache-dir --upgrade cozo-migrate
 
 COPY . ./
 ENV COZO_HOST="http://cozo:9070"
-ENV COZO_AUTH_TOKEN="myauthkey"
 
+# Expected environment variables:
+# COZO_AUTH_TOKEN="myauthkey"
+
+SHELL ["/bin/bash", "-c"]
 ENTRYPOINT \
     cozo-migrate -e http -h $COZO_HOST --auth $COZO_AUTH_TOKEN init \
     ; cozo-migrate -e http -h $COZO_HOST --auth $COZO_AUTH_TOKEN -d ./migrations apply -ay

agents-api/Dockerfile.worker

@@ -1,11 +1,14 @@
+# syntax=docker/dockerfile:1
+# check=error=true
+
 FROM python:3.12-slim
 
-ENV PYTHONUNBUFFERED True
+ENV PYTHONUNBUFFERED=1
 ENV POETRY_CACHE_DIR=/tmp/poetry_cache
 
 WORKDIR /app
 
-RUN pip install 'poetry<=1.9.0' \
+RUN pip install 'poetry>=1.8.0,<1.9.0' \
     && poetry config virtualenvs.create false
 
 COPY pyproject.toml poetry.lock ./