Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add build push docker image image in the publish ci #311

Merged
merged 22 commits into from
Nov 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6c0eac5
add build push docker image
mo-dkrz Nov 14, 2024
b19aec9
bump version
mo-dkrz Nov 14, 2024
b24192e
add test health step in publish, refactored the dockerfiles of es and os
mo-dkrz Nov 14, 2024
710b8f4
furnished readme and changelog
mo-dkrz Nov 14, 2024
795fa87
added default values and removed the unnecessary env vars from docker…
mo-dkrz Nov 16, 2024
bd5cddf
add WEB_CONCURRENCY in entrypoint
mo-dkrz Nov 17, 2024
c97d6f0
add RUN_LOCAL_OS and RUN_LOCAL_ES option to configure backend in cont…
mo-dkrz Nov 17, 2024
7226fc1
better docs
mo-dkrz Nov 17, 2024
0e70af2
check logs on produced images
mo-dkrz Nov 17, 2024
feb24d2
added better logs, and considered all different scenarios
mo-dkrz Nov 17, 2024
e8d6fc4
made README well documented
mo-dkrz Nov 17, 2024
15b0d65
added better recognition criteria for es and os
mo-dkrz Nov 17, 2024
4826f19
reflect comment on first method of installation of readme
mo-dkrz Nov 17, 2024
3e67674
minor changes in readme
mo-dkrz Nov 17, 2024
34f36a1
resolve the req change
mo-dkrz Nov 19, 2024
a2ee00b
dismantled the backends from dockerfiles and chnaged the REAME
mo-dkrz Nov 24, 2024
12af96a
prepared publish ci action
mo-dkrz Nov 24, 2024
808ee37
added latest version on image links
mo-dkrz Nov 24, 2024
f4ab110
removed env var from dockerfiles and minor change in docker-compose
mo-dkrz Nov 27, 2024
10eba9b
added forgotten build stage in docker-compose es and os services
mo-dkrz Nov 27, 2024
c033dfb
added docs in README
mo-dkrz Nov 28, 2024
562f7b3
Merge branch 'main' into add-image
jonhealy1 Nov 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 67 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
- "v*.*.*" # Triggers when a tag like 'v3.2.0' is pushed

jobs:
build-and-publish:
build-and-publish-pypi:
name: Build and Publish Packages
runs-on: ubuntu-latest

Expand Down Expand Up @@ -58,3 +58,69 @@ jobs:

# Publish to PyPI
twine upload dist/*

build-and-push-images:
name: Build and Push Docker Images
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata for Elasticsearch image
id: meta-es
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/stac-fastapi-es
tags: |
type=raw,value=latest
type=ref,event=tag

- name: Push Elasticsearch image
uses: docker/build-push-action@v6
with:
context: .
file: dockerfiles/Dockerfile.ci.es
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta-es.outputs.tags }}
labels: ${{ steps.meta-es.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Extract metadata for OpenSearch image
id: meta-os
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/stac-fastapi-os
tags: |
type=raw,value=latest
type=ref,event=tag

- name: Push OpenSearch image
uses: docker/build-push-action@v6
with:
context: .
file: dockerfiles/Dockerfile.ci.os
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta-os.outputs.tags }}
labels: ${{ steps.meta-os.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## [Unreleased]

## [v3.2.1] - 2024-11-14

### Added
- Added `dockerfiles/Dockerfile.ci.os` and `dockerfiles/Dockerfile.ci.es`, along with their respective entrypoints [#311](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/311)

### Changed
- Updated the `publish.yml` workflow to include Docker image publishing to GitHub Container Registry [#311](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/311)
- Improved the README with detailed descriptions of the new Docker images, providing guidance for images. [#311](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/311)
- Aggregation ElasticSearch `total_count` bugfix, moved aggregation text to docs. [#314](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/314)

## [v3.2.0] - 2024-10-09
Expand Down
46 changes: 41 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,17 +43,24 @@ or
pip install stac_fastapi.opensearch
```

## Build Elasticsearch API backend
### To install and run via pre-built Docker Images

jonhealy1 marked this conversation as resolved.
Show resolved Hide resolved
We provide ready-to-use Docker images through GitHub Container Registry ([ElasticSearch](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pkgs/container/stac-fastapi-es) and [OpenSearch](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pkgs/container/stac-fastapi-os) backends). You can easily pull and run these images:

jonhealy1 marked this conversation as resolved.
Show resolved Hide resolved
```shell
docker-compose up elasticsearch
docker-compose build app-elasticsearch
# For Elasticsearch backend
docker pull ghcr.io/stac-utils/stac-fastapi-es:latest

# For OpenSearch backend
docker pull ghcr.io/stac-utils/stac-fastapi-os:latest
```

## Running Elasticsearch API on localhost:8080
## Run Elasticsearch API backend on localhost:8080

You need to ensure [**Docker Compose**](https://docs.docker.com/compose/install/) or [**Podman Compose**](https://podman-desktop.io/docs/compose) installed and running on your machine. In the follwoing command instead of `docker-compose` you can use `podman-compose` as well.

```shell
docker-compose up app-elasticsearch
docker-compose up elasticsearch app-elasticsearch
```

By default, docker-compose uses Elasticsearch 8.x and OpenSearch 2.11.1.
Expand All @@ -66,6 +73,35 @@ OPENSEARCH_VERSION=2.11.0
```
The most recent Elasticsearch 7.x versions should also work. See the [opensearch-py docs](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) for compatibility information.

#### **Configuration reference keys:**

You can customize additional settings in your `.env` file:
###### Key variables to configure:

| Variable | Description | Default | Required |
|------------------------------|--------------------------------------------------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------|
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great

| `ES_HOST` | Hostname for external Elasticsearch/OpenSearch. | `localhost` | Optional |
| `ES_PORT` | Port for Elasticsearch/OpenSearch. | `9200` (ES) / `9202` (OS)| Optional |
| `ES_USE_SSL` | Use SSL for connecting to Elasticsearch/OpenSearch. | `false` | Optional |
| `ES_VERIFY_CERTS` | Verify SSL certificates when connecting. | `false` | Optional |
| `STAC_FASTAPI_TITLE` | Title of the API in the documentation. | `stac-fastapi-elasticsearch` or `stac-fastapi-opensearch` | Optional |
| `STAC_FASTAPI_DESCRIPTION` | Description of the API in the documentation. | N/A | Optional |
| `STAC_FASTAPI_VERSION` | API version. | `2.1` | Optional |
| `APP_HOST` | Server bind address. | `0.0.0.0` | Optional |
| `APP_PORT` | Server port. | `8080` | Optional |
| `ENVIRONMENT` | Runtime environment. | `local` | Optional |
| `WEB_CONCURRENCY` | Number of worker processes. | `10` | Optional |
| `RELOAD` | Enable auto-reload for development. | `true` | Optional |
| `STAC_FASTAPI_RATE_LIMIT` | API rate limit per client. | `200/minute` | Optional |
| `BACKEND` | Tests-related variable | `elasticsearch` or `opensearch` based on the backend | Optional |
| `ELASTICSEARCH_VERSION` | ElasticSearch version | `7.17.1` | Optional |
| `OPENSEARCH_VERSION` | OpenSearch version | `2.11.0` | Optional |

> [!NOTE]
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, and `ES_VERIFY_CERTS` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.

## Interacting with the API

To create a new Collection:

```shell
Expand Down
19 changes: 19 additions & 0 deletions dockerfiles/Dockerfile.ci.es
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && \
apt-get install -y --no-install-recommends \
gcc \
curl \
&& apt-get clean && \
rm -rf /var/lib/apt/lists/*

COPY . /app/

RUN pip3 install --no-cache-dir -e ./stac_fastapi/core && \
pip3 install --no-cache-dir ./stac_fastapi/elasticsearch[server]

USER root

CMD ["python", "-m", "stac_fastapi.elasticsearch.app"]
19 changes: 19 additions & 0 deletions dockerfiles/Dockerfile.ci.os
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && \
apt-get install -y --no-install-recommends \
gcc \
curl \
&& apt-get clean && \
rm -rf /var/lib/apt/lists/*

COPY . /app/

RUN pip3 install --no-cache-dir -e ./stac_fastapi/core && \
pip3 install --no-cache-dir ./stac_fastapi/opensearch[server]

USER root

CMD ["python", "-m", "stac_fastapi.opensearch.app"]
2 changes: 1 addition & 1 deletion stac_fastapi/core/stac_fastapi/core/version.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
"""library version."""
__version__ = "3.2.0"
__version__ = "3.2.1"
2 changes: 1 addition & 1 deletion stac_fastapi/elasticsearch/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
desc = f.read()

install_requires = [
"stac-fastapi.core==3.2.0",
"stac-fastapi.core==3.2.1",
"elasticsearch[async]==8.11.0",
"elasticsearch-dsl==8.11.0",
"uvicorn",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
"""library version."""
__version__ = "3.2.0"
__version__ = "3.2.1"
2 changes: 1 addition & 1 deletion stac_fastapi/opensearch/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
desc = f.read()

install_requires = [
"stac-fastapi.core==3.2.0",
"stac-fastapi.core==3.2.1",
"opensearch-py==2.4.2",
"opensearch-py[async]==2.4.2",
"uvicorn",
Expand Down
2 changes: 1 addition & 1 deletion stac_fastapi/opensearch/stac_fastapi/opensearch/version.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
"""library version."""
__version__ = "3.2.0"
__version__ = "3.2.1"
Loading