Skip to content

Commit

Permalink
Change dev port, use JSON for agent create, update README (#101)
Browse files Browse the repository at this point in the history 
* Change dev port to 8000

* Use JSON for agent POST, lowercase messages

* Update port in docker configs

* Listen on 0.0.0.0 in Dockerfile

* Update README and compose file

* Further README tweaks

* Fix heading

* Add links to frontend
  • Loading branch information
@bsquizz
bsquizz authored Jan 15, 2025
1 parent 0cb770d commit 7fc696e
Show file tree
Hide file tree
Showing 6 changed files with 226 additions and 121 deletions.
6 changes: 3 additions & 3 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ FROM registry.access.redhat.com/ubi9/ubi-minimal:latest

ENV LC_ALL=C.utf8
ENV LANG=C.utf8
ENV PYTHONUNBUFFERED=1
ENV PYTHONUNBUFFERED=1
ENV PYTHONIOENCODING=UTF-8
ENV PIP_NO_CACHE_DIR=1

Expand Down Expand Up @@ -47,8 +47,8 @@ ADD connectors ./connectors
ADD resources ./resources
ADD app.py .

EXPOSE 5000
EXPOSE 8000

ENV PATH="$APP_ROOT/.venv/bin:$PATH"
CMD ["flask", "run", "--host=0.0.0.0"]
CMD ["flask", "run", "--host=0.0.0.0", "--port=8000"]

211 changes: 139 additions & 72 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,112 +1,156 @@
# 🍊 tangerine (backend)

# 🍊 tangerine (backend) <!-- omit from toc -->

tangerine is a slim and light-weight RAG (Retieval Augmented Generated) system used to create and manage chat bot agents.

Each agent is intended to answer questions related to a set of documents known as a knowledge base (KB).

It relies on 4 key components:
- [Overview](#overview)
- [Development Envionment Setup](#development-envionment-setup)
- [With Docker Compose (not supported with Mac)](#with-docker-compose-not-supported-with-mac)
- [Using huggingface text-embeddings-inference server to host embedding model (deprecated)](#using-huggingface-text-embeddings-inference-server-to-host-embedding-model-deprecated)
- [Without Docker Compose (supported with Mac)](#without-docker-compose-supported-with-mac)
- [Syncrhonizing Documents from S3](#syncrhonizing-documents-from-s3)
- [Run Tangerine Frontend Locally](#run-tangerine-frontend-locally)
- [Available API Paths](#available-api-paths)

* A vector database (PostgresQL with the pgvector extension)
* A large language model (LLM) hosted on any OpenAI-compatible API service.
* An embedding model hosted on any OpenAI-compatible API service.
* (optional) An S3 bucket that you wish to sync documentation from.
## Overview

The backend service manages:
* Creating/updating/deleting chat bot agents
* Uploading documents to be used as context to assist the agents in answering questions
* Document ingestion including cleanup/conversion, chunking, and embedding into the vector database.
* Document chunk retrieval from the vector database.
* Interfacing with the LLM to prompt it and stream responses
* (optional) Interfacing with S3 to provide continuous document sync.
tangerine relies on 4 key components:

tangerine will work with any deployed instance of PostgresQL+pgvector and can be configured to use any OpenAI-compliant API service that is hosting a large language model or embedding model.
- A vector database
- (PostgresQL with the pgvector extension)
- A large language model (LLM)
- This can be hosted by any OpenAI-compatible API service. Locally, you can use ollama
- An embedding model
- This can be hosted on any OpenAI-compatible API service. Locally, you can use ollama
- (optional) An S3 bucket that you wish to sync documentation from.

This repository provides Open Shift templates for all infrastructure (except for the model hosting service) as well as a docker compose file that allows you to spin it up locally and use [ollama](https://ollama.com/).
The backend service manages:

The accompanying frontend service is [tangerine-frontend](https://github.com/RedHatInsights/tangerine-frontend) and a related plugin for [Red Hat Developer Hub](https://developers.redhat.com/rhdh/overview) can be found [here](https://github.com/RedHatInsights/backstage-plugin-ai-search-frontend)
- Management of chat bot "agents"
- Document ingestion
- Upload via the API, or sync via an s3 bucket
- Text cleanup/conversion
- Chunking and embedding into the vector database.
- Querying the vector database.
- Interfacing with the LLM to prompt it and stream responses

This project is currently used by Red Hat's Hybrid Cloud Management Engineering Productivity Team. It was born out of a hack-a-thon and is still a work in progress. You will find some areas of code well developed while others are in need of attention and some tweaks to make it production-ready are needed (with that said, the project *is* currently in good enough shape to provide a working chat bot system).
tangerine will work with any deployed instance of PostgresQL+pgvector and can be configured to use any OpenAI-compliant API service that is hosting a large language model or embedding model. In addition, the model you wish to use and the prompts to instruct them are fully customizable.

## Getting started
This repository provides Open Shift templates for all infrastructure (except for the model hosting service) as well as a docker compose file that allows you to spin it up locally.

The project can be deployed to a local development environment using ollama to host the LLM and huggingface's [text-embeddings-inference](https://github.com/huggingface/text-embeddings-inference) server to host the embedding model.
The accompanying frontend service is [tangerine-frontend](https://github.com/RedHatInsights/tangerine-frontend) and a related plugin for [Red Hat Developer Hub](https://developers.redhat.com/rhdh/overview) can be found [here](https://github.com/RedHatInsights/backstage-plugin-ai-search-frontend)

This project is currently used by Red Hat's Hybrid Cloud Management Engineering Productivity Team. It was born out of a hack-a-thon and is still a work in progress. You will find some areas of code well developed while others are in need of attention and some tweaks to make it production-ready are needed (with that said, the project *is* currently in good enough shape to provide a working chat bot system).

### Local Environment Setup for Linux / Intel Macs
## Development Envionment Setup

You may require further tweaks to properly make use of your GPU. Refer to the [ollama docker image documentation](https://hub.docker.com/r/ollama/ollama).
A development environment can be set up with or without docker compose. In both cases, Ollama may be able to make use of your NVIDIA or AMD GPU (see more information about GPU support [here](https://github.com/ollama/ollama/blob/main/docs/gpu.md). On a Mac, Ollama must be run as a standalone application outside of Docker containers since Docker Desktop does not support GPUs.

1. Make sure [git-lfs](https://git-lfs.com/) is installed:
### With Docker Compose (not supported with Mac)

* Fedora: `sudo dnf install git-lfs`
* MacOS: `brew install git-lfs`
The docker compose file offers an easy way to spin up all components. [ollama](https://ollama.com) is used to host the LLM and embedding model. For utilization of your GPU, refer to the comments in the compose file to see which configurations to uncomment on the 'ollama' container.

Then, activate it globally with:
1. Create the directory which will house the local environment data:

```text
git lfs install
mkdir data
```
2. Create the directory which will house the local environment data:
2. Invoke docker compose (postgres data will persist in `data/postgres`):
```text
mkdir data
docker compose up --build
```
3. Create a directory to house the embedding model and download the `snowflake-arctic-embed-m-long` model:
3. Pull the mistral LLM and nomic embedding model (data will persist in `data/ollama`):
```text
mkdir data/embeddings
git clone https://huggingface.co/Snowflake/snowflake-arctic-embed-m-long \
data/embeddings/snowflake-arctic-embed-m-long
docker exec tangerine-ollama ollama pull mistral
docker exec tangerine-ollama ollama pull nomic-embed-text
```
4. Invoke docker compose (postgres data will persist in `data/postgres`):
4. Access the API on port `8000`
```sh
curl -XGET 127.0.0.1:8000/api/agents
{
"data": []
}
```

5. (optional) Run the [tangerine-frontend](https://github.com/RedHatInsights/tangerine-frontend):

```text
git clone [email protected]:RedHatInsights/tangerine-frontend.git
cd tangerine-frontend
docker compose up --build
```
5. Pull the mistral LLM (data will persist in `data/ollama`):
You can now access the frontend at `http://localhost:3000`
#### Using huggingface text-embeddings-inference server to host embedding model (deprecated)
ollama previously did not have an OpenAI compatible API path for interacting with an embedding models (i.e. `/v1/embeddings`). We previously used huggingface's [text-embeddings-inference](https://github.com/huggingface/text-embeddings-inference) server to host the embedding model. If you wish
to use this to test different embedding models that are not supported by ollama, follow these steps:
1. Make sure [git-lfs](https://git-lfs.com/) is installed:
- Fedora: `sudo dnf install git-lfs`
- MacOS: `brew install git-lfs`
Then, activate it globally with:
```text
docker exec tangerine-ollama ollama pull mistral
git lfs install
```
6. The API can now be accessed on `http://localhost:5000`
2. Create a directory in the 'data' folder to house the embedding model and download the model, for example to use `nomic-embed-text-v1.5`:
```text
mkdir data/embeddings
git clone https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 \
data/embeddings/nomic-embed-text
```
3. Search for `uncomment to use huggingface text-embeddings-inference` in [./docker-compose.yml](docker-compose.yml) and uncomment all relevant lines
### Local Environment Setup for Apple Silicon Macs
### Without Docker Compose (supported with Mac)
Some of the images used in the `docker-compose.yml` are unsupported on Apple silicon. In order to develop on those systems you will need to start some of the processes manually.
1. You'll need to have the following installed and working before proceeding:
* brew
* pipenv
* pyenv
* docker or podman
- `pipenv`
- `pyenv`
- `docker` or `podman`
- (on Mac) `brew`
2. Install ollama
```text
brew install ollama
```
- visit the [ollama download page](https://ollama.com/download)
2. Start ollama
- (on Mac) you can use brew:
```sh
```text
brew install ollama
```
3. Start ollama
```text
ollama serve
```
3. Pull the language and embedding models
4. Pull the language and embedding models
```text
ollama pull mistral
ollama pull nomic-embed-text
```
4. Install the C API for Postgres (libpq)
5. (on Mac) install the C API for Postgres (libpq)
```sh
brew install libpq
Expand All @@ -120,10 +164,10 @@ Some of the images used in the `docker-compose.yml` are unsupported on Apple sil
export CPPFLAGS="-I/opt/homebrew/opt/libpq/include"
```
5. Start the vector database
6. Start the vector database
```text
docker run \
docker run -d \
-e POSTGRES_PASSWORD="citrus" \
-e POSTGRES_USER="citrus" \
-e POSTGRES_DB="citrus" \
Expand All @@ -132,23 +176,20 @@ Some of the images used in the `docker-compose.yml` are unsupported on Apple sil
pgvector/pgvector:pg16
```
6. Prepare your python virtual environment:
7. Prepare your python virtual environment:
```sh
pipenv install
pipenv shell
```

7. Start Tangerine Backend

> [!NOTE]
> The default tangerine port, 5000, is already claimed by Bonjour on Macs, so we need to use a different port instead.
8. Start Tangerine Backend

```sh
flask run --host=127.0.0.1 --port=8000
flask run
```

You can now communicate with the API on port `8000`
9. Access the API on port `8000`

```sh
curl -XGET 127.0.0.1:8000/api/agents
Expand All @@ -157,20 +198,7 @@ Some of the images used in the `docker-compose.yml` are unsupported on Apple sil
}
```

### Available API Paths

| Path | Method | Description |
| ---------------------------------- | -------- | -------------------------- |
| `/api/agents` | `GET` | Get a list of all agents |
| `/api/agents` | `POST` | Create a new agent |
| `/api/agents/<id>` | `GET` | Get an agent |
| `/api/agents/<id>` | `PUT` | Update an agent |
| `/api/agents/<id>` | `DELETE` | Delete an agent |
| `/api/agents/<id>/chat` | `POST` | Chat with an agent |
| `/api/agents/<id>/documents` | `POST` | Agent document uploads |
| `/api/agents/<id>/documents` | `DELETE` | Delete agent documents |
| `/api/agentDefaults` | `GET` | Get agent default settings |
| `/ping`
10. (optional) Follow steps to start the [tangerine-frontend](https://github.com/RedHatInsights/tangerine-frontend)

## Syncrhonizing Documents from S3

Expand All @@ -188,10 +216,34 @@ To do so you'll need to do the following:
export BUCKET="mybucket"
```
If using docker compose, store these environment variables in `.env`:
```sh
echo 'AWS_ACCESS_KEY_ID=MYKEYID' >> .env
echo 'AWS_DEFAULT_REGION=us-east-1' >> .env
echo 'AWS_ENDPOINT_URL_S3=https://s3.us-east-1.amazonaws.com' >> .env
echo 'AWS_SECRET_ACCESS_KEY=MYACCESSKEY' >> .env
echo 'BUCKET=mybucket' >> .env
```
2. Create an `s3.yaml` file that describes your agents and the documents they should ingest. See [s3-example.yaml](s3-example.yaml) for an example.
If using docker compose, copy this config into your container:
```text
docker cp s3.yaml tangerine-backend:/opt/app-root/src/s3.yaml
```
3. Run the S3 sync job:
- With docker compose:
```text
docker exec -ti tangerine-backend flask s3sync
```
- Without:
```sh
flask s3sync
```
Expand All @@ -203,3 +255,18 @@ The OpenShift templates contain a CronJob configuration that is used to run this
## Run Tangerine Frontend Locally
The API can be used to create/manage/update agents, upload documents, and to chat with each agent. However, the frontend provides a simpler interface to manage the service with. To run the UI in a development environment, see [tangerine-frontend](https://github.com/RedHatInsights/tangerine-frontend)
## Available API Paths
| Path | Method | Description |
| ---------------------------------- | -------- | -------------------------- |
| `/api/agents` | `GET` | Get a list of all agents |
| `/api/agents` | `POST` | Create a new agent |
| `/api/agents/<id>` | `GET` | Get an agent |
| `/api/agents/<id>` | `PUT` | Update an agent |
| `/api/agents/<id>` | `DELETE` | Delete an agent |
| `/api/agents/<id>/chat` | `POST` | Chat with an agent |
| `/api/agents/<id>/documents` | `POST` | Agent document uploads |
| `/api/agents/<id>/documents` | `DELETE` | Delete agent documents |
| `/api/agentDefaults` | `GET` | Get agent default settings |
| `/ping` | `GET` | Health check endpoint |
Loading

0 comments on commit 7fc696e

Please sign in to comment.