Concreter

A web app to enhance document with chatGPT and sympy.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

The Graph Database

(back to top)

Built With

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

(back to top)

Getting Started

Here's how you can init and start the project.

Three ways: local docker k8s

Local

• Postgres database
• AgDb database
• Python gRPC server
• Axum server

Postgres

# Init a Postgres database with the following settings:
initdb -D /path/to/pgdata --locale=en_US.UTF-8 -U $USER -W

# Start the database:
pg_ctl -D /path/to/pgdata -l /path/to/log.log start

# Create database:
createdb app_db -O $USER

# Set the PgUrl environment variable:
export PG_URL="postgres://$USER:password@localhost/app_db"

# Follows is the commands to end and clean the service:
pg_ctl -D /path/to/pgdata stop
rm -rf /path/to/pgdata

AgDb

AgDb is a graph database.

You need give it a .agdb suffix file to init the database. For example:

mkdir -p ag_db && touch ag_db/ag.agdb
export AG_FILE="path/to/ag.agdb"

python gRPC

To enable symbolic computation, I use python's sympy. For time limitation, I don't have enough time to explore pyO3, so I just use gRPC to interact with python in Rust. Sympy only support python <= 3.10

# create a virtual environment
conda create -n py310 python=3.10

# activate the virtual environment
conda activate py310

# install the dependencies
pip install -r requirements.txt
# For ARM Mac, use
pip install socksio
conda install --file requirements.txt
pip install --upgrade openai

# generate the gRPC python code
export PB="./src_py" && python -m grpc_tools.protoc -I./proto --python_out=$PB --pyi_out=$PB --grpc_python_out=$PB proto/sym.proto

# set env varibles
export OPENAI_API_KEY=sk-123456

# Run gRPC service
python ./src_py/main.py

# Ctrl + C to end the service, and deactivate the virtual environment
conda deactivate

For both methods, you may need proxy configured. For docker, the host network should set proxy. For local machine, set the proxy in src_py/openai_utils.py:

client = OpenAI(
    # In docker, do not need to set proxy, for it uses host network which does.
    http_client=Client(proxies="http://127.0.0.1:7890"), timeout=30, max_retries=0
)

fronted

cd frontend && npm i && cd ..
export WEB_FOLDER=path/to/fronted/out
export FRONTEND_FOLDER=path/to/frontend

start axum server

export RPC_ADDR=http://localhost:50051
export SERVICE_PWD_KEY=   # you can gernerate using
export SERVICE_TOKEN_KEY= # `cargo run --examples gen_key`
export SERVICE_TOKEN_DURATION_SEC=1800  # in seconds
cargo run

Docker

postgres

docker pull postgres
docker run --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres
export PG_URL=postgres://postgres:postgres@localhost:5432/postgres

Agdb

mkdir -p ag_db && touch ag_db/ag.agdb
export AG_FILE="path/to/ag.agdb"

gRPC

# At the root of the project, run:
docker build -t rpc-py --target=rpc-py .
docker run -it -p 50051:50051 -e OPENAI_API_KEY=sk-123456 --rm --name rpc-py rpc-py

Frontend

cd frontend && npm i && cd ..
export WEB_FOLDER=path/to/fronted/out
export FRONTEND_FOLDER=path/to/frontend

start axum server

export RPC_ADDR=http://localhost:50051
export SERVICE_PWD_KEY=   # you can gernerate using
export SERVICE_TOKEN_KEY= # `cargo run --examples gen_key`
export SERVICE_TOKEN_DURATION_SEC=1800  # in seconds
cargo run

Kubernetes

build images

docker build -t rpc-py --target rpc-py .
docker build -t axum --target axum .
docker pull postgres

Set the env variables in deploy/k8s:

• secrets in kustomization.yaml
• persist voloum in postgres-deployment.yaml and axum-deployment.yaml

And load the images rpc-py, axum, postgres to your cluster

cd deploy/k8s && kubectl apply -k ./
# init the postgres, see blow
# port-forward
kubectl port-forward svc/axum-xxxx-xxx 8080:8080

Initialize the postgres table

Connect to db

psql postgres://postgres:postgres@localhost:5432/postgres
# For those using k8s
kubectl port-forward svc/postgres 5432:5432

Run sql in init sql

(back to top)

Usage

Register and login in.

Upload a markdown and enter the field the markdown about, you can see examples at examples folder.

(back to top)

Roadmap

• Use SeaORM and openDAL
• Fix: service dropped in minikube but works properly on docker kube
• Use vector storage
• Train transformer latex-sympy translator
• Formular sub
• 3D model support

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the Apache 2.0 License. See LICENSE.txt for more information.

(back to top)

Contact

Louis - [email protected]

Project Link: https://github.com/kingwingfly/Concreter

(back to top)