Skip to content

Latest commit

 

History

History
68 lines (49 loc) · 2.3 KB

README.md

File metadata and controls

68 lines (49 loc) · 2.3 KB
Raw

SEED: Domain-Specific Data Curation With Large Language Models

This is the code repo for the paper "SEED: Domain-Specific Data Curation With Large Language Models".

SEED is an approach that leverages Large Language Models (LLMs) to automatically generate domain-specific data curation solutions. By describing a task, input data, and expected output, the SEED compiler produces an executable pipeline consisting of LLM-generated code, small models, and data access modules.

SEED: Legay Version

Link to the Legacy Version

SEED: Dev Version

Info

Current Version: v0.2.0

Compatibility: Tested on MacBook M1 Pro, MacOS 12.6.2, Python 3.9, Pytorch 1.13.1

Hardware Requirements: None

Installation

First install the prerequisites and the SEED package.

git clone [email protected]:Magolor/SEED.git
cd ./SEED/SeeD/
pip install -r requirements.txt
pip install -e .
cd ..

The most basic config is the auth key to OpenAI API. You can create a openai_api.json file to store it:

{
    "model": "gpt-4-turbo-preview",
    "api_key": "<YOUR_API_KEY>",
    "organization": "<YOUR_ORGANIZATION>"
}

Or you will be asked to manually input them in terminals during SEED setup.

Then run the installer to initialize configurations:

python post_install.py

Tutorials

  1. Recommended: A full tutorial for understanding how SEED works in general: amazon_google_full_tutorial.
  2. A short version of the same amazon google project: amazon_google_tutorial.
  3. A code generation agent tutorial: restaurant_tutorial.
  4. Others:

TODO

SEED is currently under development, many features and optimizations coming!

  • Improve code generation.
  • Add Seq2Seq model support from legacy.
  • Add tools agent from legacy.
  • Improve RAG.
  • Improve hyperparaeter search.
  • Integrate hyperparaeter search.
  • Support multiple outputs.
  • ...