From 4744930db87e6465650ff83b67389998e6a771eb Mon Sep 17 00:00:00 2001 From: Ben Browning Date: Wed, 20 Nov 2024 12:36:55 -0500 Subject: [PATCH] Flesh out some of the technical details of SDG README.md This fills in some placeholder sections of our updated README.md. It's not as detailed as it should eventually be, but at least gives a bit more information as users browse the repository. Signed-off-by: Ben Browning --- .spellcheck-en-custom.txt | 9 ++++++--- README.md | 35 ++++++++++++++++++----------------- 2 files changed, 24 insertions(+), 20 deletions(-) diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt index 2130dd21..92258757 100644 --- a/.spellcheck-en-custom.txt +++ b/.spellcheck-en-custom.txt @@ -1,13 +1,13 @@ # make spellcheck-sort # Please keep this file sorted: # SPDX-License-Identifier: Apache-2.0 -Backport -backported -codebase +configs Dataset dataset datasets distractor +Eval +eval FIXME freeform ICL @@ -17,12 +17,15 @@ Langchain's LLM LLMBlock MCQ +Merlinite +Mixtral MMLU Ouput Pre pre Pregenerated qna +quantized repo sdg Splitter diff --git a/README.md b/README.md index cdf85d65..d9419295 100644 --- a/README.md +++ b/README.md @@ -41,37 +41,38 @@ You can import SDG into your Python files with the following items: ## Pipelines -There are four pipelines that are used in SDG. Each pipeline requires specific hardware specifications. - +A pipeline describes a series of steps to execute in-order to generate data. -*Full* - +There are three default pipelines shipped in SDG. These are the `simple`, `full`, and `eval` pipelines. Each pipeline requires specific hardware specifications -This pipeline is targeted for running SDG on consumer grade accelerators (GPUs). +### Simple Pipeline -*Simple* - +The [simple pipeline](src/instructlab/sdg/pipelines/simple) is designed to be used with [quantized Merlinite](https://huggingface.co/instructlab/merlinite-7b-lab-GGUF) as the teacher model. It exists to enable basic data generation results on lower end consumer grade hardware, such as laptops and desktops with small or no discrete GPUs. -### Pipeline architecture +### Full Pipeline + +The [full pipeline](src/instructlab/sdg/pipelines/full) is designed to be used with [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) as the the teacher model, but has also been successfully tested with smaller models such as [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) and even some quantized versions of the two above. This is the preferred data generation pipeline on higher end consumer grade hardware and on all enterprise hardware. -All the pipelines are written in YAML format. +### Eval Pipeline -Knowledge: +The [eval pipeline](src/instructlab/sdg/pipelines/eval) is used to generate [MMLU](https://en.wikipedia.org/wiki/MMLU) benchmark data that can be used to later evaluate a trained model on your knowledge dataset. It does not generate data for use during model training. -Grounded Skills: +### Pipeline architecture -Freeform Skills: +All the pipelines are written in a YAML format and must adhere to a [specific schema](src/instructlab/sdg/pipelines/schema/v1.json). - +The pipelines that generate data for model training (simple and full pipelines) expect to have three different pipeline configs - one each for knowledge, grounded skills, and freeform skills. They are expected to exist in files called `knowledge.yaml`, `grounded_skills.yaml`, and `freeform_skills.yaml` respectively. For background on the difference in knowledge, grounded skills, and freeform skills, refer to the [InstructLab Taxonomy repository](https://github.com/instructlab/taxonomy). ## Repository structure ```bash -|-- sdg/src/instructlab/ (1) -|-- sdg/docs/ (2) -|-- sdg/scripts/ (3) -|-- sgd/tests/ (4) +|-- src/instructlab/ (1) +|-- docs/ (2) +|-- scripts/ (3) +|-- tests/ (4) ``` 1. Contains the SDG code that interacts with InstructLab. 2. Contains documentation on various SDG methodologies. -3. Contains the code that tests the SDG data types: Knowledge, grounded skills, and freeform skills. -4. Contains all the CI tests for the SDG repository. \ No newline at end of file +3. Contains some utility scripts, but not part of any supported API. +4. Contains all the tests for the SDG repository.