Use instructlab-schema package to parse qna.yaml files

.gitignore

@@ -164,3 +164,6 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# IDEs
+.vscode/

.pylintrc

@@ -90,7 +90,7 @@ persistent=yes
 
 # Minimum Python version to use for version dependent checks. Will default to
 # the version used to run pylint.
-py-version=3.9
+py-version=3.10
 
 # Discover python modules and packages in the file system subtree.
 recursive=no
@@ -379,7 +379,8 @@ int-import-graph=
 known-standard-library=
 
 # Force import order to recognize a module as part of a third party library.
-known-third-party=enchant
+known-third-party=enchant,
+    instructlab.schema,
 
 # Couples of modules and preferred modules, separated by a comma.
 preferred-modules=

.spellcheck-en-custom.txt

@@ -1,5 +1,12 @@
 # make spellcheck-sort
 # Please keep this file sorted:
 # SPDX-License-Identifier: Apache-2.0
+Dataset
+dataset
+datasets
+Pre
+pre
 sdg
+subfolder
 Tatsu
+yaml

docs/data_mixing.md

@@ -2,12 +2,13 @@
 
 As one of the last steps in data generation, the SDG library can optionally mix multiple datasets into a single output dataset in proportions specified by a recipe yaml file. The current implementation is designed to be used with mostly static recipes, that get used by default for every `ilab data generate` run. There is not yet an easy way to specify the recipe to use with each generation run, but we do make it possible to change the default recipe used for skills and/or knowledge data generation.
 
-The primary intended use of this is to specify an optional pregenerated dataset maintained by the InstructLab community that can improve training results when attempting to teach new skills to a model. This process is a bit manual for now, and the steps to do that are documented below.
+The primary intended use of this is to specify an optional pre-generated dataset maintained by the InstructLab community that can improve training results when attempting to teach new skills to a model. This process is a bit manual for now, and the steps to do that are documented below.
 
-## Using InstructLab Community Pregenerated Dataset
+## Using InstructLab Community Pre-generated Dataset
 
-To use the [InstructLab Community pregenerated dataset](https://huggingface.co/datasets/instructlab/InstructLabCommunity) with all skills training, we first need to create a default recipe that specifies this dataset to include when mixing generated skills data. This recipe will get automatically picked up if placed in a `default_data_recipes/skills.yaml` subfolder and file under one of several possible locations - `'/home/<user>/.local/share/instructlab/sdg'`, `'/usr/local/share/instructlab/sdg'`, or `'/usr/share/instructlab/sdg'`. The exact list of possible locations is platform-dependent, and can be enumerated by a Python command like below:
-```
+To use the [InstructLab Community pre-generated dataset](https://huggingface.co/datasets/instructlab/InstructLabCommunity) with all skills training, we first need to create a default recipe that specifies this dataset to include when mixing generated skills data. This recipe will get automatically picked up if placed in a `default_data_recipes/skills.yaml` subfolder and file under one of several possible locations - `'/home/<user>/.local/share/instructlab/sdg'`, `'/usr/local/share/instructlab/sdg'`, or `'/usr/share/instructlab/sdg'`. The exact list of possible locations is platform-dependent, and can be enumerated by a Python command like below:
+
+```python
 python3 -c '
 import os, platformdirs
 print(list(platformdirs.PlatformDirs(
@@ -18,7 +19,8 @@ print(list(platformdirs.PlatformDirs(
 For this example, we'll assume you want to place to default data recipe under the `~/.local/share/instructlab/sdg/` platform directory.
 
 Ensure that directory exists and create the recipe yaml file:
-```
+
+```shell
 mkdir -p ~/.local/share/instructlab/sdg/default_data_recipes/
 cat <<EOF > ~/.local/share/instructlab/sdg/default_data_recipes/skills.yaml
 datasets:
@@ -27,15 +29,15 @@ datasets:
 EOF
 ```
 
-Next, download the instructlab_community.jsonl file from https://huggingface.co/datasets/instructlab/InstructLabCommunity/tree/main and place it in `~/.local/share/instructlab/datasets/`, where the recipe we wrote above will pick it up. If you prefer to place this pregenerated dataset in a different location, you can specify the absolute path to that different location in your recipe yaml file instead of using relative paths as shown here.
+Next, download the `instructlab_community.jsonl` file from <https://huggingface.co/datasets/instructlab/InstructLabCommunity/tree/main> and place it in `~/.local/share/instructlab/datasets/`, where the recipe we wrote above will pick it up. If you prefer to place this pre-generated dataset in a different location, you can specify the absolute path to that different location in your recipe yaml file instead of using relative paths as shown here.
 
 Then, during your next `ilab data generate`, you should see output near the end like:
 
-```
+```log
 INFO 2024-08-06 16:08:42,069 instructlab.sdg.datamixing:123: Loading dataset from /home/user/.local/share/instructlab/datasets/instructlab_community.jsonl ...
 Generating train split: 13863 examples [00:00, 185935.73 examples/s]
 INFO 2024-08-06 16:08:42,414 instructlab.sdg.datamixing:125: Dataset columns: ['messages', 'metadata', 'id']
 INFO 2024-08-06 16:08:42,414 instructlab.sdg.datamixing:126: Dataset loaded with 13863 samples
 ```
 
-Your resulting skills_train_*.jsonl file will now contain the additional 13k+ examples from the precomputed dataset, which should ensure your subsequent skills training doesn't regress in already-learned skills while being taught the new skill.
+Your resulting `skills_train_*.jsonl` file will now contain the additional 13k+ examples from the pre-computed dataset, which should ensure your subsequent skills training doesn't regress in already-learned skills while being taught the new skill.

pyproject.toml

@@ -12,7 +12,7 @@ authors = [
 description = "Synthetic Data Generation"
 readme = "README.md"
 license = {text = "Apache-2.0"}
-requires-python = ">=3.9"
+requires-python = ">=3.10"
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Environment :: Console",
@@ -22,7 +22,6 @@ classifiers = [
     "Operating System :: POSIX :: Linux",
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
@@ -53,7 +52,7 @@ where = ["src"]
 include = ["instructlab.sdg"]
 
 [tool.ruff]
-target-version = "py39"
+target-version = "py310"
 # same as black's default line length
 line-length = 88
 
@@ -92,6 +91,7 @@ from-first = true
 known-local-folder = ["tuning"]
 
 [tool.mypy]
+python_version = "3.10"
 disable_error_code = ["import-not-found", "import-untyped"]
 exclude = [
     "^src/instructlab/sdg/generate_data\\.py$",

requirements.txt

@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: Apache-2.0
 click>=8.1.7,<9.0.0
 httpx>=0.25.0,<1.0.0
+instructlab-schema>=0.4.0
 langchain-text-splitters
 openai>=1.13.3,<2.0.0
 platformdirs>=4.2
@@ -9,4 +10,3 @@ platformdirs>=4.2
 # do not use 8.4.0 due to a bug in the library
 # https://github.com/instructlab/instructlab/issues/1389
 tenacity>=8.3.0,!=8.4.0
-instructlab-schema>=0.3.1