Automate validation of pipeline configs

[russellb]

Closes #131

1cfd4bb Add jsonschema for validating pipeline configuration
c3bd917 Add a script that will validate pipeline configurations
18b4741 Add tox env for validating pipeline configuration
3090e65 Run pipeline config validation in CI

commit 1cfd4bb
Author: Russell Bryant [email protected]
Date: Sun Jul 14 12:50:45 2024 -0400

Add jsonschema for validating pipeline configuration

This commit introduces a `v1.json` jsonschema for use with validating
pipeline configuration. This follows the same format used in the
`instructlab/schema` repository which we use for validating taxonomy
`qna.yaml` files.

It's very likely that the schema is missing some details that are
allowed. It does pass against all configurations currently in the
source tree, though.

Signed-off-by: Russell Bryant <[email protected]>

commit c3bd917
Author: Russell Bryant [email protected]
Date: Sun Jul 14 12:53:43 2024 -0400

Add a script that will validate pipeline configurations

This script loads the pipeline validation schema and validates all
configurations under `src/instructlab/sdg/pipelines/`.

Signed-off-by: Russell Bryant <[email protected]>

commit 18b4741
Author: Russell Bryant [email protected]
Date: Sun Jul 14 12:52:42 2024 -0400

Add tox env for validating pipeline configuration

To run the script that validates pipeline configurations, you may now
run:

    tox -e validate-pipelines

or

    make validate-pipelines

Signed-off-by: Russell Bryant <[email protected]>

commit 3090e65
Author: Russell Bryant [email protected]
Date: Sun Jul 14 13:00:03 2024 -0400

Run pipeline config validation in CI

Updating the existing linting job to include pipeline config
validation, as well.

Signed-off-by: Russell Bryant <[email protected]>

[russellb]

Also, a follow-up enhancement to this would be to utilize the schema at runtime to validate custom schemas as soon as they are loaded and prior to trying to execute them.

This would require some special care, as right now, validation will fail if any fields are present that were not expected. The code currently allows that. It may occur with a config using a newer minor version that the code doesn't understand.

If we are OK rejecting any configs that are a newer version than what the code knows about, I think it would work OK.

[markmc]

How about loading this from the instructlab.sdg package

[russellb]

yes, that would be better

[markmc]

And loading these from a command-line argument?

Then it can be a tool that can be used to validate custom pipeline configs using the installed instructlab.sdg schema?

[russellb]

Yeah, that sounds good.

We could even leave this as the default when you don't pass an argument

[markmc]

My main outstanding comment is about making this a bit more general purpose, but that can come later

I think this is good to go ... I'm fine if you'd prefer to squash all my commits though

[russellb]

My main outstanding comment is about making this a bit more general purpose, but that can come later

I think this is good to go ... I'm fine if you'd prefer to squash all my commits though

Thanks for the additions! lgtm.

I'm going to merge as having reviewed each others additions to the PR.

[russellb]

filed this for making the validation script more generally useful -- #139