Import job-runner's parsing #32
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghickman
force-pushed
the
support-job-runner
branch
2 times, most recently
from
f19047b to
0ae56ad
Compare
pipeline/legacy.py
Outdated
| permitted_privacy_levels = [ | ||
| "highly_sensitive", | ||
| "moderately_sensitive", | ||
| "minimally_sensitive", |
There was a problem hiding this comment.
I had no idea there even was a minimally sensitive level!
ghickman
force-pushed
the
support-job-runner
branch
2 times, most recently
from
7d36f70 to
35eb00d
Compare
This uses the pydantic models to load and validate a given project.yaml, but keeps all the job-runner functions using dicts. There is duplicate validation which will be removed in favour of pydantic model validation in a subsequent change.
We manually parse the run command to build up Command instances so we also have to handle basic validation like this.
load_pipeline is now a full replacement for it.
ghickman
force-pushed
the
support-job-runner
branch
3 times, most recently
from
8b6e671 to
f6126f1
Compare
bloodearnest
approved these changes
May 20, 2022
ghickman
force-pushed
the
support-job-runner
branch
from
7392e4b to
5c03b11
Compare
job-runner needs to mutate the arguments passed to extraction commands (eg cohortextractor and databuilder) based on pipeline config. However we don't think that should be exposed to other users of the library so now we have boundary where this library is concerned with parsing and validating the config.
We need various forms of the `run` key a config provides so instead of library consumers creating these themselves we now only store the raw version (renamed from run to raw to avoid run.run) and provide properties for the other forms.
Rather than pulling in is_extraction_command from job-runner this allows us a lookup table of validation functions which dispatch on the specific image and command.
ghickman
force-pushed
the
support-job-runner
branch
from
5c03b11 to
40fee6e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pulls in
project.py(aslegacy.py) from job-runner and refactors it into the pydantic models. All validation is currently handled with pydantic (although, as per #31 I don't think that's the ideal situation), with the generation of ActionSpecifications (and thus mutation of the run command for that) left to that method.I've hopefully made reviewable commits, so they're the possible to follow.
Ref opensafely-core/job-runner#366