Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix existing prescriptions #38952

Merged
merged 5 commits into from
Jan 25, 2023
Merged

Fix existing prescriptions #38952

merged 5 commits into from
Jan 25, 2023

Conversation

VannTen
Copy link
Member

@VannTen VannTen commented Jan 24, 2023 •
edited
Loading

  • Remove metadata in run.justification
  • Fix metadata in run.release_notes

Related Issues and Dependencies

part of thoth-station/adviser#2402
…

This introduces a breaking change

  • Yes

This should yield a new module release

  • Yes

This Pull Request implements

Thix fixes a bunch o files containing metadata in incorrect places

There is still 274 other invalid prescriptions files in the repo, which appears to be mostly scorecard prescriptions, (for which we have no schema in adviser). For example:

2023-01-24 11:55:50,051 79753 ERROR    thoth.adviser.prescription.v1.prescription:305: prescriptions/up_/update-checker/scorecards_code_review.yaml is invalid:
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['project_revision']. Got '2f3b1db5ae816cb3d9c7ad65c3d3a955e4b8197d'
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['project_url']. Got 'github.com/bboe/update_checker'
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['scorecard_score']. Got 1
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['tag']. Got 'no-code-review'
2023-01-24 11:55:50,138 79753 ERROR    thoth.adviser.prescription.v1.prescription:305: prescriptions/su_/subprocess32/scorecards_cii_best_practices.yaml is invalid:
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['project_revision']. Got 'b63954f56166eca312afb70b6d3a4af1128f81f3'
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['project_url']. Got 'github.com/google/python-subprocess32'
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['scorecard_score']. Got 0
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['tag']. Got 'no-cii'
2023-01-24 11:55:50,153 79753 ERROR    thoth.adviser.prescription.v1.prescription:305: prescriptions/as_/astroplan/scorecards_branch_protection.yaml is invalid:
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['project_revision']. Got '920f189701b03f13ff3328bca51d95cc858bcd34'
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['project_url']. Got 'github.com/astropy/astroplan'
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['scorecard_score']. Got 0
extra keys not allowed @ data['units']['wraps'][0]['run']['justification'][0]['tag']. Got 'no-branch-protection'

(if you want to test the validation : get latest master of adviser, merge thoth-station/adviser#2411 in your local branch, then run pre-commit try-repo path/to/adviser/repo validate-prescriptions --verbose --all-files in the prescription repo )

@goern

What do we do with the scorecards prescriptions for now ?

@VannTen
Remove metadata in run.justification
ffb0371 
Command used:

grep -Rl metadata prescriptions/ | \
         xargs -n 500 -P 16 -- yq -iy --indentless-lists \
         ".units[][] |= (.metadata = .run.justification[0].metadata[0]?
         | del(.run.justification[0].metadata))" && \
         git ch prescriptions/_containers/quay_security.yaml && \
         git diff -U0 | grepdiff 'null' --output-matching=hunk | \
         git apply -R --unidiff-zero && \
         git diff -U0 | grepdiff 'metadata' --output-matching=hunk | \
         git apply --cached --unidiff-zero
@VannTen
Fix metadata in run.release_notes
c0524bf 
grep -Rl metadata prescriptions/ | \
         xargs -n 500 -P 16 -- yq -iy --indentless-lists \
         ".units[][] |= (.metadata = .run.release_notes.metadata[0]?
         | del(.run.release_notes.metadata))" && \
         git ch prescriptions/_containers/quay_security.yaml && \
         git diff -U0 | grepdiff 'null' --output-matching=hunk | \
         git apply -R --unidiff-zero && \
         git diff -U0 | grepdiff 'metadata' --output-matching=hunk | \
         git apply --cached --unidiff-zero
@sesheta sesheta added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 24, 2023
@VannTen
Copy link
Member Author

VannTen commented Jan 24, 2023 •
edited
Loading

Fixed some other oddities in the last commit and confirmed : we have 264 scorecards prescriptions with no schema, which are currently by definition invalid.

Proposed solution: Remove all of them temporarily, once we had validation support revert the removal commit.
@goern @codificat @harshad16 advices ?

@goern
Copy link
Member

goern commented Jan 24, 2023

sounds good to me, pls create a tracker issue for the backlog so that we don't forget about adding them again...

@VannTen VannTen mentioned this pull request Jan 24, 2023
Open
@VannTen VannTen changed the title Put metadata in correct places in existing prescriptions Fix existing prescriptions Jan 24, 2023
@goern
Copy link
Member

goern commented Jan 25, 2023

/lgtm
/approve

@sesheta sesheta added the lgtm Indicates that a PR is ready to be merged. label Jan 25, 2023
@sesheta
Copy link
Member

sesheta commented Jan 25, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: goern

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sesheta sesheta added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 25, 2023
@sesheta sesheta merged commit 1b30976 into thoth-station:master Jan 25, 2023
@codificat codificat mentioned this pull request Jan 25, 2023
Open
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harshad16 harshad16 Awaiting requested review from harshad16

@KPostOffice KPostOffice Awaiting requested review from KPostOffice

Assignees

@goern goern

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@VannTen @goern @sesheta