Patch main-ci workflow

[anngvu]

What this does

• Update to schematic version 24.4.1, which supposedly should be faster for template generation and will appear soon in DCA
  

• Better recognition of when to commit, easy fix for below instances (which happens 2-3% of the time?)
  

  • Additional explanation: some changes, like edits to notes: and source: in the src files in modules are like silent mutations and do not lead to meaningful changes in the jsonld, json files

• Remove the -rr flag to exactly match our DCA config https://github.com/Sage-Bionetworks/data_curator_config/blob/prod/NF-OSI/dca_config.json#L25

Notes for reviewers on interpreting results and followup investigation

Update 1

@allaway, it looks like generation works fine, something has changed with validation (compare with the tests from your recent PR #433 which used 24.2.1).

Error:

Looked a bit at the code around the indicated line... I will modify the mock manifests to understand what the expected behavior now when we specify a "num" rule.
https://github.com/Sage-Bionetworks/schematic/blob/c51af2d8419645fa04a13cc387e7811f1afe5b35/schematic/models/validate_attribute.py#L1025

Update 2

OK, I realized we need to remove the -rr flag now since we no longer want to restrict Great Expectations rules. Now the only issue is

error: '' is not of type 'array'
[['2', 'modelSystemName', "'' is not of type 'array'", '']

[github-actions]

Test Suite Report

(outdated run -- scroll down to next report 👇🏼 )

Template Generation

template	result	link
AnimalIndividualTemplate	❌
BiospecimenTemplate	😄	template link
ClinicalAssayTemplate	😄	template link
EpigeneticsAssayTemplate	😄	template link
FlowCytometryTemplate	😄	template link
GenomicsAssayTemplate	😄	template link
GenomicsAssayTemplateExtended	😄	template link
HumanCohortTemplate	😄	template link
ImagingAssayTemplate	😄	template link
LightScatteringAssayTemplate	😄	template link
MethylationArrayTemplate	😄	template link
MRIAssayTemplate	😄	template link
PharmacokineticsAssayTemplate	😄	template link
PlateBasedReporterAssayTemplate	😄	template link
ProcessedAlignedReadsTemplate	😄	template link
ProcessedExpressionTemplate	😄	template link
ProcessedVariantCallsTemplate	😄	template link
ProteomicsAssayTemplate	😄	template link
ProtocolTemplate	😄	template link
RNASeqTemplate	😄	template link
ScRNASeqTemplate	😄	template link
UpdateMilestoneReport	😄	template link
WESTemplate	😄	template link
WGSTemplate	😄	template link

Manifest Validation

manifest	result	expectation
GenomicsAssayTemplate_0.csv	❌	Lists can be blank if attr not required using ‘list like’ rule
GenomicsAssayTemplate_1.csv	❌	Mixing blanks and regular list values works
GenomicsAssayTemplate_2.csv	❌	Conditional validation for attributes is currently not supported
ScRNASeqTemplate_0.csv	❌	Single list val works by using ‘list like’ rule
ScRNASeqTemplate_1.csv	😄	Fail because of missing data in required field libraryStrand

[github-actions]

Test Suite Report 24.4.1

Template Generation

template	result	link
AnimalIndividualTemplate	😄	template link
BiospecimenTemplate	😄	template link
ClinicalAssayTemplate	😄	template link
EpigeneticsAssayTemplate	😄	template link
FlowCytometryTemplate	😄	template link
GenomicsAssayTemplate	😄	template link
GenomicsAssayTemplateExtended	😄	template link
HumanCohortTemplate	😄	template link
ImagingAssayTemplate	😄	template link
LightScatteringAssayTemplate	😄	template link
MethylationArrayTemplate	😄	template link
MRIAssayTemplate	😄	template link
PharmacokineticsAssayTemplate	😄	template link
PlateBasedReporterAssayTemplate	😄	template link
ProcessedAlignedReadsTemplate	😄	template link
ProcessedExpressionTemplate	😄	template link
ProcessedVariantCallsTemplate	😄	template link
ProteomicsAssayTemplate	😄	template link
ProtocolTemplate	😄	template link
RNASeqTemplate	😄	template link
ScRNASeqTemplate	😄	template link
UpdateMilestoneReport	😄	template link
WESTemplate	😄	template link
WGSTemplate	😄	template link

Manifest Validation

manifest	result	expectation
GenomicsAssayTemplate_0.csv	❌	Lists can be blank if attr not required using ‘list like’ rule
GenomicsAssayTemplate_1.csv	❌	Mixing blanks and regular list values works
GenomicsAssayTemplate_2.csv	❌	Conditional validation for attributes is currently not supported
GenomicsAssayTemplate_control.csv	😄	There should be no issue with this template.
ScRNASeqTemplate_0.csv	❌	Single list val works by using ‘list like’ rule
ScRNASeqTemplate_1.csv	😄	Fail because of missing data in required field libraryStrand

[anngvu]

Hey @allaway @cconrad8, I think we can merge this, just wanted you both to be aware that:

• Some validation tests are failing in 24.4.1, so just expect to see this on future PRs for now and not be too concerned -- have communicated this issue to Amy during D&T meeting, so we just wait to see what they say/if there's a fix -- we might need a temp workaround, e.g. by modifying a validation rule, but that would be a separate PR anyway
• Also, this should fix the more minor issue with empty commits

[allaway]

this LGTM, thank you @anngvu !