Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Add logger for compiler and marshal while comparing union #6034

Merged
merged 34 commits into from
Nov 20, 2024
Merged
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
f06cdc6
feat: fix Union type with dataclass ambiguous error
mao3267 Oct 18, 2024
8660db5
Merge branch 'master' of https://github.com/mao3267/flyte into fix/#5…
mao3267 Nov 1, 2024
47ccbd1
fix: direct json comparison for superset
mao3267 Nov 1, 2024
85489dc
fix: go.mod missing entry for error
mao3267 Nov 1, 2024
cc685bb
fix: update go module and sum
mao3267 Nov 1, 2024
3a629e1
refactor: gci format
mao3267 Nov 1, 2024
aa4d98e
test: add dataset casting tests for same (one/two levels) and supers…
mao3267 Nov 1, 2024
b282e5f
Merge branch 'master' of https://github.com/mao3267/flyte into fix/#5…
mao3267 Nov 8, 2024
818afb7
fix: support Pydantic BaseModel comparison
mao3267 Nov 8, 2024
d6468b6
fix: handle nested pydantic basemodel
mao3267 Nov 8, 2024
ada05ed
Reviews from Eduardo
Future-Outlier Nov 11, 2024
56623e3
fix: support strict subset match
mao3267 Nov 15, 2024
b8f38a7
test: update strict subset match test
mao3267 Nov 15, 2024
b698769
fix: missing go mod entry
mao3267 Nov 15, 2024
70ad767
fix: missing go mod entry
mao3267 Nov 15, 2024
9dc3fa6
fix: go mod entry
mao3267 Nov 15, 2024
b224a02
make go-tidy
Future-Outlier Nov 15, 2024
7ed9be2
comments
Future-Outlier Nov 15, 2024
6fe8871
Merge branch 'fix/#5489-dataclass-mismatch' of https://github.com/mao…
mao3267 Nov 15, 2024
64343c8
fix: strict subset match with draft 2020-12 mashumaro
mao3267 Nov 18, 2024
8ccced5
Merge branch 'master' of https://github.com/mao3267/flyte into fix/#5…
mao3267 Nov 18, 2024
0def0ad
refactor: make go-tidy
mao3267 Nov 18, 2024
81445a7
fix: support strict subset match with ambiguity
mao3267 Nov 19, 2024
9b19f04
fix: change test name and fix err
mao3267 Nov 19, 2024
30aa096
Add comments
Future-Outlier Nov 20, 2024
e62ba6e
nit
Future-Outlier Nov 20, 2024
c6ac729
add flytectl go-tidy in makefile
Future-Outlier Nov 20, 2024
ba4d6f1
nit
Future-Outlier Nov 20, 2024
86a395e
fix: add comment for error checking
mao3267 Nov 20, 2024
7f28c35
test: basemodel castable test, two level dataclass and ParentToChild …
mao3267 Nov 20, 2024
66104dd
fix: add logger for jsonschema compiler
mao3267 Nov 20, 2024
b768576
Merge branch 'master' of https://github.com/mao3267/flyte into fix/#5…
mao3267 Nov 20, 2024
22d4dca
fix: add logger for marshal and compiler
mao3267 Nov 20, 2024
1f92e5c
better error msg format
Future-Outlier Nov 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 26 additions & 8 deletions flytepropeller/pkg/compiler/validators/typing.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,22 +28,40 @@
// For custom types, we expect the JSON schemas in the metadata to come from the same JSON schema package,
// specifically draft 2020-12 from Mashumaro.

srcSchemaBytes, _ := json.Marshal(sourceMetaData.GetFields())
tgtSchemaBytes, _ := json.Marshal(targetMetaData.GetFields())
srcSchemaBytes, err := json.Marshal(sourceMetaData.GetFields())
if err != nil {
logger.Infof(context.Background(), "Failed to marshal source metadata: [%v]", err)
return false
}

Check warning on line 35 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L33-L35

Added lines #L33 - L35 were not covered by tests
tgtSchemaBytes, err := json.Marshal(targetMetaData.GetFields())
if err != nil {
logger.Infof(context.Background(), "Failed to marshal target metadata: [%v]", err)
return false
}

Check warning on line 40 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L38-L40

Added lines #L38 - L40 were not covered by tests

compiler := jsonschema.NewCompiler()

err := compiler.AddResource("src", bytes.NewReader(srcSchemaBytes))
err = compiler.AddResource("src", bytes.NewReader(srcSchemaBytes))
if err != nil {
logger.Infof(context.Background(), "Failed to add resource to compiler: [%v]", err)

Check warning on line 46 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L46

Added line #L46 was not covered by tests
return false
}
err = compiler.AddResource("tgt", bytes.NewReader(tgtSchemaBytes))
if err != nil {
logger.Infof(context.Background(), "Failed to add resource to compiler: [%v]", err)

Check warning on line 51 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L51

Added line #L51 was not covered by tests
return false
}

srcSchema, _ := compiler.Compile("src")
tgtSchema, _ := compiler.Compile("tgt")
srcSchema, err := compiler.Compile("src")
if err != nil {
logger.Infof(context.Background(), "Failed to compile source schema: [%v]", err)
return false
}

Check warning on line 59 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L57-L59

Added lines #L57 - L59 were not covered by tests
tgtSchema, err := compiler.Compile("tgt")
if err != nil {
logger.Infof(context.Background(), "Failed to compile target schema: [%v]", err)
return false
}

Check warning on line 64 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L62-L64

Added lines #L62 - L64 were not covered by tests

// Compare the two schemas
errs := jscmp.Compare(tgtSchema, srcSchema)
Expand All @@ -63,20 +81,20 @@
func isSameTypeInJSON(sourceMetaData, targetMetaData *structpb.Struct) bool {
srcSchemaBytes, err := json.Marshal(sourceMetaData.GetFields())
if err != nil {
logger.Infof(context.Background(), "Failed to marshal source metadata: %v", err)
logger.Infof(context.Background(), "Failed to marshal source metadata: [%v]", err)

Check warning on line 84 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L84

Added line #L84 was not covered by tests
return false
}

tgtSchemaBytes, err := json.Marshal(targetMetaData.GetFields())
if err != nil {
logger.Infof(context.Background(), "Failed to marshal target metadata: %v", err)
logger.Infof(context.Background(), "Failed to marshal target metadata: [%v]", err)

Check warning on line 90 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L90

Added line #L90 was not covered by tests
return false
}

// Use jsondiff to compare the two schemas
patch, err := jsondiff.CompareJSON(srcSchemaBytes, tgtSchemaBytes)
if err != nil {
logger.Infof(context.Background(), "Failed to compare JSON schemas: %v", err)
logger.Infof(context.Background(), "Failed to compare JSON schemas: [%v]", err)

Check warning on line 97 in flytepropeller/pkg/compiler/validators/typing.go

View check run for this annotation

Codecov / codecov/patch

flytepropeller/pkg/compiler/validators/typing.go#L97

Added line #L97 was not covered by tests
return false
}

Expand Down
Loading