Disallow extra fields other than `"@context"` #266

candleindark · 2024-11-18T04:52:43Z

This PR closes #75. It addresses #75 in the follow manner.

Make the models disallow data instances that have extra fields other than the "@context" field in both Pydantic and JSON level.
The Pydantic models continue to not have a context field.
Data instances with an "@context" field can be validated against a Pydantic model but ignored.

The solution implemented in this PR is based on the following script.

from typing import Any
import json

from pydantic import BaseModel, ConfigDict, model_validator
from pydantic.json_schema import JsonDict, JsonValue
from jsonschema import validate, Draft202012Validator
import jsonschema


def get_dict_without_context(d: Any) -> Any:
    """
    If a given object is a dictionary, return a copy of it without the
    `@context` key. Otherwise, return the input object as is.

    :param d: The given object
    :return: If the object is a dictionary, a copy of it without the `@context` key;
             otherwise, the input object as is.
    """
    if isinstance(d, dict):
        return {k: v for k, v in d.items() if k != "@context"}
    return d


def add_context(json_schema: JsonDict) -> None:
    """
    Add the `@context` key to the given JSON schema

    :param json_schema: The dictionary representing the JSON schema

    raises: ValueError if the `@context` key is already present in the given dictionary
    """
    context_key = "@context"
    context_key_title = "@Context"
    properties: JsonDict = json_schema.get("properties", {})
    required: list[JsonValue] = json_schema.get("required", [])

    if context_key in properties or context_key in required:
        msg = f"The '{context_key}' key is already present in the given JSON schema."
        raise ValueError(msg)

    properties[context_key] = {
        "format": "uri",
        "minLength": 1,
        "title": context_key_title,
        "type": "string",
    }
    # required.append(context_key)  # Uncomment this line to make `@context` required

    # Update the schema
    # This is needed to handle the case in which the keys are newly created
    json_schema["properties"] = properties
    json_schema["required"] = required


class Foo(BaseModel):
    x: int

    # Model validator to remove the `"@context"` key from data instance before
    # "base" validation is performed.
    _remove_context_key = model_validator(mode="before")(get_dict_without_context)

    model_config = ConfigDict(extra="forbid", json_schema_extra=add_context)


json_schema_ = Foo.model_json_schema()
print(json.dumps(json_schema_, indent=2))
"""
{
  "additionalProperties": false,
  "properties": {
    "x": {
      "title": "X",
      "type": "integer"
    },
    "@context": {
      "format": "uri",
      "minLength": 1,
      "title": "@Context",
      "type": "string"
    }
  },
  "required": [
    "x",
    "@context"
  ],
  "title": "Foo",
  "type": "object"
}
"""

instance_json_str = '{"x": 1}'
instance_json_str_with_context = '{"@context": "not a valid URI", "x": 1}'
instance_json_str_with_extra = '{"x": 1, "e": 42}'

vv = Foo.model_validate_json(instance_json_str)
print("\n====================================")
print(f"vv: {vv!r}")
"vv: Foo(x=1)"

# Ignore the context field in Pydantic level
vv_with_context = Foo.model_validate_json(instance_json_str_with_context)
print("\n====================================")
print(f"vv_with_context: {vv_with_context!r}")
"vv_with_context: Foo(x=1)"

# Disallow other extra fields in Pydantic level
try:
    Foo.model_validate_json(instance_json_str_with_extra)
except ValueError as e:
    print("\n====================================")
    print(e)
    """
    1 validation error for Foo
    e
      Extra inputs are not permitted [type=extra_forbidden, input_value=42, input_type=int]
        For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden
    """

instance = {"@context": "https://schema.org", "x": 1}
instance_with_invalid_context = {"@context": "invalid context", "x": 1}
instance_missing_context = {"x": 1}
instance_with_extra = {"@context": "https://schema.org", "x": 1, "e": 42}

# Validate an instance with valid context and x field
validate(instance, json_schema_, format_checker=Draft202012Validator.FORMAT_CHECKER)

# Instance with invalid context fails validation
try:
    validate(
        instance_with_invalid_context,
        json_schema_,
        format_checker=Draft202012Validator.FORMAT_CHECKER,
    )
except jsonschema.exceptions.ValidationError as e:
    print("\n====================================")
    print(e)
    """
    'invalid context' is not a 'uri'
    Failed validating 'format' in schema['properties']['@context']:
        {'format': 'uri', 'minLength': 1, 'title': '@Context', 'type': 'string'}
    On instance['@context']:
        'invalid context'
    """

# The context field is optional
validate(
    instance_missing_context,
    json_schema_,
    format_checker=Draft202012Validator.FORMAT_CHECKER,
)
print("\n====================================")
print("Instance without the `@context` key is valid")
"Instance without the `@context` key is valid"


# Instance with extra field fails validation
try:
    validate(
        instance_with_extra,
        json_schema_,
        format_checker=Draft202012Validator.FORMAT_CHECKER,
    )
except jsonschema.exceptions.ValidationError as e:
    print("\n====================================")
    print(e)
    """
    Additional properties are not allowed ('e' was unexpected)
    Failed validating 'additionalProperties' in schema:
        {'additionalProperties': False,
         'properties': {'x': {'title': 'X', 'type': 'integer'},
                        '@context': {'format': 'uri',
                                     'minLength': 1,
                                     'title': '@Context',
                                     'type': 'string'}},
         'required': ['x', '@context'],
         'title': 'Foo',
         'type': 'object'}
    On instance:
        {'@context': 'https://schema.org', 'x': 1, 'e': 42}
    """

TODOs

ensure that tests pass
- @yarikoptic: let's for now make @context optional and then discuss
run against all dandisets metadata (manifests) and see if any gets (more) broken.
- we have them already fetched to drogon -- just run there. */*/dandiset.jsonld files under /mnt/backup/dandi/dandiset-manifests-s3cmd .
  - check at Pydantic level (analysis available at Disallow extra fields other than "@context" #266 (comment))
  - check at JSONSchema level
- The diff-manifests-reports subcommand of https://github.com/dandi/dandisets-linkml-status-tools is the tool for the checking
- Reports of the checking is at https://github.com/dandi/dandi-schema-status
check meditor
provide tests for introduced changes

codecov · 2024-11-18T04:53:49Z

Codecov Report

Attention: Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 92.21%. Comparing base (6fd04fb) to head (5ca9e5e).

Files with missing lines	Patch %	Lines
dandischema/models.py	85.00%	3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6fd04fb) and HEAD (5ca9e5e). Click for more details.

HEAD has 217 uploads less than BASE

Flag BASE (6fd04fb) HEAD (5ca9e5e)

unittests 240 23

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #266      +/-   ##
==========================================
- Coverage   97.54%   92.21%   -5.34%     
==========================================
  Files          16       16              
  Lines        1753     1772      +19     
==========================================
- Hits         1710     1634      -76     
- Misses         43      138      +95

Flag	Coverage Δ
unittests	`92.21% <85.00%> (-5.34%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@context

…Dandiset` This allows validation of data instances of `Asset` and `Dandiset` to contain the `"@context"` key

@context

…t` and `Dandiset` This requires `"@context"` field at the JSON level

This also sets the `"additionalProperties"` of each corresponding model in JSON schema to `false`

candleindark · 2024-11-18T19:39:37Z

@satra @mvandenburgh This solution will generated JSON schemas that have "@context" as a property. Currently, I made "@context" as a required property, but I can make it optional. Will the additional "@context" property, required or optional, be a problem for the web UI? Can we make the web UI ignore this property if needed?

yarikoptic · 2024-11-18T21:41:20Z

are failing tests expected?

@mvandenburgh please point @candleindark to where to change URL to the json schema so some alternative one could be tested.

mvandenburgh · 2024-11-18T21:43:40Z

@candleindark the json schema used by the frontend is set here - https://github.com/dandi/dandi-archive/blob/master/dandiapi/api/views/info.py#L11-L14
If you want to use a custom schema, you could serve out the json file locally and point it at that url.

candleindark · 2024-11-18T23:47:00Z

are failing tests expected?

Yes. Because I made "@context" as a required property in the JSON schemas, the tests need to be updated.

These are not made available in `__all__`. Use of them triggers complains in the IDE.

yarikoptic · 2024-11-19T18:21:38Z

dandischema/models.py

@@ -1815,6 +1867,12 @@ class Asset(BareAsset):
        json_schema_extra={"readOnly": True, "nskey": "schema"}
    )


look here for an example on how we add extra to json schema.

May be you could add extra for your @context which would state something like "includeInUI": False

This is needed to handle the case in which the keys are newly created

dandischema/models.py

candleindark · 2024-12-09T04:06:20Z

@candleindark the json schema used by the frontend is set here - https://github.com/dandi/dandi-archive/blob/master/dandiapi/api/views/info.py#L11-L14 If you want to use a custom schema, you could serve out the json file locally and point it at that url.

@mvandenburgh I completed the initial setup, started the API server using docker compose up, and brought up the web app using yarn run dev. Both the API server and the web app seemed to be running normally. However, whenever I modified the value of schema_url in https://github.com/dandi/dandi-archive/blob/dc454f44b9e814b8a09781e9b798662e6cb8528f/dandiapi/api/views/info.py#L11-L14 to a URL of a locally served schema I got the "Connection to server failed." error. This was true even for the unmodified dandiset schema at 0.6.8, which seems to be the current default version (as schema_url = "http://localhost:9999/0.6.8/dandiset.json" or schema_url = "https://localhost:8443/0.6.8/dandiset.json").

On a related note, I was wondering if the web app/UI depends on the published-dandiset.json, asset.json, published-asset.json as well as the dandiset.json?

candleindark · 2024-12-11T06:41:42Z

I just tried the native development setup with dandiset.json hosted at a public URL using ngrok ( and set schema_url = "https://4ed0-2600-6c50-60f0-6500-9dec-627c-49b1-f7a0.ngrok-free.app/0.6.8/dandiset.json"). I got the same result as I mentioned in the last post.

mvandenburgh · 2024-12-11T14:18:49Z

On a related note, I was wondering if the web app/UI depends on the published-dandiset.json, asset.json, published-asset.json as well as the dandiset.json

The web UI only depends on dandiset.json.

@mvandenburgh I completed the initial setup, started the API server using docker compose up, and brought up the web app using yarn run dev. Both the API server and the web app seemed to be running normally. However, whenever I modified the value of schema_url in dandi/dandi-archive@dc454f4/dandiapi/api/views/info.py#L11-L14 to a URL of a locally served schema I got the "Connection to server failed." error. This was true even for the unmodified dandiset schema at 0.6.8, which seems to be the current default version (as schema_url = "http://localhost:9999/0.6.8/dandiset.json" or schema_url = "https://localhost:8443/0.6.8/dandiset.json").

On a related note, I was wondering if the web app/UI depends on the published-dandiset.json, asset.json, published-asset.json as well as the dandiset.json?

Does the web UI always display that "Connection to server failed" message, or just when you modify schema_url? If the former, then that indicates something is going wrong with the local Django server.

candleindark · 2024-12-11T17:39:55Z

Does the web UI always display that "Connection to server failed" message, or just when you modify schema_url? If the former, then that indicates something is going wrong with the local Django server.

It is the latter. I got the attached landing page if nothing no code is modified. In fact, there is no problem even if I hardcode the schema_url to a particular version. For example,

schema_url = (
    'https://raw.githubusercontent.com/dandi/schema/master/'
    f'releases/0.6.8/dandiset.json'
)

or even

schema_url = (
    'https://raw.githubusercontent.com/dandi/schema/master/'
    f'releases/0.6.7/dandiset.json'
)

The problem occurred when schema_url was set to the URLs I mentioned in previous posts (though the localhost ones are problematic if I run the Django service in a container).

To reflect that `@context` is not set to be required

candleindark · 2025-01-20T20:07:44Z

This post provides the analysis of Pydantic validation diff reports on the manifests against the latest version of dandischema at pypi and this PR.

Causes of Pydantic validation errors for `Dandiset` or `PublishedDandiset` instances as depicted in https://github.com/dandi/dandi-schema-status/blob/main/reports/diff_reports/dandiset/pydantic_errs_summary.md#pydantic-errs-diff are the following.

('extra_forbidden', 'Extra inputs are not permitted', ('contributor', '[*]', 'Person', 'affiliation', '[*]', 'contactPoint'))
- Some data instances include an undefined attribute of contactPoint for Affiliation as an element of Person.affiliation.
('extra_forbidden', 'Extra inputs are not permitted', ('contributor', '[*]', 'Person', 'affiliation', '[*]', 'includeInCitation'))
- Some data instances include an undefined attribute of includeInCitation for Affiliation as an element of Person.affiliation.
('extra_forbidden', 'Extra inputs are not permitted', ('contributor', '[*]', 'Person', 'affiliation', '[*]', 'roleName'))
- Some data instances include an undefined attribute of roleName for Affiliation as an element of Person.affiliation.
('extra_forbidden', 'Extra inputs are not permitted', ('datePublished',))
- The draft version of some dandisets instances contains the datePublished attribute which is not specified in Dandiset but PublishedDandiset. This result is to be expected according to @yarikoptic's previous analysis.
('extra_forbidden', 'Extra inputs are not permitted', ('doi',))
- The draft version of some dandisets instances contains the doi attribute which is not specified in Dandiset but PublishedDandiset. This result can be as acceptable as the one in the previous point.
('extra_forbidden', 'Extra inputs are not permitted', ('publishedBy',))
- The draft version of some dandisets instances contains the publishedBy attribute which is not specified in Dandiset but PublishedDandiset. This result can be as acceptable as the ones in the previous two points.

('value_error', 'Value error, Contact person must have an email address.', ('contributor', '[*]', 'Person'))

This error is removed for some dandiset instances because this "after" model validation was not executed after the "core" validation of the model failed. For example, in the following script, the "after" validator, check_x, is not executed since x fails to be validated as an int.

After model validator

from pydantic import BaseModel, model_validator, ValidationError

class A(BaseModel):
    x: str

    @model_validator(mode="after")
    def check_x(self):
        print("I am executed")
        return self


try:
    a = A.model_validate({"x": 1})
except ValidationError as e:
    print(e.json())
    """
    [{"type":"string_type","loc":["x"],"msg":"Input should be a valid string","input":1,"url":"https://errors.pydantic.dev/2.9/v/string_type"}]
    """

Causes of Pydantic validation errors for `Asset` or `PublishedAsset` instances as depicted in https://raw.githubusercontent.com/dandi/dandi-schema-status/refs/heads/main/reports/diff_reports/asset/pydantic_errs_summary.md are the following.

('extra_forbidden', 'Extra inputs are not permitted', ('datePublished',))
- The draft version of some asset instances contains the datePublished attribute which is not specified in Asset but PublishedAsset. This result is to be expected according to @yarikoptic's previous analysis.
('extra_forbidden', 'Extra inputs are not permitted', ('publishedBy',))
- The draft version of some asset instances contains the publishedBy attribute which is not specified in Asset but PublishedAsset. This result can be as acceptable as the one in the previous point.
('extra_forbidden', 'Extra inputs are not permitted', ('xxxx',))
- As depicted in the last table of https://raw.githubusercontent.com/dandi/dandi-schema-status/refs/heads/main/reports/diff_reports/asset/pydantic_errs_summary.md, some versions of the 000029 dataset contain an asset instance with an undefined "xxxx" @yarikoptic note: he inserted "manually" somehow, so it is legit "bad metadata"

Summary

Among the categories of errors listed above, the bolded items result from data instances containing undefined attributes. The remaining errors are either expected outcomes or have reasonable justifications. At the Pydantic validation level, there is no indication that the changes introduced in this PR result in any unintended behavior in the schema.

Bring in up-to-date changes from the base branch

feat: add model validator to remove "@context" key in Asset and `…

241bc08

…Dandiset` This allows validation of data instances of `Asset` and `Dandiset` to contain the `"@context"` key

candleindark force-pushed the allow-context-in-data branch from 05e6307 to 241bc08 Compare November 18, 2024 17:50

candleindark added 2 commits November 18, 2024 10:14

feat: Add the "@context" key to the generated JSON schema for `Asse…

eda8db0

…t` and `Dandiset` This requires `"@context"` field at the JSON level

feat: Disallow extra field in all Pydantic DANDI models

f1409d8

This also sets the `"additionalProperties"` of each corresponding model in JSON schema to `false`

fix: remove use of JsonDict and JsonValue

0a69296

These are not made available in `__all__`. Use of them triggers complains in the IDE.

yarikoptic reviewed Nov 19, 2024

View reviewed changes

fix: update "properties" and "required" keys of JSON schema

99a66ed

This is needed to handle the case in which the keys are newly created

candleindark force-pushed the allow-context-in-data branch from f9bba62 to 99a66ed Compare November 19, 2024 21:56

yarikoptic reviewed Nov 27, 2024

View reviewed changes

dandischema/models.py Outdated Show resolved Hide resolved

Make @context optional (for now at least) at jsonschema level

391aa1a

doc: update add_context docstring

fd3df90

To reflect that `@context` is not set to be required

Merge branch 'master' into allow-context-in-data

5ca9e5e

Bring in up-to-date changes from the base branch

yarikoptic mentioned this pull request Jan 22, 2025

Add proper migration for "Organization" -> "Affiliation" change #276

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disallow extra fields other than `"@context"` #266

Disallow extra fields other than `"@context"` #266

candleindark commented Nov 18, 2024 •

edited

Loading

codecov bot commented Nov 18, 2024 •

edited

Loading

candleindark commented Nov 18, 2024

yarikoptic commented Nov 18, 2024

mvandenburgh commented Nov 18, 2024

candleindark commented Nov 18, 2024

yarikoptic Nov 19, 2024

candleindark commented Dec 9, 2024 •

edited

Loading

candleindark commented Dec 11, 2024

mvandenburgh commented Dec 11, 2024

candleindark commented Dec 11, 2024

candleindark commented Jan 20, 2025 •

edited by yarikoptic

Loading

		@@ -1815,6 +1867,12 @@ class Asset(BareAsset):
		json_schema_extra={"readOnly": True, "nskey": "schema"}
		)

Disallow extra fields other than "@context" #266

Are you sure you want to change the base?

Disallow extra fields other than "@context" #266

Conversation

candleindark commented Nov 18, 2024 • edited Loading

codecov bot commented Nov 18, 2024 • edited Loading

Codecov Report

candleindark commented Nov 18, 2024

yarikoptic commented Nov 18, 2024

mvandenburgh commented Nov 18, 2024

candleindark commented Nov 18, 2024

yarikoptic Nov 19, 2024

Choose a reason for hiding this comment

candleindark commented Dec 9, 2024 • edited Loading

candleindark commented Dec 11, 2024

mvandenburgh commented Dec 11, 2024

candleindark commented Dec 11, 2024

candleindark commented Jan 20, 2025 • edited by yarikoptic Loading

Causes of Pydantic validation errors for Dandiset or PublishedDandiset instances as depicted in https://github.com/dandi/dandi-schema-status/blob/main/reports/diff_reports/dandiset/pydantic_errs_summary.md#pydantic-errs-diff are the following.

Causes of Pydantic validation errors for Asset or PublishedAsset instances as depicted in https://raw.githubusercontent.com/dandi/dandi-schema-status/refs/heads/main/reports/diff_reports/asset/pydantic_errs_summary.md are the following.

Summary

Disallow extra fields other than `"@context"` #266

Disallow extra fields other than `"@context"` #266

candleindark commented Nov 18, 2024 •

edited

Loading

codecov bot commented Nov 18, 2024 •

edited

Loading

candleindark commented Dec 9, 2024 •

edited

Loading

candleindark commented Jan 20, 2025 •

edited by yarikoptic

Loading

Causes of Pydantic validation errors for `Dandiset` or `PublishedDandiset` instances as depicted in https://github.com/dandi/dandi-schema-status/blob/main/reports/diff_reports/dandiset/pydantic_errs_summary.md#pydantic-errs-diff are the following.

Causes of Pydantic validation errors for `Asset` or `PublishedAsset` instances as depicted in https://raw.githubusercontent.com/dandi/dandi-schema-status/refs/heads/main/reports/diff_reports/asset/pydantic_errs_summary.md are the following.