Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Column info for invalid column number #8

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Column info for invalid column number #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
58d4415
show column names for schema and data frame in case column numbers do…
Mar 12, 2018
1f93728
fix wrong show for add_schema_ and add_df_columns in if-statement
Mar 12, 2018
b0aaab7
added set to column list and unit test which demonstrates column name…
Jan 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 43 additions & 3 deletions pandas_schema/schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,51 @@ def validate(self, df: pd.DataFrame, columns: typing.List[str] = None) -> typing
schema_cols = len(self.columns)
columns_to_pair = self.columns
if df_cols != schema_cols:

schema_columns = set(self.get_column_names())
df_columns = set(df.columns)

add_schema_columns = [col for col in schema_columns if col not in df_columns]
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use set operations here, (https://docs.python.org/3/library/stdtypes.html#set) e.g. add_schema_columns = schema_columns - df_columns

add_df_columns = [col for col in df_columns if col not in schema_columns]

if not add_schema_columns:

errors.append(
ValidationWarning(
'Invalid number of columns. The schema specifies {n_schema}, '
'but the data frame has {n_df}. '
'The additional data frame columns are: {add_columns}.'.format(
n_schema=schema_cols,
n_df=df_cols,
add_columns=add_df_columns,
)
)
)
return errors

if not add_df_columns:
errors.append(
ValidationWarning(
'Invalid number of columns. The schema specifies {n_schema}, '
'but the data frame has {n_df}. The additional schema columns are: {add_columns}.'.format(
n_schema=schema_cols,
n_df=df_cols,
add_columns=add_schema_columns,
)
)
)
return errors

errors.append(
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you have a second ValidationWarning added here? You already have one error if add_schema_columns and one error if add_df_columns. If there is an error with the number of columns this will produce two ValidationWarning objects, when it should produce one.

ValidationWarning(
'Invalid number of columns. The schema specifies {}, but the data frame has {}'.format(
schema_cols,
df_cols)
'Invalid number of columns. The schema specifies {n_schema}, '
'but the data frame has {n_df}. The additional schema columns are: {add_columns_1} '
'and the additional data frame columns are: {add_columns_2}.'.format(
n_schema=schema_cols,
n_df=df_cols,
add_columns_1=add_schema_columns,
add_columns_2=add_df_columns,
)
)
)
return errors
Expand Down
7 changes: 7 additions & 0 deletions test/test_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ def test_column_subset_error(self):
# should raise a PanSchArgumentError
self.assertRaises(PanSchArgumentError, self.schema.validate, df, columns=['c'])

def test_column_not_present_shown(self):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a docstring that explains this test (can be one sentence)


df = pd.DataFrame.from_dict({'a': [1, 2, 3]})

out = self.schema.validate(df, columns=['a', 'b'])
assert out[0].message == 'The column b exists in the schema but not in the data frame'
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a self.assert() method (https://docs.python.org/3.7/library/unittest.html#unittest.TestCase.debug)



class OrderedSchema(unittest.TestCase):
schema = Schema([
Expand Down