Catalog harvest does not report error when data.json is invalid

[FuhuXia]

When a data.json harvest source contains invalid entries according to DCAT-US Schema v1.1, the harvest report a successful harvest job with 0 changes. It is reported by rrb and confirmed on job https://admin-catalog-next.data.gov/harvest/rrb-json/job/2f728b7a-d954-469e-9b3a-81a2364a17b6.

How to reproduce

Create a data.json source containing entries that has no keyword and identifier. Harvest it.

Expected behavior

Harvest report specific errors.

Actual behavior

Successful harvest on the UI and email notification.

Context

If a data.json is not valid using tool https://dashboard.data.gov/validate, the gathering process should pick up the error. We had it working before but I am not sure when it started to be broken.

[FuhuXia]

Maybe related to another bug #3532.

[FuhuXia]

doc-gov json failed due to the same reason. It is an invalid DCAT-US Schema json file but gather continues to process the file without validating it first, then failed with some error in the gather log, but the error is not reported in the job report.

[FuhuXia]

doc-gov harvest failed at gather stage when a record does not have the required identifier field. The error is not reported.

Traceback (most recent call last):
File "/home/vcap/deps/1/bin/ckan", line 8, in <module>
sys.exit(ckan())
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/cli.py", line 241, in gather_consumer
utils.gather_consumer()
File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/utils.py", line 340, in gather_consumer
gather_callback(consumer, method, header, body)
File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/queue.py", line 374, in gather_callback
harvest_object_ids = gather_stage(harvester, job)
File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/queue.py", line 432, in gather_stage
harvest_object_ids = harvester.gather_stage(job)
File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckanext/datajson/datajson.py", line 231, in gather_stage
if dataset['identifier'] in unique_datasets:
KeyError: 'identifier'
Exit status 1

[nickumia-reisys]

@FuhuXia Is this being captured now because of #3532?

[FuhuXia]

no, still an issue.

[Jin-Sun-tts]

Before schema validation, the gather_stage and import_stage were referring to the identifier and title. In case these were not present in the dataset, errors would stop the harvest without reporting. To ensure that errors are reported, we have included checks for the dataset's title and identifier before referring to them.

[Jin-Sun-tts]

Tested on DEV with one dataset with missing identifier and one dataset with missing title , the log reports the error as expected like below: