New Kingfisher Process integration

[jpmckinney]

No description provided.

[jpmckinney]

Dockerizing scrapyd makes it more complicated to deploy new spiders, so I will remove that part and deploy scrapyd normally.

[jpmckinney]

I'm not sure why this is needed:

[deploy:kingfisher]
url = http://localhost:6800
project = kingfisher

Maybe it was needed for the Docker deployment? But we will not need it for the regular deployment.

[jpmckinney]

Comment from #656:

The current extension also implements item_error and spider_error signals.

I think we decided not to continue this feature in the new version of Kingfisher Process. Analysts can instead check the Scrapy log to find these errors. See #531.

ideally, we would have coverage for the error scenarios. The existing tests provide a lot of inspiration for how to construct new tests.

This still needs to be addressed. Update: Done now.

The current extension also sends the file_name and url from the item, for which there are corresponding columns in the collection_file table. Also, for FileItem's, it sends the number.

Need to check how the new Process is implemented to see if this needs to be addressed or or not. Update: See #745

[jpmckinney]

I'm not sure how the bug referenced in this commit could occur: 04135ea

[jpmckinney]

I'm not sure how the bug referenced in this commit could occur: 04135ea

@jakubkrafka Do you remember how this error occurred? The FilesStore extension needs to be enabled for the Kingfisher integration to work (if it's disabled, no files are written to disk), and that extension will always add the files_store and path fields before the Kingfisher extension handles the item. And errors is a required field for FileError, so the item will not reach the Kingfisher extension if it's missing, because it will fail validation.

[jpmckinney]

If I understand the integration correctly, then FILES_STORE must be an absolute directory. Otherwise, I'm not sure how Kingfisher Process can reliably read the file. Update: It is now guaranteed to be absolute.

[jpmckinney]

The following were sent to the old Kingfisher Process, but are not sent to the new Kingfisher Process:

• file_name: Kingfisher Process now uses the full path, which is sent.
• data_type: Kingfisher Process now detects this using OCDS Kit's detect_format.
• encoding: Kingfisher Process expects UTF-8. I opened an issue to ensure files are written in UTF-8: Ensure files are UTF-8 encoded #783
• number: Always 0. Kingfisher Process now has a 1:1 relationship between File and FileItem. It simply loads each file written by Kingfisher Collect (which writes separate files for each FileItem).
• FileItem data: The old Kingfisher Process would receive this via web request. The new Kingfisher Process reads it from a file, same as for File data.

[jpmckinney]

I'm not sure how the bug referenced in this commit could occur: 04135ea

@jakubkrafka Do you remember how this error occurred? The FilesStore extension needs to be enabled for the Kingfisher integration to work (if it's disabled, no files are written to disk), and that extension will always add the files_store and path fields before the Kingfisher extension handles the item. And errors is a required field for FileError, so the item will not reach the Kingfisher extension if it's missing, because it will fail validation.

Got it: FileError does not have files_store or path. I can't reproduce a case where errors or url are missing though.