Re-integrate (QA | ARCHIVER | REPORT) extensions to catalog.data.gov

[nickumia-reisys]

User Story

In order to restore dilapidated features on catalog.data.gov, the Data.gov Catalog team wants to re-integrate ckanext-qa, ckanext-archiver and ckanext-report.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN the extensions have been deployed to production
  WHEN I visit a dataset on catalog.data.gov
  THEN I can see that the extensions are live at https://catalog.data.gov/api/action/status_show

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

• Incorporate the extensions into a local test environment
• Inspect how the extensions work with some test data
• Add tests
• Deploy to development on cloud.gov
• Fix bugs (if there are any)
• Deploy to production

[nickumia-reisys]

List of things that work and don't work

ckan report generate openness -- OK

ckan qa update -- OK

ckan report list -- OK

Visiting /report -- OK

Visiting /report/broken-links -- OK

Visiting /report/openness -- NOT OK

• Fix /report/openness #3964
  

Openness scores on datasets -- NOT OK

• No openness scores on datasets from ckanext-qa #3965

Openness scores on resources -- NOT OK

• No openness scores on resources from ckanext-qa #3966

[nickumia-reisys]

cf run-task catalog-web -c "ckan report generate openness" -k 2G -m 2G

• Took 6 mins to complete 28834 datasets on development
• ETA for prod: 270 mins

[nickumia-reisys]

cf run-task catalog-web -c "ckan qa update" -k 2G -m 2G

• Took 3 mins to complete 28834 dataset on development
• ETA for prod: 135 mins

[nickumia-reisys]

Note: We are on a fork of ckanext-qa

• PY3 Fix - iteritems only in PY2 ckanext-qa#10

[jbrown-xentity]

Note: We are on a fork of ckanext-qa

• PY3 Fix - iteritems only in PY2 ckanext-qa#10

Do we want to attempt to merge these changes back upstream? That might be a separate ticket... Or were there more changes than just the PR above?

[nickumia-reisys]

So... archiver tries to use Redis to process the archival queue, but falls back to celery if there is an ImportError 😕

It looks like it's falling back to celery, but I'm not sure why...

Since celery is not installed, the openness score is not computed per dataset.

---

Do we want to attempt to merge these changes back upstream? That might be a separate ticket... Or were there more changes than just the PR above?

@jbrown-xentity That was the only PR. We can try to merge it upstream. But I think upstream is still supporting PY2, so we'd have to do a conditional iteritems for PY2 and items for PY3. When they completely drop support for PY2, it's an easy merge upstream.. but right now, I don't think they'll take it.

[nickumia-reisys]

This is the solution to getting the queue to finish!!

• No rating stars in ckan 2.8 ckan/ckanext-qa#67 (comment)

We need to specify the queue that has the work to do when calling ckan jobs worker

ckan jobs worker bulk

[nickumia-reisys]

Welp... looks like they don't care about PY2 support after all, Yay!

• ckan 2.9 compatibility fixes ckan/ckanext-qa#75
• https://github.com/ckan/ckanext-qa/pull/75/files#diff-4c03098a67d3e12c420b0d7dcaead5c049f389c7c918a4c59909858f4268a736R75

The above PR fixes two issues. The .items() issue and the template not found issue 😅

[nickumia-reisys]

Also going to note, at least we know of at least one CKAN instance that is actively use the QA extension, Finnish Digital Agency

• https://github.com/vrk-kpa