Transform FGDC/CSDGM source to DCAT-US

[btylerburton]

User Story

In order to test another facet of the transformation step in our pipeline, datagovteam would like to transform an FGDC/CSDGM WAF source to DCAT-US and push that into CKAN.

This story depends on:

• Harvester 2.0: Deploy mdTranslator on cloud.gov #4467
• Integrate MDTranslator into Datagov Harvesting Logic #4565
• Harden WAF ETL pipeline #4598

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN a harvest source with an CSDGM schema, stored in a WAF file format
  WHEN I add that to our ETL Pipeline with the appropriate harvest source config
  THEN I expect the mdTranslator to transform the source into valid DCAT-US datasets
  AND I expect it to be able to pushed into a CKAN development instance without error.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

No work planned outside the cloud.gov boundary.

Sketch

• Configure new Harvest source pointing to a test WAF instance of CSDGM standard datasets
• Extract that WAF using datagov-harvesting-logic WAF extract task
• Transform datasets into DCAT-US using the MDTranslator application
• Validate that the transformation conforms to DCAT-US standard
• Push valid DCAT-US datasets into CKAN
• Confirm that integration is successful and idempotent by running a test harvest source through the pipeline multiple times.

[btylerburton]

Combined with this ticket #4565

[btylerburton]

Reopening in light of the fact that DCAT-US writer is not yet available.

[rshewitt]

confirmed with johnathan that fgdc reader is in good shape. "it’s one of the better ones".