Automate the build and deployment of sensitive-species-data.xml

[nickdos]

With the planned decommissioning of sds-webapp2, the generation of the sensitive-species-data.xml file needs to be moved to an automated process.

Details and history: https://confluence.csiro.au/display/ALASD/sds-webapp2+decommission.
The current manual process using sds-webapp2: https://confluence.csiro.au/display/ALASD/SDS+and+Sensitive+Lists

Evaluate whether AWS Code Pipeline is a suitable tool for automating this task

Requirements:

• Task should be easy to trigger manually by DM team
• download latest sds jar from Nexus (or where ever the latest builds are saved to)
• run the shell command to generate the sensitive-species-data.xml file
• perform some basic DQ checks on the XML file (file size, check for entries like Callocephalon fimbriatum, etc.)
• scp the file to the sensitive-ws server (test or prod) with the date as part of the file name. E.g. sensitive-species-data.xml-20240725
• copy the latest file to sensitive-species-data.xml, overwriting the current version (which should have a copy with its date in the filename, in case of regression).
• send a notification that it successfully completed (optional)

Check with data team, whether the XML file should be copied into a sds directory on archives.ala.org.au as an additonal step. This would allow all historical versions to be kept without clogging up the files on the sensitive-ws servers.

---

Steps needed & progress

• Implement Airflow script to automate the creation of XML file, saving it to s3
• Have the file served from sensitive-data-service root - e.g., https://sensitive-ws-test.ala.org.au/sensitive-species-data.xml
• Put a copy onto archives.ala.org.au

[nickdos]

Managed Airflow in AWS uses a Docker image for the "local runner" - where the Python code gets executed. This image can be used locally via this repo: https://github.com/aws/aws-mwaa-local-runner. The image has Java but not the "jar" command (JRE only). So work-around is to build a shaded jar, which allows the createxml.sh shell script to run a single "java" command.

[nickdos]

sensitive-data-service pulls static XML files down from sds.ala.org.au during the build phase - see docker-tasks.yml. This needs to be changed to pull from s3 instead.
Not needed as static files from sds.ala.org.au (4 of them) will still be served - just from Cloudfront and S3.