Merge additional data during phylogenetic (instead of ingest) workflow

[j23414]

Description of proposed changes

This PR refines our approach to merging private data—such as latitude and longitude annotations for samples collected at mosquito traps—into state-level phylogenetic trees (e.g. Washington). Previously, we merged private data during the ingest workflow by extending an annotations.tsv file. This led to a convoluted phylogenetic config file that varied depending on whether data was sourced from S3 or from the ingest/results/* merged files.

In this update, private user data is incorporated directly during the phylogenetic workflow. We adopt a variation of the config.additional_inputs method (copied and modified from these lines in avian-flu) and further advocates for a generic pattern for including additional user data as being explored in the pathogen-repo-guide#72.

Config changes

Before

sequences_url: "https://data.nextstrain.org/files/workflows/WNV/sequences.fasta.zst"
metadata_url: "https://data.nextstrain.org/files/workflows/WNV/metadata.tsv.zst"

# Pull in metadata and sequences from the ingest directory after it has been annotated with Washington state-specific metadata
input_metadata: "../ingest/results/metadata.tsv"
input_sequences: "../ingest/results/sequences.fasta"

After

inputs:
  - name: ncbi
    metadata: "s3://nextstrain-data/files/workflows/WNV/metadata.tsv.zst"
    sequences: "s3://nextstrain-data/files/workflows/WNV/sequences.fasta.zst"

additional_inputs:
  - name: private-lat-longs
    metadata: "data-private/metadata.tsv"
    sequences: "data-private/sequences.fasta"

This updated configuration greatly simplifies our documentation, making it easier for other states to modify the Washington state builds and incorporate their own private data.

Related issue(s)

• Optional enhancement: Revisit current way of merging private data (via annotations.tsv) #65

Checklist

• Checks pass

[DOH-LMT2303]

I tested these changes in my instance and it ran without issues. Proceeded to check the JSON file in Auspice and the locations specified in the phylogenetic/data-private/metadata.tsv file were displayed in the map as expected when specifying the map geographic location to "location."