feat(openchallenges): add EDAM Extract and Transform Processes

[mdsage1]

Description

EDAM ETL processes need to be developed to incorporate ETAM ontology in the Maria DB linking the ontology to existing data. This PR will address the extract and transform portion.

Related Issue

Contribute to #2524
Contribute to #2548

Fixes #2547
Fixes #2563

Changelog

• Add

1. Download a specified version of the EDAM ontology from https://github.com/Sage-Bionetworks/edamontology
2. Transform the raw data into a Pandas dataframe that match the content of this file
3. Start id values from 1 to mimic the behavior of SQL AUTO_INCREMENT.
4. Print info and statistic about the data to the stdout
5. Version of EDAM processed
6. Number of concepts transformed (overall, operation, data, etc.)

Preview

[mdsage1]

@tschaffter Quality Gate doesn't seem to be performing checks for this PR.

[mdsage1]

@tschaffter This is ready for review. Version is now an environment variable and the Description includes a preview.

[vpchung]

@mdsage1 thanks for working on this!! You didn't ask me to, but I added some comments to the PR. Feel free to use them or ignore 😄

[sonarcloud]

Quality Gate passed for 'openchallenges-edam-etl'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[tschaffter]

Fixes #2546

This PR is part of #2546, so this PR should not be configured to close this ticket. I will remove it from the list.

[tschaffter]

@mdsage1 Why do you show the std, mean and other metrics for the id column?

Update: here are the information that the script should print

• Version of EDAM processed
• Number of concepts that will be added to the table
  • Total number of concepts
  • Number of concepts for the following category
    • Data concepts
    • Operation concepts
    • Format concepts
    • Operation concepts
    • Other concepts

[tschaffter]

The config parameters should be validated before using them, otherwise the following behavior will occur:

$ nx serve-detach openchallenges-edam-etl

> nx run openchallenges-edam-etl:serve-detach

 Container openchallenges-mariadb  Recreate
 Container openchallenges-mariadb  Recreated
 Container openchallenges-edam-etl  Recreate
 Container openchallenges-edam-etl  Recreated
 Container openchallenges-mariadb  Starting
 Container openchallenges-mariadb  Started
 Container openchallenges-mariadb  Waiting
 Container openchallenges-mariadb  Healthy
 Container openchallenges-edam-etl  Starting
 Container openchallenges-edam-etl  Started

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target serve-detach for project openchallenges-edam-etl (33s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/Ivfcx2Zd35

vscode@dee30b82cf44:/workspaces/sage-monorepo$ docker logs openchallenges-edam-etl
EDAM Version: None
OC DB URL: jdbc:mysql://openchallenges-mariadb:3306/challenge_service
Downloading the EDAM concepts from GitHub (CSV file)...
Error downloading EDAM concepts: 404 Client Error: Not Found for url: https://github.com/edamontology/edamontology/raw/main/releases/EDAM_None.csv
Processing the EDAM concepts...
File EDAM_None.csv not found.
No data available.

[mdsage1]

closed in error

[tschaffter]

You could get this information from the column class_id and a regex.

See the suggestion I made above.

[vpchung]

You could get this information from the column class_id and a regex.

@mdsage1 alternatively, you can also do a replace() to remove the substring you don't need from class_id, if you're not comfortable with regex.

EDIT: Since you're interested in the number of concepts per category, you can actually use pandas' contains to get you closer to the count 🙂 e.g.

>>> df["class_id"].str.contains("data")
0        True
1        True
2        True
3        True
4        True
        ...  
3468    False
3469    False
3470    False
3471    False
3472    False

[tschaffter]

Prefer exact match to using contains (more future proof): contains would not work if the ontology were to have the concept Data and DataFormat, for example.

[vpchung]

if the ontology were to have the concept Data and DataFormat, for example.

Good point. Just shooting my shot here, but this can be overcome by using data_ (assuming they use "dataformat_"). Also, you can use regex with contains().

[mdsage1]

@tschaffter I've updated the concept counts to use the class_id column and regex. The case has been ignored to avoid any future issues. I didn't use contains but used search() function from the regex module. I have prevented future issues with data, and any other concept name, listing as a match when there is an additional word following the word of interest by adding the underscore to the regex as @vpchung suggested.

[vpchung]

I have one final suggestion, but otherwise, the script looks good on my end!

[mdsage1]

tested the link and updated it

[vpchung]

Suggested change

	def count_occurrences(identifier_pattern: str, df) -> int:
	def count_occurrences(identifier_pattern: str, df: pd.DataFrame) -> np.int64:

^^ IIRC. May need to double-check whether it is a numpy type being returned.

[mdsage1]

Numpy int is being returned and made changes.

[vpchung]

Cool cool. May need to add import numpy as np in this case!

[tschaffter]

Nice catch! I need to submit and patch for this project in #2594 and will fix this issue at the same time.