feat(matrices): allow other formats for internal matrices storage

[MartinBelthle]

This PR does several things:

• Introduces a new field inside the application.yaml: matrixstore_format that dictates the internal storage format. Default value is still tsv to ensure backward compatibility
• When a new format is selected, and an user saves a matrix:
  • If the matrix didn't exist, it will be saved in the new format
  • If it already existed, we will migrate the matrix from its existing format to the new one. This way we'll migrate the matrixstore on the fly without having to interrupt the app.
• It also introduces a script that can be run if one day we want to migrate all matrices at once in a new format.

[laurent-laporte-pro]

Hi @MartinBelthle,

Given that the script successfully reduces the size of the matrices folder and needs to be run while the app is down, I was thinking it might be a good idea to integrate this script into the application startup process—specifically during the FastAPI setup phase.

This way, the migration would happen automatically each time the app starts. Since the script won’t do anything once all the TSV files have been converted to HDF5, it wouldn’t matter if it runs multiple times. If it doesn’t find any TSV files, it simply won’t perform any actions.

Of course, we'd need to ensure there’s enough space for the migration, especially on production where the data size could be much larger. Testing on integration and recette environments first, as you suggested, would be crucial.

What do you think about this approach?

Here is a possible implementation using a FastAPI event:

from fastapi import FastAPI

app = FastAPI()

def migrate_tsv_to_hdf5():
    print("Migrating TSV files to HDF5 format...")

@app.on_event("startup")
async def startup_event():
    migrate_tsv_to_hdf5()
    print("Startup event completed.")

[MartinBelthle]

Indeed I think that's a better way to do it.
I'll try to implement this behavior.

[MartinBelthle]

I believe that with the new solution Laurent proposed, this PR is mature and can be reviewed.

[MartinBelthle]

Seen with Sylvain, we have to discuss on this