We should move from using a sql database for file hosting to file hosting
None
Any assumptions that are made and their corresponding risks
The non-functional requirements that must be satisified to complete this epic
File storage service being used
Sql database being used but only to track downloads
API provides back-dated data and can be called at api.mergedfolding.net
We currently use the sql database to store the data. This is costly and slow, therefore the files should never be entirely stored in the sql database and should use file storage like azure files
The files can stay compressed to save on costs and the API will decompress them and read them into memory. This may still take some time but we can cache the files if it becomes a problem, it'll eventually return quickly.
Use the sql database to keep track of the files and their state because we still want to validate the files on download, we can always reprocess them to validate them before they will be considered by the API
Move the daily download service to run under the API service instead of a full windows server
- Update Database Scheme
- Update Download Service Implementation
- Add File Validation to Download Service
- Update API to Interface w/ File Storage
- Cache Files for Quick Access on Repeated Calls
- Create File Validation Service
- Cleanup V2 Changes
- Implement Azure as Data Store
- Create Daily Download Schedule Task
- Update Documentation
https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet?tabs=windows https://docs.microsoft.com/en-us/azure/storage/common/storage-samples-dotnet