Implement solution to write mapped data to storage #51
Conversation
sb-2011
commented
Apr 3, 2024
sb-2011
commented
- Added functionality to write post mapped data to disk
- Introduced landing zone for location to write this data
- Added tests including a test for uploading mapped data to storage using minio
sb-2011
force-pushed
the
add-datawriter-class
branch
2 times, most recently
from
6a3efb2 to
79fd4bf
Compare
.gitignore
Outdated
| @@ -182,3 +182,6 @@ node_modules/ | |||
|
|
|||
| # Mac Desktop Services Store | |||
| *.DS_Store | |||
|
|
|||
| # Landing zone data storage | |||
| # /data/landing_zone/* | |||
There was a problem hiding this comment.
I'm not sure whether we're intending to keep data/landing_zone/ under version control given that this is commented out. Do we want to keep the directory and the README.md under VC and ignore everything else in the folder? If so I think we can accomplish that as such in the .gitignore file:
data/landing_zone/*
!data/landing_zone/README.md
There was a problem hiding this comment.
Yes good call, I'll make the changes.
| @@ -68,11 +65,13 @@ def create_column_map_repository(self): | |||
| def create_logger(self): | |||
| return BasicLogger(__name__, logging.DEBUG) | |||
|
|
|||
| def create_storage(self): | |||
| @staticmethod | |||
There was a problem hiding this comment.
Does the create_storage method here and in the other ApplicationContext classes necessarily need to be static? Just curious.
There was a problem hiding this comment.
I made it static because I need to use a storage class instance in the task_queue.py file and I don't want to create an application context just for that use case. I wasn't able to pass the application context to the celery task like we discussed before due to serialization so this was a work around.
sb-2011
force-pushed
the
add-datawriter-class
branch
from
79fd4bf to
45a7e56
Compare
