Chore/refactor import process to accept data from bucket #1204
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tity.ts and indicator-record.entity.ts entities
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 3 Skipped Deployments
|
alexeh
force-pushed
the
chore/refactor-import-process-to-accept-data-from-bucket
branch
2 times, most recently
from
33d61e0 to
144d1bb
Compare
alexeh
force-pushed
the
chore/refactor-import-process-to-accept-data-from-bucket
branch
from
144d1bb to
9da5b0f
Compare
alexeh
force-pushed
the
chore/refactor-import-process-to-accept-data-from-bucket
branch
from
66ab571 to
d5e5aca
Compare
alexeh
deleted the
chore/refactor-import-process-to-accept-data-from-bucket
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes significant updates to the data import and entity management processes, focusing on adding new workflows, refactoring entities, and improving data import scripts. The most important changes include adding a new GitHub Actions workflow for data import tests, refactoring entity classes to remove unused properties, and enhancing data import scripts for better logging and S3 integration.
New GitHub Actions Workflow:
.github/workflows/testing-data-import.yml: Added a new workflow for running data import tests on push and manual triggers.Entity Refactoring:
api/src/modules/indicator-coefficients/indicator-coefficient.entity.ts: RefactoredIndicatorCoefficientclass to extendBaseEntityinstead ofTimestampedBaseEntityand removed unused properties and imports. [1] [2]api/src/modules/indicator-records/indicator-record.entity.ts: RefactoredIndicatorRecordclass to extendBaseEntityinstead ofTimestampedBaseEntityand removed unused properties and imports. [1] [2] [3] [4]api/src/modules/users/user.entity.ts: Removed theindicatorCoefficientsrelationship from theUserentity.Data Import Script Enhancements:
data/base_data_importer/csv_to_table.py: Enhanced the script to include better logging, environment variable loading, and S3 integration for downloading base data files.data/indicator_coefficient_importer/indicator_coefficient_importer.py: Improved the script to handle downloading indicator coefficient files from S3, better logging, and configuration loading from environment variables. [1] [2] [3]Testing and Dependencies:
data/pytest.ini: Added configuration for pytest to specify test paths and patterns.data/requirements.txt: Added new dependencies forpython-dotenv,pytest, and related packages to support the enhanced data import scripts and testing.data/test/test_base_data_importer.py: Added a new test script to validate the base data import process using pytest.